Hi CJ,
There are several things you can do to prevent content from being spidered and indexed by search engines:
1. Block search engines from indexing specific directory on the site using a robots.txt file:
http://davidwalsh.name/robots-txt
Here is some example code:
User-agent: *
Disallow: /cmsAdmin/uploads/
2. Disable Directory Listing using .htaccess:
http://www.besthostratings.com/articles/prevent-directory-listing.html
Example code to add to your .htaccess file to prevent directory browsing:
Options –Indexes
3. Remove any links to the content that is on the public side of the site (not password protected). If the content was found via a Google search, looking at the file name of the PDF (or link) and searching for that will help find the location of the link Google used to find it originally.
4. The last thing to do is include an index.html or index.php file in directories so that it will be display be default.
Hope this helps. If you have any questions, please let mek now.
Thanks!
Cheers,
Damon Edis - interactivetools.com
Hire me! Save time by getting our experts to help with your project.
http://www.interactivetools.com/consulting/