 My site is pretty well hidden from the search engines on purpose.
My problem is that I noticed that the forums show up in google search results, which is a problem for me. Is there a way to get that out of there? Google doesn't touch the rest of my site due to settings I already have. |
You can try adding a /robots.txt file in the root on your server with the following lines to (theoretically) stop all robots, including search engines, from crawling and listing your site: User-agent: * |
It's a very sticky situation.. from what I understand even if you block the crawlers the content will still be indexed if anyone ever links to a page from another site. The only surefire way to block the indexing (even if someone links to the page) is to use the noindex metatag. I think if you put it in the Dolphin header that would block EVERYTHING from being indexed. http://support.google.com/webmasters/bin/answer.py?hl=en&answer=93710 Even with that your content still might get scraped by any bots that don't follow the rules. If that's the case I think you might have to set it up so the content is only displayed if the user is logged in.
About the pages that are already indexed, if I'm not mistaken you can join Google Webmasters and request that they be removed from the index. I've never tried it so I'm not sure how it works or if it's effective. BoonEx Certified Host: Zarconia.net - Fully Supported Shared and Dedicated for Dolphin |
Thanks for responses here. I should have noted that my robots.txt is exactly as ggsinc put it: User-agent:* Disallow: / And my settings for the pages are just for members to see it. Its frustrating that only the forums stuff is showing up so the rest of the site is remaining un touched. I will contact google through the web master tools and post how I resolve this. |
