Which directories should I be allowing to be crawled, and which should not be crawled. Anybody have a list of bad bot names and IP ranges that should be blocked in .htaccess?
robots.txt what should be allowed?
 |
If any directories should be marked as disallowed, it should be the administration directory. Bad bots will ignore robots.txt, so it does no good to list them. BoonEx Certified Host: Zarconia.net - Fully Supported Shared and Dedicated for Dolphin |
Below is the legacy version of the Boonex site, maintained for Dolphin.Pro 7.x support.
The new Dolphin solution is powered by UNA Community Management System.
The new Dolphin solution is powered by UNA Community Management System.