Another option is install the Rip Prevention mod written by Brian McFadyen and Brent Hemphil. Many bots scrape data and often avoid bot traps. The Rip prevention mods checks if a visitor's accesses are rapid and repeated. If they are, a warning is issued and if the accesses continue rapid and repeatedly, the visitor is temporarily banned and an explanation page is displayed. Warnings and bans are disabled for administrators and logged in users. The mod creates a check_access.php file that can be edited manually to add or remove bots. You can optionally install the Rip Challenge Mod. This mod works with the Rip Prevention Mod by adding a CAPTCHA challenge after a configurable number of accesses (default 30) for non registered users. Using these two mods, along with Bot-Trap, Bot access has been dramatically reduced on my website.
Bryan Larson developed the TNG Bot-trap Mod which I used until Apache 2.4 was installed. This version of Bryan's Bot-trap was modified to work with Apache 2.4 authorization containers. It is NOT compatible with Apache 2.2 and uses "Require not ip" instead of "Deny from ip". If you have an existing Apache .htaccess file, extract the Bot-trap files to the TNG mod folder and run mod manager. Select "Run Checks" from the Bot-trap install menu. Run checks will look for the Apache containers and add them if they are not present. If your previous .htaccess file has denied ip addresses or hosts, edit the .htaccess file and change "Deny from" to "Require not" and move those entries between the <RequireAll> and </RequireAll> tags. Failure to place those lines between the tags may cause a server 500 error.
What was modified in this version?
- Compatible with Apache 2.4 using the <RequireAll></RequireAll> Apache Authorization Container tags.
- When a bot hits the trap, two file writes occur instead of three. One for .htaccess and one for blacklist.dat.
- Run Checks preserves existing lines in robots.txt and .htaccess and adds the correct values if they are missing.
- Added the option to make a backup copy of the existing .htaccess file before adding an IP address. If a failure occurs you can manually restore the copy.
- Created a optional file to protect TNG folders from unauthorized direct access. (optional)
- Created an error message page that loads provided you add the error handler line to .htaccess. (optional)
While Bot-trap works great at banning nosy bots, it does not protect TNG folders you list in robots.txt from direct access. TNG has a index.html file to prevent direct access but it will not ban bots and they will continue their scans of other folders. An index.html file is included, in the optional files folder, to further enhance Bot-trap. Using this file is optional and is not required for Bot-trap. To protect a folder, rename the existing TNG index.html file and copy the new Bot-trap index.html file to any TNG folder you want to protect using Bot-trap. DO NOT place this file in the TNG root folder or TNG will not load and you will be banned. When a protected folder is accessed directly, the new index file loads Bot-trap giving the user an opportunity to avoid being banned. If you use this file make sure you add the protected folder names to the robots.txt file. Failure to do so may ban good bots that index your web site. The new index file file does not prevent users or bots from accessing information through the TNG program.
If you have TNG installed inside a CMS, and are using the CMS footer, you will need to add a line, similar to the one below, in the theme's footer.php file to set the Bot-trap.
Code: Select all
echo "<a href="../TNGFOLDER/bot-trap/"><img src="../TNGFOLDER/bot-trap/pixel.gif" border="0" alt=" " width=”1" height="1"></a>\n";
This file was tested on a Synology server using TNG v12.0.1, v12.1, Apache 2.4, PHP5.6, PHP7.0 and PHP7.2. It should be compatible with earlier versions of TNG that use the stdsitecredit file but was not tested.
Use caution as you could lock yourself out of your own site if you don't unban yourself or otherwise remove your own IP address from the .htaccess file.
If your TNG website requires a login you DO NOT need Bot-trap. If you install Bot-trap anyway, users will not be able to unban themselves. However, there may be a situation where a user inherits a banned IP from their provider. Since their IP is banned they may never get a chance to unban themselves. To give users a way to contact admin for your website, add an error handler to your .htaccess file. When a user is denied access the error page will load with information including a contact email address, provided they are not using IE. Apparently Microsoft does like their users receiving informational messages if access is denied. If you want to display a message, instead of a blank 403 error page, add the code below to your TNG .htaccess file. This creates an error handler that loads the 403.php file included with Bot-trap.
Code: Select all
ErrorDocument 403 "<meta http-equiv='refresh' content='0; url=/bot-trap/403.php'/>"
If you manually remove an IP address from .htaccess file, do not forget to remove the address from the blacklist.dat file.
Thanks to Daniel Webb for originally creating Bot-trap and Bryan Larson for developing TNG Bot-trap.