Bot-trap for TNG and Apache 2.4

Posts: 119
Joined: Sun Oct 01, 2017 3:08 pm

Bot-trap for TNG and Apache 2.4

Post by Steve »

Should you use Bot-trap or not? The answer depends on your opinions, the data on your website and what you want to achieve. Most bots are helpful because without them you would not be reading this or able to find virtually anything else on the internet. We expect bots to follow the robots.txt rules but just like motorists following speed limits, many bots do not pay any attention to robots.txt. Unfortunately a bot-trap bans the IP address not the bot. If the person running the bot changes their IP address, the bot is no longer banned and another user, getting a dynamically assigned IP address, could be banned instead. If you install Bot-trap consider, how will an innocent user inheriting a banned IP address contact you? One way to remedy the problem is create an error handler (described below the download link). Error handlers allow you to display informational messages including an email address if you feel it is necessary.

Another option is install the Rip Prevention mod written by Brian McFadyen and Brent Hemphil. Many bots scrape data and often avoid bot traps. The Rip prevention mods checks if a visitor's accesses are rapid and repeated. If they are, a warning is issued and if the accesses continue rapid and repeatedly, the visitor is temporarily banned and an explanation page is displayed. Warnings and bans are disabled for administrators and logged in users. The mod creates a check_access.php file that can be edited manually to add or remove bots. You can optionally install the Rip Challenge Mod. This mod works with the Rip Prevention Mod by adding a CAPTCHA challenge after a configurable number of accesses (default 30) for non registered users. Using these two mods, along with Bot-Trap, Bot access has been dramatically reduced on my website.

Bryan Larson developed the TNG Bot-trap Mod which I used until Apache 2.4 was installed. This version of Bryan's Bot-trap was modified to work with Apache 2.4 authorization containers. It is NOT compatible with Apache 2.2 and uses "Require not ip" instead of "Deny from ip". If you have an existing Apache .htaccess file, extract the Bot-trap files to the TNG mod folder and run mod manager. Select "Run Checks" from the Bot-trap install menu. Run checks will look for the Apache containers and add them if they are not present. If your previous .htaccess file has denied ip addresses or hosts, edit the .htaccess file and change "Deny from" to "Require not" and move those entries between the <RequireAll> and </RequireAll> tags. Failure to place those lines between the tags may cause a server 500 error.

What was modified in this version?
  1. Compatible with Apache 2.4 using the <RequireAll></RequireAll> Apache Authorization Container tags.
  2. When a bot hits the trap, two file writes occur instead of three. One for .htaccess and one for blacklist.dat.
  3. Run Checks preserves existing lines in robots.txt and .htaccess and adds the correct values if they are missing.
  4. Added the option to make a backup copy of the existing .htaccess file before adding an IP address. If a failure occurs you can manually restore the copy.
  5. Created a optional file to protect TNG folders from unauthorized direct access. (optional)
  6. Created an error message page that loads provided you add the error handler line to .htaccess. (optional)
How does Bot-trap work? Bot-trap adds a small graphic with a link that humans do not see but bots can. When the link is opened bot-trap bans the IP address immediately but provides an option for a user to unban themselves. Users can select the "I'm human" button and type the correct response to unban themselves. If a user or Bot abandons the page without unbanning themselves the IP address remains banned.

While Bot-trap works great at banning nosy bots, it does not protect TNG folders you list in robots.txt from direct access. TNG has a index.html file to prevent direct access but it will not ban bots and they will continue their scans of other folders. An index.html file is included, in the optional files folder, to further enhance Bot-trap. Using this file is optional and is not required for Bot-trap. To protect a folder, rename the existing TNG index.html file and copy the new Bot-trap index.html file to any TNG folder you want to protect using Bot-trap. DO NOT place this file in the TNG root folder or TNG will not load and you will be banned. When a protected folder is accessed directly, the new index file loads Bot-trap giving the user an opportunity to avoid being banned. If you use this file make sure you add the protected folder names to the robots.txt file. Failure to do so may ban good bots that index your web site. The new index file file does not prevent users or bots from accessing information through the TNG program.

If you have TNG installed inside a CMS, and are using the CMS footer, you will need to add a line, similar to the one below, in the theme's footer.php file to set the Bot-trap.

Code: Select all

echo "<a href="../TNGFOLDER/bot-trap/"><img src="../TNGFOLDER/bot-trap/pixel.gif" border="0" alt=" " width=”1" height="1"></a>\n";
Bot-trap writes banned IPs to the TNG .htaccess file but does not change the CMS .htaccess file.

This file was tested on a Synology server using TNG v12.0.1, v12.1, v12.2, v12.3, Apache 2.4, PHP5.6, PHP7.0 and PHP7.2. It should be compatible with earlier versions of TNG that use the stdsitecredit file but was not tested.

Use caution as you could lock yourself out of your own site if you don't unban yourself or otherwise remove your own IP address from the .htaccess file.

Creating an 403 error page for your website
If your TNG website requires a login you DO NOT need Bot-trap. If you install Bot-trap anyway, users will not be able to unban themselves. However, there may be a situation where a user inherits a banned IP from their provider. Since their IP is banned they may never get a chance to unban themselves. To give users a way to contact admin for your website, add an error handler to your .htaccess file. When a user is denied access the error page will load with information including a contact email address, provided they are not using IE. Apparently Microsoft does like their users receiving informational messages if access is denied. If you want to display a message, instead of a blank 403 error page, add the code below to your TNG .htaccess file. This creates an error handler that loads the 403.php file included with Bot-trap.

Code: Select all

ErrorDocument 403 "<meta http-equiv='refresh' content='0; url=/bot-trap/403.php'/>"
The 403 error page displays a message along with a contact email address, provided you enter one in the Bot-trap options menu. Once you've added the error handler, if someone is banned or trips the bot-trap and does not complete the unban procedure correctly the message below will display so they can contact you. If you DO NOT want to display the error page, do not add the ErrorDocument code to .htaccess.
If you manually remove an IP address from .htaccess file, do not forget to remove the address from the blacklist.dat file.

Thanks to Daniel Webb for originally creating Bot-trap and Bryan Larson for developing TNG Bot-trap.
You do not have the required permissions to view the files attached to this post.
Last edited by Steve on Fri Aug 24, 2018 10:14 pm, edited 6 times in total.