Bot-trap for TNG and Apache 2.4

Post Reply
Steve
Posts: 63
Joined: Sun Oct 01, 2017 3:08 pm

Bot-trap for TNG and Apache 2.4

Post by Steve » Mon Aug 20, 2018 6:56 am

Should you use Bot-trap or not? The answer depends on your opinions, the data on your website and what you want to achieve. For example, SKDavis.net DOES NOT use a bot-trap. The contents of this site provides information and code to help people. I don't really care about bots since this is a public site for everyone, including bots. Most bots are helpful because without them you would not be reading this or able to find virtually anything else on the internet.

My genealogy site is another matter. In that case it's just not my opinion but the opinion of family members too. Although the site is largely public it was met to share information between family members and help others, that might be related, find their ancestors. The site contains private information on living individuals and photos not met for public distribution. It may be my imagination, but for some reason that site seems to attract more bots, especially bad ones. We expect bots to follow the robots.txt rules but just like motorists following speed limits, many bots do not pay any attention to robots.txt. So I opted to use Bot-trap as a way to filter out some of the bad bots. Unfortunately a bot-trap bans the IP address not the bot. If the person running the bot changes their IP address, the bot is no longer banned and another user, getting a dynamically assigned IP address, could be banned instead. I use a CMS (Content Management Systems are programs like WordPress, Joomla, Drupal etc.) for TNG which provides a separate contact us page. SInce the CMS and TNG have different .htaccess files, anyone getting banned can still contact us so they can be unbanned manually. Before installing Bot-trap consider, how will an innocent user inheriting a banned IP address contact you? One way to remedy the problem is create an error handler (described below the download link). Error handlers allow you to display informational messages including an email address if you feel it is necessary.

Bryan Larson developed the TNG Bot-trap Mod which I used until Apache 2.4 was installed. This version of Bryan's Bot-trap was modified to work with Apache 2.4 authorization containers. It is NOT compatible with Apache 2.2 and uses "Require not ip" instead of "Deny from ip". If you have an existing Apache .htaccess file, extract the Bot-trap files to the TNG mod folder and run mod manager. Select "Run Checks" from the Bot-trap install menu. Run checks will look for the Apache containers and add them if they are not present. If your previous .htaccess file has denied ip addresses or hosts, edit the .htaccess file and change "Deny from" to "Require not" and move those entries between the <RequireAll> and </RequireAll> tags. Failure to place those lines between the tags may cause an error.

What was modified in this version?
  1. Compatible with Apache 2.4 using the <RequireAll></RequireAll> Apache Authorization Container tags.
     
  2. When a bot hits the trap, two file writes occur instead of three. One for .htaccess and one for blacklist.dat.
     
  3. Run Checks preserves existing lines in robots.txt and .htaccess and adds the correct values if they are missing.
     
  4. Added the option to make a backup copy of the existing .htaccess file before adding an IP address. If a failure occurs you can manually restore the copy.
     
  5. Created a optional file to protect TNG folders from unauthorized direct access. (optional)
     
  6. Created an error message page that loads provided you add the error handler line to .htaccess. (optional)
How does Bot-trap work? Bot-trap creates a small graphic with a link that humans do not see but bots can. When the link is opened bot-trap bans the IP address immediately but provides an option for a user to unban themselves. Users can select the "I'm human" button and type the correct response to unban themselves. If a user or Bot abandons the page without unbanning themselves the IP address remains banned.

While Bot-trap works great at banning nosy bots, it does not protect TNG folders you list in robots.txt from direct access. TNG has a index.html file to prevent direct access but it will not ban bots and they will continue their scans of other folders. An index.html file is included, in the optional files folder, to further enhance Bot-trap. Using this file is optional and is not required for Bot-trap. To protect a folder, rename the existing TNG index.html file and copy the new Bot-trap index.html file to any TNG folder you want to protect using Bot-trap. DO NOT place this file in the TNG root folder or TNG will not load and you will be banned. When a protected folder is accessed directly, the new index file loads Bot-trap giving the user an opportunity to avoid being banned. If you use this file make sure you add the protected folder names to the robots.txt file. Failure to do so may ban good bots that index your web site. The new index file file does not prevent users or bots from accessing information through the TNG program.

If you have TNG installed inside a CMS, and are using the CMS footer, you will need to add a line, similar to the one below, in the theme's footer.php file to set the Bot-trap.

Code: Select all

echo "<a href=\"../TNGFOLDER/bot-trap/\"><img src=\"../TNGFOLDER/bot-trap/pixel.gif\" border=\"0\" alt=\" \" width=\1\" height=\"1\"/></a>\n";
Bot-trap writes banned IPs to the TNG .htaccess file but does not change the CMS .htaccess file.

This file was tested on a Synology server using TNG 12.0.1, Apache 2.4, PHP5.6 and PHP7.0. It should be compatible with earlier versions of TNG that use the stdsitecredit file but was not tested.

Use caution as you could lock yourself out of your own site if you don't unban yourself or otherwise remove your own IP address from the .htaccess file.

Bot-trap_v12.0.0.6-180824.zip
(21.84 KiB) Downloaded 18 times


Creating an 403 error page for your website
 
If your TNG website requires a login you DO NOT need Bot-trap. If you install Bot-trap anyway, users will not be able to unban themselves. However, there may be a situation where a user inherits a banned IP from their provider. Since their IP is banned they may never get a chance to unban themselves. To give users a way to contact admin for your website, add an error handler to your .htaccess file. When a user is denied access the error page will load with information including a contact email address, provided they are not using IE. Apparently Microsoft does like their users receiving informational messages if access is denied. If you want to display a message, instead of a blank 403 error page, add the code below to your TNG .htaccess file. This creates an error handler that loads the 403.php file included with Bot-trap.

Code: Select all

ErrorDocument 403 "<meta http-equiv='refresh' content='0; url=/bot-trap/403.php'/>"
The 403 error page displays a message along with a contact email address, provided you enter one in the Bot-trap options menu. Once you've added the error handler, if someone is banned or trips the bot-trap and does not complete the unban procedure correctly the message below will display so they can contact you. If you DO NOT want to display the error page, do not add the ErrorDocument code to .htaccess.
 
Image
 
If you manually remove an IP address from .htaccess file, do not forget to remove the address from the blacklist.dat file.

Thanks to Daniel Webb for originally creating Bot-trap and Bryan Larson for developing TNG Bot-trap.
Last edited by Steve on Fri Aug 24, 2018 10:14 pm, edited 6 times in total.

Post Reply