Update for bot blocker

Update for bot blocker as mentioned here: https://3xn.nl/projects/2023/09/20/crude-solution-to-ban-bots-by-their-user-agent/

My website is still being kicked around sometimes, but I’m winning D:

'ubermetrics-technologies', 'seokicks', 'ClaudeBot', 'serpstatbot', 'Barkrowler', 'YisouSpider', 'babbar.tech', 'AwarioBot', 'bytespider', 'amazonbot', 'MJ12bot', 'YandexBot', 'SemrushBot', 'dotbot', 'AspiegelBot', 'DataForSeoBot', 'DotBot', 'Pinterestbot', 'PetalBot', 'HeadlessChrome', 'GPTBot', 'Sogou', 'ALittle Client', 'fidget-spinner-bot', 'intelx.io_bot', 'Mediatoolkitbot', 'BLEXBot', 'AhrefsBot'

Loading

Update in the script to block bots, spiders and indexers

Original post: https://3xn.nl/projects/2023/09/20/crude-solution-to-ban-bots-by-their-user-agent/

I’ve very much simplified the script that instantly redirects unwanted traffic away from the server. Currently, I am using a very cheap VPS to receive all that traffic.

Here ya go:

<?php

// CC-BY-NC (2023)

// Author: FoxSan - fox@cytag.nl

// This is a functional but dirty hack to block bots, spiders and indexers by looking at the HTTP USER AGENT.
// Traffic that meets the conditions is being yeeted away to any place of your choice.

//////////////////////////////////////////////////////////////
// Emergency bypass
// goto end;
//////////////////////////////////////////////////////////////

// attempt to basically just yeet all bots to another website

$targetURL = "https://DOMAIN.TLD/SUB/";

// Function to check if the user agent appears to be a bot or spider

function isBot()

{

    $user_agent = $_SERVER['HTTP_USER_AGENT'];

$bot_keywords = ['bytespider', 'amazonbot', 'MJ12bot', 'YandexBot', 'SemrushBot', 'dotbot', 'AspiegelBot', 'DataForSeoBot', 'DotBot', 'Pinterestbot', 'PetalBot', 'HeadlessChrome', 'GPTBot', 'Sogou', 'ALittle Client', 'fidget-spinner-bot', 'intelx.io_bot', 'Mediatoolkitbot', 'BLEXBot', 'AhrefsBot'];

    foreach ($bot_keywords as $keyword) {

        if (stripos($user_agent, $keyword) !== false) {

            return true;

        }
    }

    return false;

}

// Check if the visitor is a bot or spider

if (isBot()) {

// yeet

header("Location: $targetURL");

    // Exit to prevent further processing

    exit;

}

end:

// If the visitor is not a bot, spider, or crawler, continue with your website code.

//////////////////////////////////////////////////////////////////////

?>

Loading

More tweaks for the .htaccess file!

Here’s a list of stuff that I have in my .htaccess files on various websites.

I want to work on my website, but any other visitor should be booted to another website so I can work in peace. Sidenote: It's forever since I last used this, so it might work. Or not.

---

# YOUR IP address goes here:
RewriteCond %{REMOTE_ADDR} !^000\.000\.000\.000$
# And provides you access to:
RewriteCond %{REQUEST_URI} !^https://DOMAIN.TLD$ [NC]
# Fine, go have all the media as well
RewriteCond %{REQUEST_URI} !\.(jpg|jpeg|png|gif|svg|swf|css|ico|js)$ [NC]
# Any other visitor can go visit the following website:
RewriteRule .* https://DOMAIN.TLD/ [R=302,L]

# Hey, no viewing access to this file
<FilesMatch "^.ht">
Order deny,allow
Deny from all
</FilesMatch>

# Disable Server Signature
ServerSignature Off

# SSL all the things!
RewriteCond %{HTTPS} !=on
RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1 [R,L]

# No WWW
RewriteCond %{HTTP_HOST} ^www\.DOMAIN\.TLD$
RewriteRule ^/?$ "https\:\/\/DOMAIN\.TLD\/" [R=301,L]

# Do we like Symlinks? Yeah we do.
Options +FollowSymlinks

# No open directories or directory listings. What is this... 1998?
Options All -Indexes
IndexIgnore *

# Rewrite rules to block out some common exploits.
RewriteCond %{QUERY_STRING} base64_encode[^(]*\([^)]*\) [OR]
RewriteCond %{QUERY_STRING} (<|%3C)([^s]*s)+cript.*(>|%3E) [NC,OR]
RewriteCond %{QUERY_STRING} GLOBALS(=|\[|\%[0-9A-Z]{0,2}) [OR]
RewriteCond %{QUERY_STRING} _REQUEST(=|\[|\%[0-9A-Z]{0,2})
RewriteRule .* index.php [F]

# PHP doohickies
php_flag register_globals off 
php_flag safe_mode off 
php_flag allow_url_fopen off 
php_flag display_errors off 
php_value session.save_path '/tmp' 
php_value disable_functions "exec,passthru,shell_exec,system,curl_multi_exec,show_source,eval"

# File Injection Protection, or a code-condom. What.
RewriteCond %{REQUEST_METHOD} GET
RewriteCond %{QUERY_STRING} [a-zA-Z0-9_]=http:// [OR]
RewriteCond %{QUERY_STRING} [a-zA-Z0-9_]=(\.\.//?)+ [OR]
RewriteCond %{QUERY_STRING} [a-zA-Z0-9_]=/([a-z0-9_.]//?)+ [NC]
RewriteRule .* - [F]

# /proc/self/environ? Go away!
RewriteCond %{QUERY_STRING} proc/self/environ [NC,OR]

# Disallow Access To Sensitive Files. Enter your own file names.
RewriteRule ^(htaccess.txt|configuration.php(-dist)?|joomla.xml|README.txt|web.config.txt|CONTRIBUTING.md|phpunit.xml.dist|plugin_googlemap2_proxy.php)$ - [F]

# Don't allow any pages to be framed - Defends against CSRF
<IfModule mod_headers.c>
Header set X-Frame-Options SAMEORIGIN
</IfModule>

# Disallow Php Easter Eggs
RewriteCond %{QUERY_STRING} \=PHP[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12} [NC]
RewriteRule .* index.php [F]

# Libwww-perl Access Block
RewriteCond %{HTTP_USER_AGENT} libwww-perl.* 
RewriteRule .* ? [F,L]

# Uh. I forgot.
<IfModule mod_autoindex.c>
IndexIgnore *
</IfModule>

# NO SNIFFYWIFFY OwO
<IfModule mod_headers.c>
Header always set X-Content-Type-Options "nosniff"
</IfModule>

# NEEDS TESTING
# Turn on IE8-IE9 XSS prevention tools
#Header set X-XSS-Protection "1; mode=block"

# NEEDS TESTING TOO
# Only allow JavaScript from the same domain to be run.
# Don't allow inline JavaScript to run.
#Header set X-Content-Security-Policy "allow 'self';"

# Example if you don't like Russia and Turkey (Optional A1 is to block anonymous proxies)
RewriteCond %{ENV:GEOIP_COUNTRY_CODE} ^(RU|TR)$
RewriteRule .* https://DOMAIN.TLD/directorywithindexdothtml/ [R=302,L]

# Caching stuff
<FilesMatch "\.(ico|jpg|jpeg|png|gif|js|css|swf)$">
Header set Cache-Control "public, max-age=3600"
</FilesMatch>

# Compress text, html, javascript, css, xml!
<IfModule mod_deflate.c>
AddOutputFilterByType DEFLATE text/html text/xml text/css text/plain
AddOutputFilterByType DEFLATE image/svg+xml application/xhtml+xml application/xml
AddOutputFilterByType DEFLATE application/rdf+xml application/rss+xml application/atom+xml
AddOutputFilterByType DEFLATE text/javascript application/javascript application/x-javascript application/json
AddOutputFilterByType DEFLATE application/x-font-ttf application/x-font-otf
AddOutputFilterByType DEFLATE font/truetype font/opentype
</IfModule>

# Joomla! core SEF Section.
RewriteRule .* - [E=HTTP_AUTHORIZATION:%{HTTP:Authorization}]
RewriteCond %{REQUEST_URI} !^/index\.php
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule .* index.php [L]

# PHP TWEAKS 
php_value upload_max_filesize 200M
php_value post_max_size 200M
php_value max_input_vars 2000
php_value max_execution_time 120
php_value memory_limit 1024M

## BEGIN EXPIRES CACHING 
<IfModule mod_expires.c>
ExpiresActive on
ExpiresDefault "access plus 1 month"
ExpiresByType text/cache-manifest "access plus 0 seconds"
ExpiresByType text/html "access plus 0 seconds"
ExpiresByType text/xml "access plus 0 seconds"
ExpiresByType application/xml "access plus 0 seconds"
ExpiresByType application/json "access plus 0 seconds"
ExpiresByType application/rss+xml "access plus 1 hour"
ExpiresByType application/atom+xml "access plus 1 hour"
ExpiresByType image/x-icon "access plus 1 week"
ExpiresByType image/gif "access plus 1 month"
ExpiresByType image/png "access plus 1 month"
ExpiresByType image/jpg "access plus 1 month"
ExpiresByType image/jpeg "access plus 1 month"
ExpiresByType video/ogg "access plus 1 month"
ExpiresByType audio/ogg "access plus 1 month"
ExpiresByType video/mp4 "access plus 1 month"
ExpiresByType video/webm "access plus 1 month"
ExpiresByType text/x-component "access plus 1 month"
ExpiresByType application/x-font-ttf "access plus 1 month"
ExpiresByType font/opentype "access plus 1 month"
ExpiresByType application/x-font-woff "access plus 1 month"
ExpiresByType image/svg+xml "access plus 1 month"
ExpiresByType application/vnd.ms-fontobject "access plus 1 month"
ExpiresByType text/css "access plus 1 year"
ExpiresByType text/javascript "access plus 1 year"
ExpiresByType application/javascript "access plus 1 year"
<IfModule mod_headers.c>
Header append Cache-Control "public"
</IfModule>
</IfModule>
## END EXPIRES CACHING

Loading

Crude solution to ban bots by their user-agent

Okay, this is a very crude way to block bots, spiders and crawlers by their user-agent, but so far, this has been very, very efficient.

Even when one chooses ” yes “, the question will be repeated. This is not a problem, because no one in their right mind is going to add “bot”, “spider” or “crawler” as their user-agent.

So here’s the PHP script that I rammed into a certain website to prevent it from being DDOSsed by (malicious) bots.

<?php

// CC-BY-NC (2023)
// Author: FoxSan - fox@cytag.nl
// This is a functional but dirty hack to block bots, spiders and indexers by looking at the HTTP USER AGENT.
// The form is, iirc, not even working, but that's fine if you only want human visitors.
// It can also throw a 403, but the effect is the same.

////////////////////////////////////////////////////////////////////////////////
// Emergency bypass
// goto end;
////////////////////////////////////////////////////////////////////////////////

// Function to check if the user agent appears to be a bot or spider.
// Enter the bots you would like to block in a list as shown below.
function isBot()
{
    $user_agent = $_SERVER["HTTP_USER_AGENT"];
    $bot_keywords = ['bytespider', 
                     'amazonbot', 
                     'MJ12bot', 
                     'YandexBot', 
                     'SemrushBot', 
                     'dotbot', 
                     'AspiegelBot',
                     'DataForSeoBot',
                     'DotBot',
                     'Pinterestbot',
                     'PetalBot',
                     'HeadlessChrome', 
                     'AhrefsBot'];

    foreach ($bot_keywords as $keyword) {
        if (stripos($user_agent, $keyword) !== false) {
            return true;
        }
    }

    return false;
}

// Check if the visitor is a bot or spider
if (isBot()) {
    // This visitor appears to be a bot or spider, so display a choice.
    // Check if the choice form is submitted
    if (isset($_POST["submit"])) {
        // Check the choice made by the visitor
        $choice = isset($_POST["choice"]) ? $_POST["choice"] : "";

        if ($choice === "yes") {
            // User selected "Yes," block access
            echo "Access denied. If you believe this is an error, please contact us by writing the word [MAILBOX] before the at sign, followed by [DOMAIN.TLD]";
        } elseif ($choice === "no") {
            // User selected "No," proceed to end
            goto end;
        }
    } else {
        // Output the message to the user and make the choice mandatory
        echo "Your user agent suggests you might be a bot, spider, or crawler. Are you one of these three?";

        // Output the radio button choices within a form
        echo '</p>
<form method="post" action="">';
        echo ' <label><input type="radio" name="choice" value="yes" required>Yes</label>';
        echo ' <label><input type="radio" name="choice" value="no">No</label>';
        echo ' <button type="submit" name="submit">Proceed</button>';
        echo "</form>
<p>";
    }

    // Exit to prevent further processing
    exit();
}
end:
// Original website code starts from here.
/////////////////////////////////////////////////////////////
?>

Loading

A working Apache2 server with PHP7.4

I was in need of a server solution that could be quickly deployed as a VM.

      1. Install Debian 11 as a VM with web- and SSH server
      2. Create a USER next to your root account during the installation
      3. Find the IP address of the new installation. The easiest is if you have NoVNC running. Log in as USER and type
        ip a
      4. Time to so the sudo thing
        su

        log in as root

        apt-get update && apt-get install -y sudo
        usermod -aG sudo USER
        exit
        exit

        log back in as USER

      5. Okay, let’s install some more stuff but first we do an update
        sudo apt-get update && sudo apt-get upgrade -y

        Now we want some essentials

        sudo apt-get install -y dirmngr gnupg2 nano wget gpg curl fail2ban ufw software-properties-common

        Preparing the PHP install

        wget -q https://packages.sury.org/php/apt.gpg -O- | sudo apt-key add -
        echo "deb https://packages.sury.org/php/ $(lsb_release -sc) main" | sudo tee /etc/apt/sources.list.d/php.list
        sudo apt-get update
        sudo apt-get install -y php7.4 libapache2-mod-php7.4 php7.4-mysql php7.4-curl php7.4-gd php7.4-mbstring php7.4-xml php7.4-xmlrpc php7.4-zip

        And restart the Apache2 Webserver

        sudo systemctl restart apache2
      6. Alright, that’s done. Next step is to test things.
        sudo nano /var/www/html/test.php

        Enter this into the php file and press Control X and type Y to save and exit.

        <?php
        // Show all PHP information
        phpinfo();
        ?>
      7. Go to the IP address of the server you just created and type
        HTTP://IP ADDRESS/test.php
        

        If you see a PHP page with all sorts of data, you’re good. If not, go fix. Don’t ask me, I’m not there yet!

Loading

How to correct the Themler crash after updating to Joomla 3.9

If you updated to the latest version and the following error appears when you try to launch Themler please follow the instruction below:

Fatal error: require_once(): Failed opening required '[...]/administrator/includes/toolbar.php' (include_path='.:/usr/lib/php7.2') in [...]/templates/[theme_name]/app/index.php on line 25

  1. Open theme folder on the server: www/Joomla_directory/templates;
  2. open /app folder;
  3. open index.php for editing in any html or text editor;
  4. find the line
    require_once JPATH_BASE . DS . 'includes' . DS . 'toolbar.php';
    and replace it with the following line:
    $prefix = version_compare(JVERSION, '3.9', '>=') ? 'sub' : '';
    require_once JPATH_BASE . DS . 'includes' . DS . $prefix . 'toolbar.php';
  5. open /app/start folder;
  6. open data.php file for editing and follow step 4;
  7. open manifest.php file for editing and follow step 4;
  8. start Themler.

Loading

Force a file scan in NextCloud

To force a scan after a manual file transfer, for example, can be done as follows:

sudo -u [USER] php occ files:scan --all

for example:

sudo -u john php occ files:scan --all
Usage:
  files:scan [-p|--path="..."] [-q|--quiet] [-v|vv|vvv --verbose] [--all]
  [user_id1] ... [user_idN]

Arguments:
  user_id               will rescan all files of the given user(s)

Options:
  --path                limit rescan to the user/path given
  --all                 will rescan all files of all known users
  --quiet               suppress any output
  --verbose             files and directories being processed are shown
                        additionally during scanning
  --unscanned           scan only previously unscanned files

Source: https://docs.nextcloud.com/server/15/admin_manual/configuration_server/occ_command.html

Loading

Upgrade to PHP7.2

  1. apt-get install apt-transport-https lsb-release ca-certificates
  2. wget -O /etc/apt/trusted.gpg.d/php.gpg https://packages.sury.org/php/apt.gpg
  3. sh -c 'echo "deb https://packages.sury.org/php/ $(lsb_release -sc) main" > /etc/apt/sources.list.d/php.list'
  4. apt-get update
  5. apt-get install php7.2-common libapache2-mod-php7.2 php7.2-cgi php7.2-cli php7.2-phpdbg php7.2-fpm libphp7.2-embed php7.2-dev php7.2-curl php7.2-gd php7.2-imap php7.2-interbase php7.2-intl php7.2-ldap php7.2-readline php7.2-odbc php7.2-pgsql php7.2-pspell php7.2-recode php7.2-tidy php7.2-xmlrpc php7.2 php7.2-json php-all-dev php7.2-sybase php7.2-sqlite3 php7.2-mysql php7.2-opcache php7.2-bz2 libapache2-mod-php7.2 php7.2-mbstring php7.2-pdo php7.2-dom php7.2-enchant php7.2-gmp php7.2-soap php7.2-zip

In case you need mcrypt:

  1. apt-get -y install gcc make autoconf libc-dev pkg-config
  2. apt-get -y install libmcrypt-dev
  3. apt-get install php7.2-dev
  4. pecl install mcrypt-1.0.1

    When you see the prompt “libmcrypt prefix? [autodetect] :” -> Press [Enter] to autodetect

  5. php -i | grep "mcrypt"

  1. a2dismod php7.0

    *or whatever version you are upgrading from)

  2. a2enmod php7.2

 

Loading