From Heston Blumenthal’s In Search of Perfection, Season 1, Episode 4 named “Fish and Chips”.
Buy floury potatoes with a dry content matter of around 22%. This is very hard to measure, so just see what you can get.
Peel the potatoes and cut them into thick(ish) and equal parts. Perhaps a centimetre thick.
Wash the chips and place them in a pot with lightly salted water.
Cook them on simmering. Simmer to as close as possible to breaking up. If they break up, you will have potato soup. You should be able to scoop them out, and they need to look a bit fluffy. Cooking time is somewhere between 10-15 minutes.
Place the chips carefully on a cake rack and place the rack in the refrigerator for 30-45 minutes. You want the moisture content to be as low as possible, and the dry air in a refrigerator helps greatly with that.
The first fry is at 130C for about 5 minutes. The outside of the chip should get a bit of colour by then.
Take the chips out, place them on the cake rack again and place the rack in the refrigerator for another 30-45 minutes.
The second and last fry is at 190C for about 3-5 minutes. Look at the colour and test while frying the chips.
Original post: https://3xn.nl/projects/2023/09/20/crude-solution-to-ban-bots-by-their-user-agent/
I’ve very much simplified the script that instantly redirects unwanted traffic away from the server. Currently, I am using a very cheap VPS to receive all that traffic.
Here ya go:
<?php
// CC-BY-NC (2023)
// Author: FoxSan - fox@cytag.nl
// This is a functional but dirty hack to block bots, spiders and indexers by looking at the HTTP USER AGENT.
// Traffic that meets the conditions is being yeeted away to any place of your choice.
//////////////////////////////////////////////////////////////
// Emergency bypass
// goto end;
//////////////////////////////////////////////////////////////
// attempt to basically just yeet all bots to another website
$targetURL = "https://DOMAIN.TLD/SUB/";
// Function to check if the user agent appears to be a bot or spider
function isBot()
{
$user_agent = $_SERVER['HTTP_USER_AGENT'];
$bot_keywords = ['bytespider', 'amazonbot', 'MJ12bot', 'YandexBot', 'SemrushBot', 'dotbot', 'AspiegelBot', 'DataForSeoBot', 'DotBot', 'Pinterestbot', 'PetalBot', 'HeadlessChrome', 'GPTBot', 'Sogou', 'ALittle Client', 'fidget-spinner-bot', 'intelx.io_bot', 'Mediatoolkitbot', 'BLEXBot', 'AhrefsBot'];
foreach ($bot_keywords as $keyword) {
if (stripos($user_agent, $keyword) !== false) {
return true;
}
}
return false;
}
// Check if the visitor is a bot or spider
if (isBot()) {
// yeet
header("Location: $targetURL");
// Exit to prevent further processing
exit;
}
end:
// If the visitor is not a bot, spider, or crawler, continue with your website code.
//////////////////////////////////////////////////////////////////////
?>
Here’s a list of stuff that I have in my .htaccess files on various websites.
I want to work on my website, but any other visitor should be booted to another website so I can work in peace. Sidenote: It's forever since I last used this, so it might work. Or not.
---
# YOUR IP address goes here:
RewriteCond %{REMOTE_ADDR} !^000\.000\.000\.000$
# And provides you access to:
RewriteCond %{REQUEST_URI} !^https://DOMAIN.TLD$ [NC]
# Fine, go have all the media as well
RewriteCond %{REQUEST_URI} !\.(jpg|jpeg|png|gif|svg|swf|css|ico|js)$ [NC]
# Any other visitor can go visit the following website:
RewriteRule .* https://DOMAIN.TLD/ [R=302,L]
# Hey, no viewing access to this file
<FilesMatch "^.ht">
Order deny,allow
Deny from all
</FilesMatch>
# Disable Server Signature
ServerSignature Off
# SSL all the things!
RewriteCond %{HTTPS} !=on
RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1 [R,L]
# No WWW
RewriteCond %{HTTP_HOST} ^www\.DOMAIN\.TLD$
RewriteRule ^/?$ "https\:\/\/DOMAIN\.TLD\/" [R=301,L]
# Do we like Symlinks? Yeah we do.
Options +FollowSymlinks
# No open directories or directory listings. What is this... 1998?
Options All -Indexes
IndexIgnore *
# Rewrite rules to block out some common exploits.
RewriteCond %{QUERY_STRING} base64_encode[^(]*\([^)]*\) [OR]
RewriteCond %{QUERY_STRING} (<|%3C)([^s]*s)+cript.*(>|%3E) [NC,OR]
RewriteCond %{QUERY_STRING} GLOBALS(=|\[|\%[0-9A-Z]{0,2}) [OR]
RewriteCond %{QUERY_STRING} _REQUEST(=|\[|\%[0-9A-Z]{0,2})
RewriteRule .* index.php [F]
# PHP doohickies
php_flag register_globals off
php_flag safe_mode off
php_flag allow_url_fopen off
php_flag display_errors off
php_value session.save_path '/tmp'
php_value disable_functions "exec,passthru,shell_exec,system,curl_multi_exec,show_source,eval"
# File Injection Protection, or a code-condom. What.
RewriteCond %{REQUEST_METHOD} GET
RewriteCond %{QUERY_STRING} [a-zA-Z0-9_]=http:// [OR]
RewriteCond %{QUERY_STRING} [a-zA-Z0-9_]=(\.\.//?)+ [OR]
RewriteCond %{QUERY_STRING} [a-zA-Z0-9_]=/([a-z0-9_.]//?)+ [NC]
RewriteRule .* - [F]
# /proc/self/environ? Go away!
RewriteCond %{QUERY_STRING} proc/self/environ [NC,OR]
# Disallow Access To Sensitive Files. Enter your own file names.
RewriteRule ^(htaccess.txt|configuration.php(-dist)?|joomla.xml|README.txt|web.config.txt|CONTRIBUTING.md|phpunit.xml.dist|plugin_googlemap2_proxy.php)$ - [F]
# Don't allow any pages to be framed - Defends against CSRF
<IfModule mod_headers.c>
Header set X-Frame-Options SAMEORIGIN
</IfModule>
# Uh. I forgot.
<IfModule mod_autoindex.c>
IndexIgnore *
</IfModule>
# NO SNIFFYWIFFY OwO
<IfModule mod_headers.c>
Header always set X-Content-Type-Options "nosniff"
</IfModule>
# NEEDS TESTING
# Turn on IE8-IE9 XSS prevention tools
#Header set X-XSS-Protection "1; mode=block"
# NEEDS TESTING TOO
# Only allow JavaScript from the same domain to be run.
# Don't allow inline JavaScript to run.
#Header set X-Content-Security-Policy "allow 'self';"
# Example if you don't like Russia and Turkey (Optional A1 is to block anonymous proxies)
RewriteCond %{ENV:GEOIP_COUNTRY_CODE} ^(RU|TR)$
RewriteRule .* https://DOMAIN.TLD/directorywithindexdothtml/ [R=302,L]
Okay, this is a very crude way to block bots, spiders and crawlers by their user-agent, but so far, this has been very, very efficient.
Even when one chooses ” yes “, the question will be repeated. This is not a problem, because no one in their right mind is going to add “bot”, “spider” or “crawler” as their user-agent.
So here’s the PHP script that I rammed into a certain website to prevent it from being DDOSsed by (malicious) bots.
<?php
// CC-BY-NC (2023)
// Author: FoxSan - fox@cytag.nl
// This is a functional but dirty hack to block bots, spiders and indexers by looking at the HTTP USER AGENT.
// The form is, iirc, not even working, but that's fine if you only want human visitors.
// It can also throw a 403, but the effect is the same.
////////////////////////////////////////////////////////////////////////////////
// Emergency bypass
// goto end;
////////////////////////////////////////////////////////////////////////////////
// Function to check if the user agent appears to be a bot or spider.
// Enter the bots you would like to block in a list as shown below.
function isBot()
{
$user_agent = $_SERVER["HTTP_USER_AGENT"];
$bot_keywords = ['bytespider',
'amazonbot',
'MJ12bot',
'YandexBot',
'SemrushBot',
'dotbot',
'AspiegelBot',
'DataForSeoBot',
'DotBot',
'Pinterestbot',
'PetalBot',
'HeadlessChrome',
'AhrefsBot'];
foreach ($bot_keywords as $keyword) {
if (stripos($user_agent, $keyword) !== false) {
return true;
}
}
return false;
}
// Check if the visitor is a bot or spider
if (isBot()) {
// This visitor appears to be a bot or spider, so display a choice.
// Check if the choice form is submitted
if (isset($_POST["submit"])) {
// Check the choice made by the visitor
$choice = isset($_POST["choice"]) ? $_POST["choice"] : "";
if ($choice === "yes") {
// User selected "Yes," block access
echo "Access denied. If you believe this is an error, please contact us by writing the word [MAILBOX] before the at sign, followed by [DOMAIN.TLD]";
} elseif ($choice === "no") {
// User selected "No," proceed to end
goto end;
}
} else {
// Output the message to the user and make the choice mandatory
echo "Your user agent suggests you might be a bot, spider, or crawler. Are you one of these three?";
// Output the radio button choices within a form
echo '</p>
<form method="post" action="">';
echo ' <label><input type="radio" name="choice" value="yes" required>Yes</label>';
echo ' <label><input type="radio" name="choice" value="no">No</label>';
echo ' <button type="submit" name="submit">Proceed</button>';
echo "</form>
<p>";
}
// Exit to prevent further processing
exit();
}
end:
// Original website code starts from here.
/////////////////////////////////////////////////////////////
?>
I have installed CloudPanel and the new website caused a “Too many redirects” bug. This is because my SSL certificates are controlled by a proxy and this can cause some confusion between the systems. Also, because CloudPanel installs its own certificates.
This application can also install a Let’s Encrypt certificate, but this works only in more conventional systems. Mine is going through a DNS to a Proxy that listens to a certain IP address and that proxy redirects the request to a Virtual Machine on one of my servers.
So, here is my, probably unconventional method of disabling the SSL certificates on my CloudPanel installation:
Done! Your website should now say “Hello world :-)”
You can see that I have disabled the listen to port 443, the certificate keys, the forced https and the path to the keys. I chose to switch off the forced HTTP, because my proxy is already taking care of that.
This post is subject to change, but this helps you along your way!
Updating Mastodon and creating backups are important steps to ensure the security and stability of your instance. Here’s a comprehensive tutorial on how to update Mastodon, including making backups of the database and assets:
Note 1: Always perform updates on a test/staging instance before applying them to your live instance. This tutorial assumes you have some basic command line and server administration knowledge.
Note 2: If you made alterations to your files and want to update to a new branch, like v4.2.0, don’t forget to stash your files first. DO THIS AT YOUR OWN RISK.
cd /home/mastodon/live
git stash
You will not have an error message any more when entering: I currently do not know how to put the stashed files back, this will be found out later.
git checkout v4.2.0
Note 3: If you switch to a new branch and version, like 4.2.0, you might run into an error stating the following:
rbenv: version '3.2.2' is not installed (set by /home/mastodon/live/.ruby-version)
Take the following steps to solve this problem: (Type the version you need where it says x.x.x)
rbenv install x.x.x
If that fails, try the following command:
brew update && brew upgrade ruby-build
Followed by: (Type the version you need where it says x.x.x)
rbenv install x.x.x
OPTIONAL: If bundler or rails fails to work, try the following commands:
gem install bundler
gem install rails
Click here for the backup steps. It basically comes down to a Database dump, a settings file backup and a Redis dump. If you wish to backup your assets like images and stuff (User-uploaded files), backup the folder named “public/system”. Keep in mind that this folder can be rather large. Actually, it can become rather massive.
After a good 90 minutes, I gave up on trying to show you how large the asset folder is. So beware if you are going to make a backup of it. Perhaps you can just skip the cache folder?
You can always check the folder size by using NCDU, for which you can find the manual here. Also, installations may vary, but this is an example of my instance.
Upgrade procedure.
su - mastodon
cd /home/mastodon/live
git fetch --tags
git checkout [type the most recent version here, starting with the letter v. For example; v4.0.1
Command example: git checkout v4.0.1
If you don’t want to restart your server, use the following commands instead of “reboot now”:
exit
systemctl restart mastodon-sidekiq
systemctl reload mastodon-web
The reload operation is a zero-downtime restart, also called a “phased restart”. As such, Mastodon upgrades usually do not require any advance notice to users about planned downtime. In rare cases, you can use the restart operation instead, but there will be a (short) felt interruption of service for your users.
The streaming API server is also updated and requires a restart; doing so will result in all connected clients being disconnected, which can increase the load on your server:
systemctl restart mastodon-streaming
Done!
PS: When I updated my instance from 4.1.9 to 4.2.0, a lot of warnings flew by, and this is, in my experience, not a problem as the instance is working perfectly.
I got a question about whether it was better to have a 5x6m parking garage or a 6×6 one. Well, it depends on your budget, of course. It also comes down to how comfortable you want your parking to be and if you want additional storage space.
One (5,7m) door is expensive, and when it breaks down, you won’t be able to get in (easily) any more. One 3,5m door requires careful parking but is the cheapest option. Personally, I’d go for two 2.6m wide doors, so both cars can be parked comfortably. It also allows for a 20x20cm centre support beam, which can make your construction lighter and more economical.
If you want more storage space, you can make it 6m deep instead of 5m, but this comes at an extra cost. It doesn’t look like much, but it is an additional 2m of wall and 6m² of floor and (flat) roof.
