Just digging deeper in automating thing, so I made two scripts:
backup_cron.sh
#!/bin/bash
# Define the path to the backup script
BACKUP_SCRIPT="/backup_mysql.sh"
# Define the cron job command to run the backup script once a week (every Sunday at midnight)
CRON_COMMAND="0 0 * * 0 $BACKUP_SCRIPT"
# Add the cron job to the crontab
(crontab -l 2>/dev/null; echo "$CRON_COMMAND") | crontab -
echo "Backup cron job set up successfully."
And the script that executes the backup every Sunday:
backup_mysql.sh
#!/bin/bash
# Define the directory where backup files will be stored
BACKUP_DIR="/var/backups/mysql"
# Ensure the backup directory exists
mkdir -p "$BACKUP_DIR"
# Define the filename for the backup file (include date in the filename)
BACKUP_FILE="$BACKUP_DIR/mysql_backup_$(date +%Y-%m-%d_%H-%M-%S).sql"
# Define the compressed filename
COMPRESSED_FILE="$BACKUP_FILE.gz"
# Define MySQL username and password (replace with your MySQL credentials)
DB_USER="USERNAME"
DB_PASS="PASSWORD"
# Dump all databases into a single SQL file
mysqldump -u "$DB_USER" -p"$DB_PASS" --all-databases > "$BACKUP_FILE"
# Add permissions to the backup file
chmod 600 "$BACKUP_FILE"
# Compress the backup file
gzip "$BACKUP_FILE"
echo "Database backup completed. Backup stored in: $COMPRESSED_FILE"
Currently successfullyneeds more testing on a Debian OS 11.5.0 installation. (I now remember why I didn’t want to touch this stuff any more. But oh well, I may as well completely overhaul this script again. The current one does work, but not well enough for me)
Step-by-step is functioning, will now wipe the server to start from scratch again. Also, the previous article will now be deleted.
Note:
If you are using this on a fresh Debian install, SUDO might not have been installed, nor has the user (so not the root) been add to SUDO. This is how you do that if you are logged in as user:
You need to exit twice so you log out as root and as user. After that, you need to log back in as user so you can start using sudo from now on.
Let’s begin! You can name the script something like
setup.sh
#!/bin/bash
# Update system
sudo apt update
sudo apt upgrade -y
# Install necessary packages
sudo apt install gnupg nginx mariadb-server php7.4-fpm php7.4-mysql php7.4-curl php7.4-gd php7.4-mbstring php7.4-xml php7.4-zip ssh ufw nano fail2ban curl wget sudo openssl net-tools unzip -y
# Add repository key for PHP 8.0
sudo apt install software-properties-common -y
sudo wget -O /etc/apt/trusted.gpg.d/php.gpg https://packages.sury.org/php/apt.gpg
# Add repository for PHP 8.0
echo "deb https://packages.sury.org/php/ $(lsb_release -sc) main" | sudo tee /etc/apt/sources.list.d/php.list
sudo apt update
# Install PHP 8.0 and extensions
sudo apt install php8.0-fpm php8.0-mysql php8.0-curl php8.0-gd php8.0-mbstring php8.0-xml php8.0-zip -y
# Configure UFW to allow web traffic and SSH
sudo ufw allow OpenSSH
sudo ufw allow 'Nginx Full'
sudo ufw allow 3306/tcp
# Enable UFW
sudo ufw enable
# Configure automatic security updates
echo 'APT::Periodic::Update-Package-Lists "1";' | sudo tee -a /etc/apt/apt.conf.d/10periodic
echo 'APT::Periodic::Download-Upgradeable-Packages "1";' | sudo tee -a /etc/apt/apt.conf.d/10periodic
echo 'APT::Periodic::AutocleanInterval "7";' | sudo tee -a /etc/apt/apt.conf.d/10periodic
echo 'APT::Periodic::Unattended-Upgrade "1";' | sudo tee -a /etc/apt/apt.conf.d/20auto-upgrades
# Disable root login via SSH
sudo sed -i 's/#PermitRootLogin prohibit-password/PermitRootLogin no/' /etc/ssh/sshd_config
sudo systemctl restart sshd
# Generate SSH key pairs
ssh-keygen -t rsa -b 4096
# Configure Nginx
sudo systemctl start nginx
sudo systemctl enable nginx
# Configure PHP 7.4
sudo systemctl start php7.4-fpm
sudo systemctl enable php7.4-fpm
# Configure PHP 8.0
sudo systemctl start php8.0-fpm
sudo systemctl enable php8.0-fpm
# Configure MariaDB
sudo mysql_secure_installation
# Allow Nginx to use PHP
sudo sed -i 's/;cgi.fix_pathinfo=1/cgi.fix_pathinfo=0/' /etc/php/7.4/fpm/php.ini
sudo sed -i 's/;cgi.fix_pathinfo=1/cgi.fix_pathinfo=0/' /etc/php/8.0/fpm/php.ini
# Restart Nginx and PHP services
sudo systemctl restart nginx
sudo systemctl restart php7.4-fpm
sudo systemctl restart php8.0-fpm
# Create a temporary file to store server and PHP info
tmpfile=$(mktemp)
# Get server info
uname -a > "$tmpfile"
# Get PHP version
php -v >> "$tmpfile"
# Append server and PHP info to index.html
cat "$tmpfile" | sudo tee /var/www/html/index.html >/dev/null
# Remove temporary file
rm "$tmpfile"
# Install adminer
sudo apt install adminer -y
sudo ln -s /usr/share/adminer/adminer.php /var/www/html/adminer.php
sudo chown -R www-data:www-data /usr/share/adminer /var/www/html/adminer.php
# Set the permissions correct for www
sudo chmod g+w /var/www
sudo chown -R :www-data www
sudo chmod g+s www
change_ownership() {
sudo chown -R "$1":www-data /var/www
echo "Ownership of the www folder has been set to $1:www-data."
}
# Loop until a valid username is provided
while true; do
# Prompt the user to enter the desired username
read -p "Enter the username for permissions: " username
# Check if the username provided exists
if id "$username" &>/dev/null; then
change_ownership "$username"
break # Exit the loop if a valid username is provided
else
echo "Error: User $username does not exist."
fi
done
echo "LEMP setup completed successfully."
echo "Check if Adminer is working by opening a browser, and entering http://IP_address/adminer.php in the address bar"
To make the script executable:
sudo chmod +x setup.sh
and run it with
sudo ./setup.sh
When things like PHP fail to work (becuase y’know… PHP)
These are the files that have to look this way to make PHP happen.
sudo nano /etc/nginx/sites-available/default
##
# You should look at the following URL's in order to grasp a solid understanding
# of Nginx configuration files in order to fully unleash the power of Nginx.
# https://www.nginx.com/resources/wiki/start/
# https://www.nginx.com/resources/wiki/start/topics/tutorials/config_pitfalls/
# https://wiki.debian.org/Nginx/DirectoryStructure
#
# In most cases, administrators will remove this file from sites-enabled/ and
# leave it as reference inside of sites-available where it will continue to be
# updated by the nginx packaging team.
#
# This file will automatically load configuration files provided by other
# applications, such as Drupal or WordPress. These applications will be made
# available underneath a path with that package name, such as /drupal8.
#
# Please see /usr/share/doc/nginx-doc/examples/ for more detailed examples.
##
# Default server configuration
#
server {
listen 80 default_server;
listen [::]:80 default_server;
# SSL configuration
#
# listen 443 ssl default_server;
# listen [::]:443 ssl default_server;
#
# Note: You should disable gzip for SSL traffic.
# See: https://bugs.debian.org/773332
#
# Read up on ssl_ciphers to ensure a secure configuration.
# See: https://bugs.debian.org/765782
#
# Self signed certs generated by the ssl-cert package
# Don't use them in a production server!
#
# include snippets/snakeoil.conf;
root /var/www/html;
# Add index.php to the list if you are using PHP
index index.php index.html index.htm index.nginx-debian.html;
server_name _;
location / {
# First attempt to serve request as file, then
# as directory, then fall back to displaying a 404.
try_files $uri $uri/ =404;
}
# pass PHP scripts to FastCGI server
#
location ~ \.php$ {
include snippets/fastcgi-php.conf;
# # With php-fpm (or other unix sockets):
fastcgi_pass unix:/run/php/php8.0-fpm.sock;
#non-standard addition
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
include fastcgi_params;
}
# # With php-cgi (or other tcp sockets):
# fastcgi_pass 127.0.0.1:9000;
#}
# deny access to .htaccess files, if Apache's document root
# concurs with nginx's one
#
#location ~ /\.ht {
# deny all;
#}
}
# Virtual Host configuration for example.com
#
# You can move that to a different file under sites-available/ and symlink that
# to sites-enabled/ to enable it.
#
#server {
# listen 80;
# listen [::]:80;
#
# server_name example.com;
#
# root /var/www/example.com;
# index index.html;
#
# location / {
# try_files $uri $uri/ =404;
# }
#}
You can give your Debian installation a static IP address by following these steps:
1. Determine your network interface name: Use the following command to determine the name of your network interface:
ip addr
This will display information about all of your network interfaces. Look for the interface that you want to configure with a fixed IP address. The interface name will be listed on the left-hand side of the output (e.g. eth0, enp0s3, etc.)
2. Edit the network configuration file: Use a text editor to edit the /etc/network/interfaces file. For example, you can use the nano editor by running the following command:
sudo nano /etc/network/interfaces
3. Configure the network interface: Add the following lines to the file, replacing the interface name and IP address with your own values:
For example, if your network interface name is “eth0” and you want to set the IP address to “192.168.0.10”, the subnet mask to “255.255.255.0”, the default gateway to “192.168.0.1”, and the DNS server to “8.8.8.8”, the configuration would look like this:
4. Save and close the file: Press Ctrl+O to save the file, and then press Ctrl+X to close the editor.
5. Restart the networking service: Use the following command to restart the networking service and apply the changes:
sudo systemctl restart networking
After you’ve completed these steps, your Debian installation should have a fixed IP address. You can verify the configuration by using the “ip addr” command again and looking for the interface that you configured.
More info here: https://wiki.debian.org/NetworkConfiguration
Thanks to https://fosstodon.org/@HankB for the tips and better choice of words!
So a new release came out and it is important to get this update as soon as possible! This manual is a transcript of the way that I have updated my Mastodon instance. Please make sure you make proper backups and use your brain while updating things.
A guide to making a Mastodon backup can be found here.
Linux flavour: Debian
Update from: 4.2.xx
Log into your server
su - mastodon
cd /home/mastodon/live
git fetch --tags
git checkout [type the most recent version here, starting with the letter v. For example; v4.2.5
git checkout v4.2.10
If you get a ruby version error, please see bottom of this article for a fix! bundle install
yarn install
RAILS_ENV=production bundle exec rails db:migrate
#NOTE: You might get a ruby error which then suggests you to enter the command "bundle install". Do that and then run the RAILS command again.
My system was unable to find the required v3.2.3 of Ruby and I have fixed this by doing the following steps:
Please make sure that your path is correct.
git -C ~/.rbenv/plugins/ruby-build pull
rbenv install 3.2.3
*WAIT TILL DONE* (it may take a little while)
To check all the installed versions type:
rbenv versions
To set v3.2.3 as the global version, type:
rbenv global 3.2.3
To double-check the active, installed version, type:
rbenv versions
Done!
This manual is a transcript of the way that I have updated my Mastodon instance. Please make sure you make proper backups and use your brain while updating things.
Original post: https://3xn.nl/projects/2023/09/20/crude-solution-to-ban-bots-by-their-user-agent/
I’ve very much simplified the script that instantly redirects unwanted traffic away from the server. Currently, I am using a very cheap VPS to receive all that traffic.
Here ya go:
<?php
// CC-BY-NC (2023)
// Author: FoxSan - fox@cytag.nl
// This is a functional but dirty hack to block bots, spiders and indexers by looking at the HTTP USER AGENT.
// Traffic that meets the conditions is being yeeted away to any place of your choice.
//////////////////////////////////////////////////////////////
// Emergency bypass
// goto end;
//////////////////////////////////////////////////////////////
// attempt to basically just yeet all bots to another website
$targetURL = "https://DOMAIN.TLD/SUB/";
// Function to check if the user agent appears to be a bot or spider
function isBot()
{
$user_agent = $_SERVER['HTTP_USER_AGENT'];
$bot_keywords = ['bytespider', 'amazonbot', 'MJ12bot', 'YandexBot', 'SemrushBot', 'dotbot', 'AspiegelBot', 'DataForSeoBot', 'DotBot', 'Pinterestbot', 'PetalBot', 'HeadlessChrome', 'GPTBot', 'Sogou', 'ALittle Client', 'fidget-spinner-bot', 'intelx.io_bot', 'Mediatoolkitbot', 'BLEXBot', 'AhrefsBot'];
foreach ($bot_keywords as $keyword) {
if (stripos($user_agent, $keyword) !== false) {
return true;
}
}
return false;
}
// Check if the visitor is a bot or spider
if (isBot()) {
// yeet
header("Location: $targetURL");
// Exit to prevent further processing
exit;
}
end:
// If the visitor is not a bot, spider, or crawler, continue with your website code.
//////////////////////////////////////////////////////////////////////
?>
Here’s a list of stuff that I have in my .htaccess files on various websites.
I want to work on my website, but any other visitor should be booted to another website so I can work in peace. Sidenote: It's forever since I last used this, so it might work. Or not.
---
# YOUR IP address goes here:
RewriteCond %{REMOTE_ADDR} !^000\.000\.000\.000$
# And provides you access to:
RewriteCond %{REQUEST_URI} !^https://DOMAIN.TLD$ [NC]
# Fine, go have all the media as well
RewriteCond %{REQUEST_URI} !\.(jpg|jpeg|png|gif|svg|swf|css|ico|js)$ [NC]
# Any other visitor can go visit the following website:
RewriteRule .* https://DOMAIN.TLD/ [R=302,L]
# Hey, no viewing access to this file
<FilesMatch "^.ht">
Order deny,allow
Deny from all
</FilesMatch>
# Disable Server Signature
ServerSignature Off
# SSL all the things!
RewriteCond %{HTTPS} !=on
RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1 [R,L]
# No WWW
RewriteCond %{HTTP_HOST} ^www\.DOMAIN\.TLD$
RewriteRule ^/?$ "https\:\/\/DOMAIN\.TLD\/" [R=301,L]
# Do we like Symlinks? Yeah we do.
Options +FollowSymlinks
# No open directories or directory listings. What is this... 1998?
Options All -Indexes
IndexIgnore *
# Rewrite rules to block out some common exploits.
RewriteCond %{QUERY_STRING} base64_encode[^(]*\([^)]*\) [OR]
RewriteCond %{QUERY_STRING} (<|%3C)([^s]*s)+cript.*(>|%3E) [NC,OR]
RewriteCond %{QUERY_STRING} GLOBALS(=|\[|\%[0-9A-Z]{0,2}) [OR]
RewriteCond %{QUERY_STRING} _REQUEST(=|\[|\%[0-9A-Z]{0,2})
RewriteRule .* index.php [F]
# PHP doohickies
php_flag register_globals off
php_flag safe_mode off
php_flag allow_url_fopen off
php_flag display_errors off
php_value session.save_path '/tmp'
php_value disable_functions "exec,passthru,shell_exec,system,curl_multi_exec,show_source,eval"
# File Injection Protection, or a code-condom. What.
RewriteCond %{REQUEST_METHOD} GET
RewriteCond %{QUERY_STRING} [a-zA-Z0-9_]=http:// [OR]
RewriteCond %{QUERY_STRING} [a-zA-Z0-9_]=(\.\.//?)+ [OR]
RewriteCond %{QUERY_STRING} [a-zA-Z0-9_]=/([a-z0-9_.]//?)+ [NC]
RewriteRule .* - [F]
# /proc/self/environ? Go away!
RewriteCond %{QUERY_STRING} proc/self/environ [NC,OR]
# Disallow Access To Sensitive Files. Enter your own file names.
RewriteRule ^(htaccess.txt|configuration.php(-dist)?|joomla.xml|README.txt|web.config.txt|CONTRIBUTING.md|phpunit.xml.dist|plugin_googlemap2_proxy.php)$ - [F]
# Don't allow any pages to be framed - Defends against CSRF
<IfModule mod_headers.c>
Header set X-Frame-Options SAMEORIGIN
</IfModule>
# Uh. I forgot.
<IfModule mod_autoindex.c>
IndexIgnore *
</IfModule>
# NO SNIFFYWIFFY OwO
<IfModule mod_headers.c>
Header always set X-Content-Type-Options "nosniff"
</IfModule>
# NEEDS TESTING
# Turn on IE8-IE9 XSS prevention tools
#Header set X-XSS-Protection "1; mode=block"
# NEEDS TESTING TOO
# Only allow JavaScript from the same domain to be run.
# Don't allow inline JavaScript to run.
#Header set X-Content-Security-Policy "allow 'self';"
# Example if you don't like Russia and Turkey (Optional A1 is to block anonymous proxies)
RewriteCond %{ENV:GEOIP_COUNTRY_CODE} ^(RU|TR)$
RewriteRule .* https://DOMAIN.TLD/directorywithindexdothtml/ [R=302,L]
I have installed CloudPanel and the new website caused a “Too many redirects” bug. This is because my SSL certificates are controlled by a proxy and this can cause some confusion between the systems. Also, because CloudPanel installs its own certificates.
This application can also install a Let’s Encrypt certificate, but this works only in more conventional systems. Mine is going through a DNS to a Proxy that listens to a certain IP address and that proxy redirects the request to a Virtual Machine on one of my servers.
So, here is my, probably unconventional method of disabling the SSL certificates on my CloudPanel installation:
Done! Your website should now say “Hello world :-)”
You can see that I have disabled the listen to port 443, the certificate keys, the forced https and the path to the keys. I chose to switch off the forced HTTP, because my proxy is already taking care of that.
This post is subject to change, but this helps you along your way!
Updating Mastodon and creating backups are important steps to ensure the security and stability of your instance. Here’s a comprehensive tutorial on how to update Mastodon, including making backups of the database and assets:
Note 1: Always perform updates on a test/staging instance before applying them to your live instance. This tutorial assumes you have some basic command line and server administration knowledge.
Note 2: If you made alterations to your files and want to update to a new branch, like v4.2.0, don’t forget to stash your files first. DO THIS AT YOUR OWN RISK.
cd /home/mastodon/live
git stash
You will not have an error message any more when entering: I currently do not know how to put the stashed files back, this will be found out later.
git checkout v4.2.0
Note 3: If you switch to a new branch and version, like 4.2.0, you might run into an error stating the following:
rbenv: version '3.2.2' is not installed (set by /home/mastodon/live/.ruby-version)
Take the following steps to solve this problem: (Type the version you need where it says x.x.x)
rbenv install x.x.x
If that fails, try the following command:
brew update && brew upgrade ruby-build
Followed by: (Type the version you need where it says x.x.x)
rbenv install x.x.x
OPTIONAL: If bundler or rails fails to work, try the following commands:
gem install bundler
gem install rails
Click here for the backup steps. It basically comes down to a Database dump, a settings file backup and a Redis dump. If you wish to backup your assets like images and stuff (User-uploaded files), backup the folder named “public/system”. Keep in mind that this folder can be rather large. Actually, it can become rather massive.
After a good 90 minutes, I gave up on trying to show you how large the asset folder is. So beware if you are going to make a backup of it. Perhaps you can just skip the cache folder?
You can always check the folder size by using NCDU, for which you can find the manual here. Also, installations may vary, but this is an example of my instance.
Upgrade procedure.
su - mastodon
cd /home/mastodon/live
git fetch --tags
git checkout [type the most recent version here, starting with the letter v. For example; v4.0.1
Command example: git checkout v4.0.1
If you don’t want to restart your server, use the following commands instead of “reboot now”:
exit
systemctl restart mastodon-sidekiq
systemctl reload mastodon-web
The reload operation is a zero-downtime restart, also called a “phased restart”. As such, Mastodon upgrades usually do not require any advance notice to users about planned downtime. In rare cases, you can use the restart operation instead, but there will be a (short) felt interruption of service for your users.
The streaming API server is also updated and requires a restart; doing so will result in all connected clients being disconnected, which can increase the load on your server:
systemctl restart mastodon-streaming
Done!
PS: When I updated my instance from 4.1.9 to 4.2.0, a lot of warnings flew by, and this is, in my experience, not a problem as the instance is working perfectly.