Perfect Server: v19

This is the latest iteration of my perfect server. The biggest change is that this is updated to reflect the requirements of Debian 9.4 which are slightly different than previous versions. Also Certbot has changed somewhat.

 

The first step is to provision a new server. I use Digital Ocean. (Referral Coupon) I will be logged in as root for all of this since this is all stuff that needs to be done as root. If you don’t want to log in as root, you can instead use sudo at the beginning of each command.

(In previous versions, we needed to add new sources to install certbot. This is no longer necessary.)

 

apt-get update && apt-get upgrade

Now install all the packages we will need, and a few that everyone should really have which are no longer included by default;

apt-get -y install fail2ban apache2 php7.0 php-pear php7.0-mysql php7.0-mcrypt php7.0-mbstring libapache2-mod-php7.0 php7.0-curl screenfetch htop nload curl git unzip ntp mcrypt postfix mailutils php7.0-memcached mysql-server certbot python-certbot-apache man-db && a2enmod rewrite && service apache2 restart && mysql_secure_installation

You will no longer be prompted to create a mysql password when installing mysql-server. Now, you create it during the configuration command at the end of the line above.

Name Thyself

Now navigate to the virtualhost directory;

cd /etc/apache2/sites-available

Remove the default ssl virtualhost. We will be creating a new one instead.

rm default-ssl.conf

Rename the default virtualhost to the fqdn of the server. Example: server3.website.com. Note that this is not the fqdn of the site(s) we are hosting on the server.

mv 000-default.conf [fqdn].conf

Edit the default configuration file. We need to change the admin email to your email, and change the webroot to the webroot you want to use. I like /var/www

Restart Apache and apply the changes so it knows where the files are…

a2dissite 000-default && a2ensite [fqdn].conf && service apache2 restart

 

Free SSL

We already set up LetsEncrypt so now we just need to run their Certbot. Once the domains are set up and pointed to the server’s ip, along with a virtualhost being configured as shown above, all it takes is running Certbot which takes care of everything.

certbot –authenticator webroot –installer apache

Certbot will ask you to enter the webroot from the previous step for validation.

Make sure to choose the most secure options as specified by Certbot.

Now you have an SSL certificate installed!

 

Hardening Apache

Edit our default configuration file and comment out the DocumentRoot with a # sign at the beginning of the line. You will notice LetsEncrypt has added some redirect rules. We need to modify one of them. Look for the line that looks like this, and change it as shown;

RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,QSA,R=permanent]

Becomes… (Use your fqdn where it says [fqdn].)

RewriteRule ^ https://[fqdn]%{REQUEST_URI} [END,QSA,R=permanent]

Save that file and exit.

Now, let’s make sure no one can navigate to the IP of the server and access any virtualhosts that way.

cp [fqdn].conf [ip].conf && a2ensite [ip]

Where [ip] is the public ip of your server.

Now edit the newly created ssl virtualhost configuration file replace the default webroot with the one you want to use. It will be called something like /etc/apache2/sites-available/[fqdn]-le-ssl.conf. Add the following block within the virtualhost tag of the file and save it. Substitute the directory path with your chosen webroot path.

<Directory “/var/www”>
AuthType Basic
AuthName “Restricted Content”
AuthUserFile /etc/apache2/.htpasswd
Require valid-user
</Directory>

Lock it Down

Let’s create a credential set for our new virtualhost. This is sort of a catch-all for any domains we point here which are not yet set up.

htpasswd -c /etc/apache2/.htpasswd [username]

You will be prompted for a password. This is very bruteforceable. My best practice is to use a very high entropy strings for both the username and the password. Typically at least 64 bits of random base 64 for each.

Apply Changes

Now restart apache

 

service apache2 restart

Test our changes by navigating to the public ip of the server. You should be redirected to a https url with the fqdn of the server and prompted for a username and password. If this happens, everything so far has worked!

Administrative Tools

We will need to put some tools in here so we can administer the server.

PHPMyAdmin

This will allow us to manage the databases we will be creating on the server. Head over to their website and get the download link for the current version.

Navigate to our new secure DocumentRoot directory and download that link.

cd /var/www && wget [link]

Now unzip it and remove the zip file we downloaded.

unzip [file] && rm [file]

Now that we have a PHPMyAdmin directory in our secure virtualhost, we need to configure it. Luckily it can do that itself! Use this command and enter the mysql root password when prompted.

mysql -uroot -p < /[unzipped phpmyadmin folder]/sql/create_tables.sql

The last thing PHPMyAdmin needs is a secret string. Edit the config file config.sample.inc.php and save it as nano config.inc.php.

Make sure to add a random string where prompted at the top of the file.

Postfix Outbound-Mail Server

We need to edit the config files for postfix and change the interface to loopback-only like so. We already set up a firewall rule to block connections to port 25, but those rules can be changed by mistake, so this will be a good second line of defense to prevent public access to sending mail through our server, while allowing us to still use it locally.

nano /etc/postfix/main.cf

Find this line;

inet_interfaces = all

And change to;

inet_interfaces = 127.0.0.1

Now edit the email aliases;

nano /etc/aliases

At the end of the file, make sure there is a line that starts with root and ends with your email, like so;

root: email@domain.com

Save the file and exit. Then run newaliases to let Postfix apply the changes. Restarting Postfix is not enough because we changed the interfaces line in the config file. We need to stop and start it like so;

newaliases && postfix stop && postfix start

Now our sites will be able to send emails!

VPS Home

This is something simple I built which serves as a better index page for the secure virtual host and includes several helpful tools for diagnostic purposes. To try it out, run this command from the DocumentRoot directory.

wget https://raw.githubusercontent.com/cjtrowbridge/vps-home/master/index.php

PHPInfo

It’s helpful to be able to access details of the server’s php installation from this directory. I like to create a file called phpinfo.php which contains simply

<?php phpinfo();

Automatic Backups

Create a new file called /root/backup.sh and add the following to it. Make sure to replace the mysql password with yours.

#!/bin/bash

#deletes old backups
find /var/www/backups/www -mindepth 1 -mmin +$((60*24)) -delete
find /var/www/backups/mysql -mindepth 1 -mmin +$((60*24)) -delete

#backs up webs
cd /var/www/webs
for i in *
do
tar -czf “/var/www/backups/www/webs-$( date +’%Y-%m-%d’ )-$i.tar.gz” “/var/www/webs/$i”
done

#backs up databases
for i in `mysql -uroot -p[MySQL Root Password] -e “SHOW DATABASES;” | grep -v Database`; do
if [[ ( “$i” != “mysql” && “$i” != “phpmyadmin” && “$i” != “performance_schema” && “$i” != “information_schema” ) ]]
then
mysqldump -c -uroot -p[MySQL Root Password] ${i} | gzip > /var/www/backups/mysql/mysql.$( date +’%Y-%m-%d’ ).${i}.sql.gz
fi
done

 

#fix permissions just in case they changed for some reason

chmod 644 /var/www/webs -R
find /var/www/webs/ -type d -exec chmod 750 {} +
find /var/www/webs/ -type f -exec chmod 640 {} +
chown www-data:www-data /var/www/webs -R

Now edit the crontab with nano /etc/crontab and add this line. This will automatically run that script every day at 8pm.

0 20 * * * root /root/backup.sh > /dev/null 2>&1

Make sure to give the script permission to execute.

chmod +x backup.sh

Offsite Backups

The system I have used for this is no longer available. Will update when I decide on a new system.

Migrating Sites In

Move over the files for all the sites you want to host into individual directories in the /var/www/webs directory.

Now navigate to your virtualhosts directory.

cd /etc/apache2/sites-available

We created a default virtualhost file for the server and named it [fqdn].conf. This was the fqdn of the server, but not the sites it will host. Now we want to create our first hosted site. Copy the default file we made to create a new virtualhost like so…

cp [server fqdn].conf [site fqdn].conf

You can use any naming convention you like, but managing dozens or hundreds of these will become impossible if you are not naming them clearly.

Next, we need to add some new things to this hosted site fqdn. Add a new line inside the virtualhost tag like this;

ServerName [site fqdn]

And change the line which has DocumentRoot to point to the directory for this hosted site. For example;

DocumentRoot /var/www/webs/[site fqdn]

Lastly add these two blocks at the end of the file.

<Directorymatch “^/.*/\.git/”>
Order deny,allow
Deny from all
</Directorymatch>

<Directory /var/www/webs/[site fqdn]>
Options FollowSymLinks
AllowOverride All
Require all granted
</Directory>

The first block will prevent anyone from navigating into a git repository and accessing sensitive data like credentials or from cloning the repository.

The second block will allow htaccess files or directory rewrites, and prevent directory listing. These are required changes if you want to host WordPress sites, and best practices all around.

Now we just need to enable these changes and make the site live with;

a2ensite [site fqdn] && service apache2 restart

From this point on, this new virtualhost can be copied to create new sites, rather than recreating each one from the original virtualhost file.