Technology may have changed in the meantime.
UPDATE:
My first book describes a more elaborate setup, including LDAP authentication, and a lot more (firewall, DNS, Apache, Nginx, databases, synchronisation of files, calendars and address books, etc.; including many examples for FreeBSD, Debian and CentOS).
en: Practical Internet Server Configuration
fr: Installation et configuration d’un serveur internet
nl: Configuratie van een internetserver
After having depended on my friends for a few years, for web and email hosting, I recently acquired a private server again. In this HOWTO I document the entire mail setup.
At the end of this article, I will have a mail server that does:
- Postfix MTA (Mail Transfer Agent)
- Dovecot IMAP (Internet Message Access Protocol)
- Let’s Encrypt (SSL certificates)
- Dovecot SASL (Simple Authentication and Security Layer)
- mail accounts (partly) separated from system accounts
- Dovecot LDA (Local Delivery Agent)
- Sieve / ManageSieve
- Greylisting
- RBLs (Real-time Blackhole Lists)
- Bogofilter (spam filter)
- ClamAV (virus scanner)
- SPF (Sender Policy Framework)
- DKIM (DomainKeys Identified Mail)
- and more…
Just so you know that there’s more to mail than just typing a text and hitting the Send button…
Currently my server only serves 1 user with 3 domains. But with the setup I describe here, it is extremely simple to upgrade that to many users running many domains.
This tutorial is not for the complete layman, but should be comprehensible for those who have played around with Linux and its services and command line a bit. I have tried to make it interesting and useful for both the beginning and the experienced sysadmin.
Corrections, clarifications, supplements and typos are welcomed at tuto.mailserver@ohreally.nl.
Spam is not, as you will understand after reading this article.
Table of Contents
License
This work is licensed under a
Creative Commons Attribution-ShareAlike 4.0 International License.
What that means, in short, is that you may copy, adapt and/or redistribute this text and the scripts and examples contained in it, but only if you give appropriate credit to the original author (me), and share your own contributions under the same or a compatible license.
To find out how to give appropriate credit, to find information about compatible licenses, or to read the full license text, follow this link.
This license does not, in any way, limit your rights to link to this article at www.ohreally.nl.
Warranty and responsibility
Before I discuss anything else, it is important that you know and understand that there is no warranty, and that I do not accept any responsibility for anything you do or omit on the basis of this documentation.
The below works for me, but that does not automatically mean it will work for you as well.
If you do decide to copy any or all of the actions I describe here, I strongly advise you to create backups where needed, and not to experiment on critical production servers.
About this document
Well, it’s a long read… Do not count on setting up your mail server in 1 or 2 hours.
Most other websites would split this tutorial up into 15-20 pages. But since my website doesn’t have any banners that are paid per view, I don’t have any reason to make you go back and forth. With the advantage that you can now use Ctrl-F to search through the entire tutorial, instead of just a chapter.
The setup I describe is suited for a small-ish mail server; think company or organisation (or family/friends). I think that once you pass 100-150 users, you may wish to make some changes; for example disconnect mail users from system users entirely.
But even for larger, or even much larger setups, there should be some valuable info below.
All files mentioned in this document should be created and/or edited as the root user, unless stated otherwise.
For commands, a root# prefix indicates that the command should be executed as root user, and a userx$ prefix indicates that the command should be executed as regular user. For example:
userx$ sudo bash root# ls /root/
If you’re inclined to include the root# or userx$ prefixes in the commands, you’re probably not ready for this tutorial, and should learn some Linux basics first.
This article was published on 19 November 2018. If you’re reading this after, say, 2023, you may want to bookmark it, and see if you can find something more recent first.
Background and preparation
Store your spam
This is the first step in your preparation.
You probably receive mail somewhere already. Configure your mail account and/or client so that no spam is thrown away. Instead put it all into a separate folder. We’ll be using this later to train the spam filter.
Operating system
The operating system I’ve chosen for my server is Gentoo Linux. For those wondering why, I’ll very briefly indicate my motivations.
I had 3 requirements:
- because of support for software I’d like to experiment with later, I needed it to run Linux and not *BSD, which I’d normally prefer for a server
- it had to be a rolling distribution
- it may not use systemd
and I had 1 wish:
- I would appreciate a software installation system like the ports collection on *BSD
Gentoo was the only OS that met my 3 requirements, and as a bonus met my 1 wish as well. So it wasn’t a difficult choice.
Even though I have 20-25 years of experience running and administering Linux and *BSD servers and desktops, this is the first time I use Gentoo.
An experienced sysadmin will have no problem translating the below instructions to any other Unix-like system.
Software
For the setup that I envision, all the software listed below needs to be installed. If, on your current system, software is installed that overlaps with the software we need (e.g. Sendmail is installed, while I say we need Postfix), I leave it up to you and your imagination to decide how to deal with that; obviously, it would be appreciated if you could document your choice and/or solution somewhere, for people who run into that same problem after you have solved it.
To fine tune software installations, Gentoo uses so-called USE flags. Below I describe the USE flags for the software I installed; if you’re on a system other than Gentoo, you will have to find out for yourself whether and how you need to fine tune the installation (and again: please document).
The USE flags you need may be stored in /etc/portage/package.use/*, in a simple text file per application.
mail-mta/postfix
The MTA (Mail Transfer Agent) is the software that receives the mails that are delivered to the server, and handles them further. This concerns the mails that are sent from elsewhere on the internet to me, but also the mails that I send to addresses elsewhere on the internet. The handling may consist of the invocation of a spam filter or virus scanner on the mail (for incoming mail) or, for example, authenticating and authorising me to use this server to send mail out on the internet (for outgoing mail).
The protocol handled by the MTA is called SMTP (Simple Mail Transfer Protocol) or SMTPS (the SSL encrypted version of SMTP).
I chose Postfix, because:
- after many years of use, I still don’t grok the Sendmail configuration
- the Qmail project seems to have died
- chasquid looks promising, but ebuilds for Gentoo are not available, yet
- online comparisons usually prefer Postfix to Exim, regarding performance and security
This is /etc/portage/package.use/mail_mta-postfix:
# Support Berkeley DB files. mail-mta/postfix berkdb # Have Dovecot authenticate users. mail-mta/postfix dovecot-sasl
net-mail/dovecot
There are 2 protocols that allow me to open the mails received by my server, on my pc or smartphone: IMAP (Internet Message Access Protocol) and POP3 (Post Office Protocol, version 3); sometimes also refered to as IMAPS and POP3S, the SSL encrypted versions of those protocols. Dovecot is an application that can present my mails to me, using those 2 protocols. I choose to only use IMAP, though; POP3 moves the mail from the server to the computer, while IMAP leaves the mail on the server, permitting one to reach it from multiple computers/devices.
I chose Dovecot, because I have been a satisfied Dovecot user for a long, long time, and I don’t really know any other IMAP servers. Apart from serving messages over IMAP and POP3, Dovecot also does:
- Dovecot SASL (Simple Authentication and Security Layer)
asks for my username and password when I want to read my mail, but can also be used by Postfix to ask for my username and password when I want to send mail (see /etc/portage/package.use/mail_mta-postfix above) - Dovecot LDA (Local Delivery Agent)
receives incoming mails from Postfix, and drops them in my mailbox - Sieve / ManageSieve
allows one to filter incoming mails, for example to automatically sort them into different folders
This is /etc/portage/package.use/net_mail-dovecot:
# Enable mail filtering. net-mail/dovecot managesieve net-mail/dovecot sieve
app-admin/sudo
Chances are, Sudo is already installed on your system. And if it isn’t, it should, and not only for this mail setup.
In the setup we’re doing now, sudo will be used to allow users to change their email passwords.
app-admin/sudo does not need any special USE flags for this mail server setup.
app-crypt/certbot-apache
Let’s Encrypt is an organisation that gives out free SSL certificates. We will be using those to encrypt the communication between the pc or smartphone, and the server. Certbot is an application that may be used to request, retrieve and install those certificates.
I chose certbot-apache, because I have the Apache web server installed. If Nginx is your web server of choice, then install certbot-nginx. And at the Let’s Encrypt website you’ll find even more solutions.
Below, I will only briefly glance at the retrieval of SSL certificates, to give you an idea of how it works. But I will not elaborate on it, because I don’t want to extend this text, which is very long already, with the installation and configuration of a web server. You run Linux, so I expect you to be able to configure a web server, or at least be able to find the information you need to do so.
app-crypt/certbot-apache does not need any special USE flags for this mail server setup.
mail-filter/postgrey
Emails travel from mail server to mail server: the sender delivers the mail to his mail server (or his provider’s mail server, or Gmail’s), and that mail server delivers the mail to the recipient’s mail server; there may even be more servers in between. This is the Simple Mail Transfer Protocol that I described above.
When, on delivery, the receiving mail server returns an error, the sending mail server will retry later. Spamming software, however, is often developed to discard any mails that are undeliverable for some reason; a retry for each undeliverable mail would take too much time.
Greylisting (between blacklisting or ‘reject’, and whitelisting or ‘accept’) is a technique that utilises this to fight spam: each incoming mail is rejected with a temporary failure, the first time it is presented. A legitimate, well-configured mail server will retry later, while spamming software gives up. The second time a same mail is presented, it is let through.
I don’t have exact figures, but the logs on my server tell me that a lot of spam presented to my server is successfully dropped using this technique. So these mails never reach my spam filter, which saves a whole lot of CPU cycles and memory, which may now be used for other things.
Postgrey is a Postfix ‘add-on’ to implement greylisting.
This is /etc/portage/package.use/mail_filter-postgrey:
# Include the taRgrey patch, to enable tarpitting. # http://k2net.hakuba.jp/targrey/index.en.html # https://en.wikipedia.org/wiki/Tarpit_%28networking%29 mail-filter/postgrey targrey
mail-filter/bogofilter
My spam filter of choice is Bogofilter. The only other project that is not dead or dying, or under-developed, seems to be Spamassassin, which I find way to heavy. And since I don’t need all it’s bells and whistles, I can save myself some CPU cycles and memory, and install Bogofilter.
Bogofilter is a learning, so-called Bayesian, spam filter. This means that it learns from the mails that are classified by the user as being spam or non-spam, so filtering improves with time, as Bogofilter adjusts itself more and more to your personal mailbox.
mail-filter/bogofilter does not need any special USE flags for this mail server setup.
app-antivirus/clamav
As I’ve been using Linux/*BSD exclusively for a really long time, I have never really worried about viruses or virus scanners. But because I’m documenting this setup for others (you!), who may be using Windows on their home pcs, I decided to add a virus scanner for completeness.
This is /etc/portage/package.use/app_antivirus-clamav:
# Enable the milter interface. app-antivirus/clamav milter
mail-filter/pypolicyd-spf
You’ve probably received spam sometimes, that had your own email address as the sender address. This is because one can put any sender address one wants on an email.
SPF (Sender Policy Framework) is a protocol that tries to fight this by enabling sysadmins to indicate in the DNS which mail server(s) may be used to send mail for a certain domain. The receiving mail server may then verify with a simple DNS request whether a mail for a certain sender originates from the correct mail server. If this is not the case, the sender address is likely to be forged, and the mail is likely to contain spam or a virus.
pypolicyd-spf is a Postfix ‘add-on’ to execute this verification.
mail-filter/pypolicyd-spf does not need any special USE flags for this mail server setup.
mail-filter/opendkim
DKIM (DomainKeys Identified Mail) is in part a bit comparable to SPF. The sending mail server adds a signature to the headers of the mail, which may be verified with a DNS request to the sender’s domain’s DNS server. Furthermore, this signature may be used to verify that the mail has not been tempered with between the originating server and the receiving server.
OpenDKIM is an open source implementation of the DKIM specification.
mail-filter/opendkim does not need any special USE flags for this mail server setup.
net-dns/bind
The ISC Bind DNS server. Where DNS stands for Domain Name Service.
I would have prefered to use djbdns, because of it’s extremely simple configuration, but that project seems to have passed on.
To tell the server for which domains it handles the mail, we do not need a DNS server; we will specify that in the MTA configuration. But we do need DNS to tell other mail servers where to deliver the mail for our domains. And to host the DNS TXT records for SPF and DKIM.
net-dns/bind does not need any special USE flags for this mail server setup.
Installation
So, all USE flags have been set. Time to install everything.
root# emerge -at \ mail-mta/postfix \ net-mail/dovecot \ app-crypt/certbot-apache \ mail-filter/postgrey \ mail-filter/bogofilter \ app-antivirus/clamav \ mail-filter/pypolicyd-spf \ mail-filter/opendkim \ net-dns/bind
(You can put all that on a single line; if you do, remove the backslashes. If you break it up into multiple lines, as I have, each line ends in backslash-enter, with no space in between).
More…
System account(s)
As said, my server currently only serves 1 user. In this documentation, that user will be called userx. However, this setup is easily extendable to more users, so pick as many names as you want.
I will not document the creation of users; if you don’t know how to create users on a Linux box, maybe you’re not quite ready to set up a mail server.
The user needs to have a home directory, to save Sieve scripts to.
Currently, the user also needs shell access, to be able to set/change their mail password. I’d advise you to only use SSH keys for shell access, and disable password authentication; this way the password can never be cracked (or copied in an internet cafe, or anything).
Someday I hope to create a web and/or email interface for changing the password. If someone beats me to it, though, I promise I won’t get mad; send me a link to the source and documentation, so I can link to it.
Domain name(s)
As said as well, I currently have 3 domains. In this documentation those will be example.com, example.net and example.org.
The fully qualified hostname for the server we’re setting up will be yellow.example.com (no particular reason; most sysadmins have a theme for their hostnames, so why not colours?). We’ll get to the other names for this server when I talk about configuring the DNS.
It’s not mandatory to use 3 domains, but you do need at least 1. If you set up this mail server to play with on your own local network, pick a domain name that ends in ‘.lan’ (Local Area Network), to make sure that you’ll never clash with domain names on the internet.
If the server is connected to the internet, and using officially registered domain names, it is up to you to make sure that the chosen domain names, or at least their MX records and the hostname, resolve to this server.
Directories
To begin with, we’ll create a directory /srv, if it doesn’t exist yet. Not every distribution includes this directory, but it is defined in the Filesystem Hierarchy Standard, and I actually think that it is a good idea to collect all world-facing services in a single directory (eases security, backup, etc.). Ideally this directory would reside on a separate partition or disk.
root# mkdir -m 755 /srv
Sometimes mail is served from the user’s home directory. I have chosen not to do so, but instead use 1 single central directory whence all mail for all users is served.
root# mkdir -m 751 /srv/mail
And certbot-apache is going to need a virtual host to verify the server’s address for the SSL certificates. My websites are also served from a single central directory.
root# mkdir -m 751 /srv/www
Configuration
The basics
The basics, for me, would be everything we need to be able to send and receive email; that is working SMTP and IMAP servers, including user authentication and SSL encryption for the communication between pc and server. Which is quite some work, already.
As said before, I don’t use POP3. If you want to support POP3, I suggest you follow this tutorial first, and then use the knowledge you’ve gained to add POP3 support.
Firewall / packet filter
Before anything else, let’s pierce some holes in the firewall; typically something one forgets when debugging problems.
Obviously, if you’re not running a firewall on your server (even though you should!), you skip this step.
There are many firewalls, packet filters and whatever they’re called, so I can’t give you the details for yours; maybe I’ll do a little article about packet filtering some other time.
Anyway, for this setup, you’ll need to open up these incoming ports on the server:
- 25 (smtp ⇒ Postfix)
- 53 (dns ⇒ Bind)
- 443 (https ⇒ Apache, for Certbot)
- 587 (submission ⇒ Postfix)
- 993 (imaps ⇒ Dovecot)
- 4190 (sieve ⇒ Dovecot)
DNS
I will assume that you will have your domain name registrar (not yet, later) delegate your entire domain to this server; this server will be the authoritive primary DNS server for your domains. If, for some reason, you decided to do it differently, I guess you know Linux/DNS well enough to be able to improvise.
And if you’re not using an officially registered domain name, remember to use a domainname that ends in ‘.lan’; that could be just domainname lan with hostname yellow.lan, or domainname mycooldomain.lan with hostname yellow.mycooldomain.lan.
We’ll pretend that our assigned IP address is 198.51.100.157; you should, obviously, replace that with the actual IP address for your server.
If you are not at all familiar with the configuration of the Bind DNS server, turn to your favorite search engine, and search for ‘bind dns tutorial‘.
So, as said before, for this document we will be serving the domains example.com, example.net and example.org, and our server is known as yellow.example.com. We could make the MX record for our domains point directly at this hostname, but let’s follow conventions and create a separate name for MX; we’ll just call it mail.example.com, and it needs to be an A record, because MX records may not point at CNAMEs. And then, just because it looks pretty in our email client configuration, we will create 2 CNAMEs indicating the protocols we serve: smtp.example.com and imap.example.com.
Finally, for completeness, we will add ns.example.com to refer to the name server, and www.example.com to host a website; the first must also be an A record, the latter may be a CNAME.
Add this to the end of /etc/bind/named.conf:
zone "example.com" {
type master;
file "/var/bind/pri/example.com.zone";
allow-query { any; };
allow-transfer { xfer; };
};
zone "example.net" {
type master;
file "/var/bind/pri/example.net.zone";
allow-query { any; };
allow-transfer { xfer; };
};
zone "example.org" {
type master;
file "/var/bind/pri/example.org.zone";
allow-query { any; };
allow-transfer { xfer; };
};
And these are the zone files for our domains:
; /var/bind/pri/example.com.zone
$ORIGIN example.com.
$TTL 3h
@ IN SOA ns.example.com. sysadmin.example.com. (
1 ; Serial
3h ; Refresh
1h ; Retry
1w ; Expire
1h ; TTL
);
IN NS ns.example.com.
IN MX 10 mail.example.com.
example.com. IN A 198.51.100.157
ns IN A 198.51.100.157
mail IN A 198.51.100.157
imap IN CNAME mail
smtp IN CNAME mail
www IN CNAME example.com
; /var/bind/pri/example.net.zone
$ORIGIN example.net
$TTL 3h
@ IN SOA ns.example.com. sysadmin.example.com. (
1 ; Serial
3h ; Refresh
1h ; Retry
1w ; Expire
1h ; TTL
);
IN NS ns.example.com.
IN MX mail.example.com.
example.net. IN A 198.51.100.157
www IN CNAME example.net.
As you can see, example.net does not need it’s own ‘ns‘ and ‘mail‘ subdomains; DNS and mail are handled at example.com. Same for example.org:
; /var/bind/pri/example.org.zone
$ORIGIN example.org
$TTL 3h
@ IN SOA ns.example.com. sysadmin.example.com. (
1 ; Serial
3h ; Refresh
1h ; Retry
1w ; Expire
1h ; TTL
);
IN NS ns.example.com.
IN MX mail.example.com.
example.org. IN A 198.51.100.157
www IN CNAME example.org.
And if you’ve paid attention, you’ve seen that we should not forget to create an email alias sysadmin@example.com when we get to the email aliases.
One thing to remember: each time you change these files, you must increment the Serial. If you don’t, you’re changes will not be propagated to the internet.
Usually there is a second NS record, pointing to the secondary name server, which could be your colocation provider, a friend’s server, or a free or paid service elsewhere on the internet. So, go find yourself a secondary, and add it below the first NS record. (You can do this later if you want, but just don’t forget it; it’s important, but not critical.)
Obviously, some configuration will need to be done on their end, and you will have to allow transfer requests from their server; the service you select will surely have documentation on this.
If you’ve just installed Bind, this is the moment where you add it to the default runlevel and start it:
root# rc-update add named default root# rc-service named start
On the other hand, if Bind was installed and running already, you only need to reload the configuration:
root# rc-service named reload
And this is the moment where you contact your domain name registrar to have them delegate your domain to your server.
Now, there is one ‘but’: what if you already have mail for these domains running at some other server, and you don’t want to switch servers before this server is all ready to process incoming mails?
Well, in that case, have the MX record point at the other server for now (don’t forget to increment the Serial, and reload named) before you contact your registrar. That way, your mail will still be delivered at the other server, but at this server we can continue the new setup.
SSL certificates
Now that DNS is up and running, we can get our SSL certificates.
Or actually, we only need a certificate for the example.com domain, because all the MX records point to mail.example.com, and for reading and sending mail we will use imap.example.com and smtp.example.com respectively.
To get our certificate from Let’s Encrypt, we use certbot. This is how it works, roughly:
- we create a virtual HTTP host
- certbot places a file in the virtual host
- certbot sends a request to Let’s Encrypt
- Let’s Encrypt does a HTTP request to get the file created in step 2
- if the file is found, we have proven that we own this domain, and we receive the certificate
Apache web server configuration is really beyond the scope of this tutorial, so I will just give you the configuration for the virtual host.
First create the directories we need:
root# mkdir -p /srv/www/mail.example.com/.well-known/acme-challenge root# chown -R userx:apache /srv/www/mail.example.com root# chmod -R 750 /srv/www/mail.example.com
And then dump this file in /etc/apache2/vhosts.d/:
# /etc/apache2/vhosts.d/03_mail.example.com.conf
<VirtualHost *:80>
ServerName mail.example.com
ServerAlias smtp.example.com imap.example.com
RewriteEngine on
RewriteRule "^(.*)$" https://%{HTTP_HOST}$1 [QSA,R,L]
</VirtualHost>
<VirtualHost *:443>
ServerName mail.example.com
ServerAlias smtp.example.com imap.example.com
DocumentRoot /srv/www/mail.example.com
CustomLog /var/log/apache2/mail.example.com.log combined
ErrorLog /var/log/apache2/mail.example.com.err
SSLEngine on
SSLProtocol ALL -SSLv2 -SSLv3
SSLCipherSuite ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128:AES256:HIGH:!RC4:!aNULL:!eNULL:!EXPORT:!DES:!3DES:!MD5:!PSK
SSLHonorCipherOrder On
SSLCertificateFile /etc/letsencrypt/live/mail.example.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/mail.example.com/privkey.pem
<Directory /srv/www/mail.example.com>
Options FollowSymLinks
Require all granted
</Directory>
</VirtualHost>
And then reload the Apache config:
root# rc-service apache2 reload
and see if you can reach mail.example.com in a web browser.
But Apache will refuse to (re)start because it can’t find the SSL certificates. You could temporarily use one of the pre-generated certificates in /etc/ssl/. You can then change them back after the first certbot run. (Yes, I could have done it correctly right away. But are you here to copy stuff, or to learn stuff…?)
And then we can request the certificate, indicating the location of the VirtualHost document root, and the 3 ‘hosts’ for which it should be valid:
root# certbot certonly --webroot -w /srv/www/mail.example.com -d mail.example.com -d imap.example.com -d smtp.example.com
This will ask several questions. These questions will only be asked once; any next times you run certbot, your answers will be re-used.
It is important to remember or note down the email address you give, as you will have to create an email alias for this later. For the sake of this documentation, let’s say we picked sslcerts@example.com.
If your hostnames have previously been running on other IP addresses, it may take some time and some re-trying before the Let’s Encrypt server finds the correct IP address for verification.
If all has gone well, a directory /etc/letsencrypt will now have been created, holding certificates, keys and what not.
(If all has not gone well, you could try one of the other solutions listed on the Let’s Encrypt website.)
On some Linux distros, a cronjob for automatic renewal of the certificates is automatically installed when installing certbot, but not on Gentoo. Save this as /etc/cron.daily/letsencrypt:
#!/bin/sh /usr/bin/certbot renew --quiet
Make sure the script is executable:
root# chmod 755 /etc/cron.daily/letsencrypt
This will check which certificates near expiry, and renew them, if any. This will work for all Let’s Encrypt certificates on the system, so you don’t need to create a second cronjob if and when you install other certificates for, say, a website.
When the SSL certificate has been renewed, the configuration for Postfix and Dovecot will need to be reloaded. Luckily we can run a script on renewal completion.
Save this as /usr/local/bin/mail_reload.sh:
#!/bin/sh # /usr/local/bin/mail_reload.sh # Postfix and Dovecot reload after SSL renewal. # See /etc/letsencrypt/renewal/mail.example.com.conf rc-service postfix reload rc-service dovecot reload
(Yes, those comments are quite important. You will agree when you run into this script again 8 months from now, and wonder what it’s for.)
Make sure the script is executable.
And add this line to section [renewalparams] of the renewal configuration in /etc/letsencrypt/renewal/mail.example.com.conf:
# /etc/letsencrypt/renewal/mail.example.com.conf ... ... [renewalparams] ... renew_hook = /usr/local/bin/mail_reload.sh ... [[webroot_map]] ...
This concludes the encryption setup.
Side note:
After creating the VirtualHost above, it won’t be long before your logs tell you that script kiddies have found it. Since certbot only needs the directory /srv/www/mail.example.com/.well-known/acme-challenge/ (which translates to https://mail.example.com/.well-known/acme-challenge/), you may wish to put up a ‘catch all page’ for these 1337 h4x0r5. If you lack inspiration for the contents of such page, you could look up goatse or meatspin. 😉
This .htaccess will redirect all requests for non-existant files or directories to /skiddie.html:
# /srv/www/mail.example.com/.htaccess
RewriteEngine on
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /skiddie.html [L]
Passwords
Both Postfix and Dovecot will need to authenticate users who wish to send or read mail. Actually, Dovecot will take care of the authentication for both, but we’ll get to that later. Let’s set up the password database first.
We’ve already created the system user userx. But Dovecot allows us to separate system passwords and mail passwords. And that’s cool, because now if ever a user has their mail password compromised, it won’t compromise the system account; someone who manages to copy a mail password of one of our mail users, will not automatically have shell access. We could even choose to not create a system password for our users, at all.
Allthough it’s tempting to go for a MariaDB password database, I decided to go for a plaintext database. Running a DBMS means more software to maintain, and another daemon that may crash. Maybe I’ll switch to MariaDB if and when I decide to create a web interface for setting the password. Or maybe not.
The mail passwords will be saved in the file /etc/passwd.mail, a file that doesn’t (or at least shouldn’t) exist yet. For now, all this file needs is the username and a colon:
userx:
The mail username should match the system username.
This is the script that changes the password; save it as /usr/local/bin/mailpass:
#!/usr/bin/env bash
# Change IMAP/SMTP password.
# For details, see https://www.ohreally.nl/2018/11/19/mail-server/
######################################################################
# Copyright (c) 2018 Rob LA LAU <mailtuto@ohreally.nl>
# This work is licensed under the
# Creative Commons Attribution-ShareAlike 4.0 International License.
# To view a copy of this license, visit
# http://creativecommons.org/licenses/by-sa/4.0/
# or send a letter to
# Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.
######################################################################
######################################################################
# CONFIGURATION
# Where to store the passwords.
PASSFILE=/etc/passwd.mail
# END OF CONFIGURATION
######################################################################
DOVEADM=`which doveadm` || exit 1
SED=`which sed` || exit 1
[ -z "${SUDO_USER}" ] && {
echo "This script should be executed using sudo."
exit 1
}
[ ! -f "${PASSFILE}" ] && {
echo "Password file ${PASSFILE} does not exist."
exit 1
}
newpass=`"${DOVEADM}" pw -s blf-crypt`
retval=$?
[ ${retval} -eq 0 ] && "${SED}" -e "\|^${SUDO_USER}:|s|^.*$|${SUDO_USER}:${newpass}|" -i "${PASSFILE}"
Naturally, this script should be executable, but it need not be executable to everybody, as it will only be run through sudo:
root# chmod 700 /usr/local/bin/mailpass
Save this as /etc/sudoers.d/mailpass to allow all users in system group mailusers to run that script:
# /etc/sudoers.d/mailpass %mailusers ALL=(root) NOPASSWD: /usr/local/bin/mailpass
And finally, create a group mailusers, and add our user to that group:
root# groupadd mailusers root# usermod -aG mailusers userx
Our user should now be able to set (and change) their password (provided they have shell access):
userx$ sudo mailpass
...
userx$ cat /etc/passwd.mail
userx:{BLF-CRYPT}$2y$...and.so.on...
(Yes, you should do so now. If you don’t, you will forget it, and then later it will take you hours to find out why things don’t work.)
If your users do not have shell access, it should be quite trivial to create a simple web interface for this. If you do so, please open source it and notify me, so I can add a link here.
IMAP server
Well, finally we get to some real email stuff!
To start, I don’t want to do anything fancy, yet. We will just allow our user to read their mail, using the IMAP protocol, over an SSL encrypted connection. Well, actually that is quite fancy. But for now we won’t do any filtering and all that.
The configuration for Dovecot can be found in /etc/dovecot/ (what a surprise 😉 ). These are the changes I made for this first phase:
# /etc/dovecot/dovecot.conf # Only serve IMAP. protocols = imap
# /etc/dovecot/conf.d/10-auth.conf # Do not include auth-system.conf.ext. #!include auth-system.conf.ext # Include auth-local.conf.ext. !include auth-local.conf.ext
# /etc/dovecot/conf.d/auth-local.conf.ext
# Use separate file for password verification.
passdb {
driver = passwd-file
args = /etc/passwd.mail
}
# Use regular system password file for other data.
userdb {
driver = passwd
}
# /etc/dovecot/conf.d/10-mail.conf # Change mail location. mail_location = maildir:/srv/mail/%u
# /etc/dovecot/conf.d/10-ssl.conf # Require SSL. ssl = required # Use Let's Encrypt SSL certificate. ssl_cert = </etc/letsencrypt/live/mail.example.com/fullchain.pem ssl_key = </etc/letsencrypt/live/mail.example.com/privkey.pem
And that’s about it; time to start Dovecot:
root# rc-update add dovecot default root# rc-service dovecot start
My version of Dovecot suffers from a bug where it requires DH parameters even when you’ve not enabled Diffie-Hellman key exchange. If your server also complains about that, add this line to /etc/dovecot/conf.d/10-ssl.conf:
ssl_dh = </etc/dovecot/dh.pem
And run this command to generate the file in question:
root# openssl dhparam 4096 > /etc/dovecot/dh.pem
This command will take a LONG time, especially on a new server, that hasn’t collected a lot of entropy yet. And I don’t mean Go grab a cup of coffee-long, but more like Go to work and hope it’s finished when you get back-long.
Your IMAP server should now be up and running. Try and connect to it using these settings:
- server name: imap.example.com
- protocol: IMAP
- port: 993
- connection security: SSL/TLS
- authentication method: normal password (sometimes called plaintext)
- user name: userx
- password: you’ve set this above
And when you’ve managed to connect, it’s time for the final test.
Open a shell at the server as userx, and save this as ${HOME}/testmail:
Return-Path: <userx@yellow.example.com> From: userx@yellow.example.com Date: Sat, 18 Nov 2018 11:44:18 +0200 To: userx@yellow.example.com Subject: Testing IMAP setup It works!
and then run this command:
userx$ cat ${HOME}/testmail | /usr/libexec/dovecot/dovecot-lda -d userx
If this mail arrives in the mailbox in your email client, then IMAP works (and LDA as well).
For more information on Dovecot’s configuration, head on over to the Dovecot wiki.
SMTP server
Okay, it’s nice to have a running IMAP server. But if you want a bit more than just sending yourself mails from the command line, we’re going to need an SMTP server.
Postfix listens for connections on 2 ports: 25 (smtp) where external mail servers deliver mails destined for us, and 587 (submission) where our authenticated users deliver their mails to have them sent out on the internet.
For our users to get authenticated, we’ll make use of Dovecot-SASL (Simple Authentication and Security Layer), which in short means that Dovecot will take care of the authentication, using the password database we created above.
For this to work, we need to add a few lines to the Dovecot configuration; add the blue and bold lines:
# /etc/dovecot/conf.d/10-master.conf
service auth {
unix_listener /var/spool/postfix/private/auth {
mode = 0660
user = postfix
group = postfix
}
}
When Dovecot is restarted, this will create a Unix socket at said location, that may be used by Postfix to authenticate users.
The Postfix configuration, you guessed it, can be found in /etc/postfix/. These are the changes for this first phase:
# /etc/postfix/master.cf # Added some options to existing service! submission inet n - n - - smtpd -o syslog_name=postfix/submission -o smtpd_tls_wrappermode=no -o smtpd_tls_security_level=encrypt -o smtpd_sasl_auth_enable=yes -o smtpd_sasl_type=dovecot -o smtpd_sasl_path=private/auth -o smtpd_sasl_security_options=noanonymous -o smtpd_sasl_local_domain=$myhostname -o smtpd_client_restrictions=permit_sasl_authenticated,reject -o smtpd_recipient_restrictions=reject_non_fqdn_recipient,reject_unknown_recipient_domain,permit_sasl_authenticated,reject
# /etc/postfix/main.cf
# Server info.
myhostname = yellow.example.com
mydomain = example.com
myorigin = $mydomain
mynetworks_style = host
# Disable mailbox in home directory.
#home_mailbox = .maildir/
# Use Dovecot as LDA.
mailbox_command = /usr/libexec/dovecot/dovecot-lda -f "$SENDER" -a "$RECIPIENT"
# Local aliases.
alias_database = hash:/etc/postfix/aliases
alias_maps = hash:/etc/postfix/aliases
# Virtual aliases.
virtual_alias_domains = example.com example.net example.org
virtual_alias_maps = hash:/etc/postfix/virtual
# SASL
smtpd_sasl_auth_enable = yes
smtpd_sasl_type = dovecot
smtpd_sasl_path = private/auth
# SSL / TLS
smtpd_tls_security_level = may
smtpd_tls_protocols = !SSLv2 !SSLv3
smtpd_tls_cert_file = /etc/letsencrypt/live/mail.example.com/fullchain.pem
smtpd_tls_key_file = /etc/letsencrypt/live/mail.example.com/privkey.pem
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_cache
smtp_tls_session_cache_database = btree:${data_directory}/smtp_cache
# Restrictions
smtpd_relay_restrictions =
permit_mynetworks
permit_sasl_authenticated
reject_unauth_destination
smtpd_recipient_restrictions =
permit_sasl_authenticated
permit_mynetworks
permit
The values for that last parameter look a bit funny, 3 times ‘permit’ in a row, but we’ll add some other values in between later.
I have the feeling that there is quite some overlap between my master.cf and my main.cf (my config is a collection of very, VERY many examples I found online). The Postfix documentation, at least to me, is not quite clear about the relation between the options with the same names, specified in the one file or in the other.
That said, my setup does what I expect it to do, so at least the options do not bite each other.
In the above configuration, we defined 2 alias databases: alias_database and virtual_alias_maps. These files define for which recipients the mail should be redirected to which other recipients.
/etc/mail/aliases
This file is for local aliases. The format is very simple:
original_recipient: new_recipient
If you open the file in a text editor, you will see that all (local) mail actually ends up with user root. So if you would send a mail to abuse@yellow.example.com, it would be redirected to postmaster@yellow.example.com, which would be redirected to root@yellow.example.com.
There is one thing to be added to this file: mail for root should be redirected to a user who actually reads their mail. Which in our case would be the only ‘real’ user we have: userx.
Add this line to the file, and add it somewhere near the top, so it’s easy to find:
root: userx
These days, with virtual domains, it’s quite rare to have to change more than that.
Postfix does not read this file directly, but prefers a lookup table, which we generate with the command
root# newaliases
/etc/postfix/virtual
The second aliases database defines aliases for virtual users. This database also consists of lines with an LHS (left-hand-side) and an RHS (right-hand-side). There are a few differences:
- LHS does not end in a colon
- LHS and RHS may contain a domain name
- LHS defines a virtual user or virtual domain
- RHS may indicate a local or virtual user
And for each virtual domain, there must be a line where the LHS defines the domain name, and the RHS can be any string.
Here’s an example:
# /etc/postfix/virtual example.com -- any string -- accounting@example.com office@example.com sales@example.com office@example.com office@example.com userx@example.com userx@example.com userx example.net -- any string -- acquisition@example.net sales@example.com sales@example.net @example.com @example.net userx example.org -- any string -- @example.org @example.com
This virtual alias database would result in this:
- accounting@example.com and sales@example.com will be redirected to office@example.com
- office@example.com, including the mails redirected above, will be redirected to userx@example.com
- userx@example.com will be delivered to local user userx
- other addresses in the example.com domain will be bounced with a message indicating a ‘user unknown’ error
- acquisition@example.net will be redirected to sales@example.com
- sales@example.net will be redirected to sales@example.com
- mail for any other address in the example.net domain will be delivered to local user userx
- mail for any address in the example.org domain will be forwarded to that same address in the example.com domain, where it will be subject to the rules for that domain; so warehouse@example.org will be bounced, because warehouse@example.com does not exist
Which ever addresses you create, do not forget that we need at least sysadmin@example.com (see the chapter about DNS) and sslcerts@example.com (see the chapter about SSL certificates).
This file must also be converted to a Postfix lookup table after having been edited:
root# postmap /etc/postfix/virtual
And with that, it’s time to fire up Postfix as well:
root# rc-update add postfix default root# rc-service postfix start
More info at the Postfix website, with all configuration parameters (and their possible values, obviously) listed alphabetically on the postconf(5) man page.
End of phase 1
That concludes ‘the basics’. You now have a fully functional mail server, and if you’ve had enough of this tutorial, you can stop here, and just enjoy what you have.
But if I were you, I’d continue to the next chapter…
Anyway, stop or continue, if you’ve pointed your MX records elsewhere while setting up SMTP an IMAP, you can now change them to point to this server.
If you decide to read on, it might be a good idea to open another terminal and
root# tail -f /var/log/mail/current
for the rest of this manual.
Hit Ctrl-C to end that command.
Sieve
If you’re anything like me, you have about a zillion folders and subfolders in your mailbox. So, now that the server can receive mail, you may want to organize the automatic sorting of your incoming mails before doing anything else.
To this end, we’ve included Sieve support when installing Dovecot.
Sieve is a scripting language for filtering email messages. You could edit the scripts using your favorite text editor when you’re logged in at the server. But ManageSieve clients — and add-ons for email clients — exist as well, permitting the user to edit their filters in a more user friendly way, and using their SMTP/IMAP password.
A small appetiser:
require ["fileinto", "reject"];
if header :matches "Subject" "*viagra*" {
discard;
stop;
}
elsif address :is :localpart "from" "exgirlfriend" {
reject "We're over. Get a life.";
stop;
}
elsif address :is ["to", "cc"] "sales@example.com" {
fileinto "Sales";
stop;
}
# Everything that falls through, is delivered to the INBOX.
Go to Sieve.info for tutorials, references, example scripts, clients, email client add-ons, and more.
The ManageSieve daemon listens on server port 4190. You connect with the same username and password as you use for SMTP and IMAP.
Personally I really enjoy the Sieve add-on for Thunderbird for editing my scripts; it has syntax check, a Sieve reference, et cetera. But on the other hand, I confess that I have not tried any other clients (except ssh+vim, of course).
Address tagging
You’ve probably seen email addresses that contain a plus sign in the local part. This is called address tagging. For example, the user has address userx@example.com; but to subscribe to different websites, she uses userx+hyves@example.com and userx+myspace@example.com. Mails to those tagged addresses are delivered to the original address as if the +hyves and +myspace parts don’t exist; no changes need to be made to the MDA configuration to create these addresses, which means the user may use as many different addresses (or more accurately ‘aliases’) as she wants, without needing the help of a sysadmin. The user can then use these tags to easily filter incoming mails, or to see which websites sell email addresses to spammers.
To enable address tagging, we only need to set 2 parameters in the Postfix configuration.
However, many websites use badly written regular expressions to verify email addresses, and do not allow the plus sign (violating RFC2822). To work around these poorly developed sites, I suggest we use the minus sign (‘-‘) instead; a dot (‘.’) could be another valid choice.
# /etc/postfix/main.cf recipient_delimiter = - propagate_unmatched_extensions = canonical, virtual
root# rc-service postfix reload
From now on, mails to userx-hyves@example.com and userx-myspace@example.com will be delivered to userx@example.com.
To easily filter these addresses when mail comes in, we also have to set 2 parameters in the Dovecot configuration (in blue and bold).
# /etc/dovecot/conf.d/15-lda.conf
recipient_delimiter = -
# /etc/dovecot/conf.d/90-sieve.conf
plugin {
sieve = file:~/sieve;active=~/.dovecot.sieve
recipient_delimiter = -
}
root# rc-service dovecot reload
(Why we need to set it twice for Dovecot is beyond me.)
Those mails can now be Sieve filtered like this:
require ["fileinto", "subaddress"];
if address :detail "to" ["hyves", "myspace"] {
fileinto "socialmedia";
stop;
}
Fine tuning Postfix
The cheapest way of fighting spam is to have Postfix bounce the most obvious spam mails at the door.
HELO / EHLO checking
When a mail server wishes to deliver a message to another mail server, it begins the session by issuing a HELO or EHLO command, followed by it’s hostname.
We’ll tell Postfix that we do not want it to accept delivery of mails without this HELO command, and also that the HELO hostname should adhere to certain restrictions.
# /etc/postfix/main.cf
smtpd_helo_required = yes
smtpd_helo_restrictions =
permit_mynetworks
reject_invalid_helo_hostname
reject_non_fqdn_helo_hostname
reject_unauth_pipelining
regexp:/etc/postfix/helo.regexp
permit
The file /etc/postfix/helo.regexp looks like this:
# /etc/postfix/helo.regexp
/^yellow\.example\.com$/ 550 Rejected
/^mail\.example\.com$/ 550 Rejected
/^smtp\.example\.com$/ 550 Rejected
/^imap\.example\.com$/ 550 Rejected
/^ns\.example\.com$/ 550 Rejected
/^www\.example\.com$/ 550 Rejected
/^198\.51\.100\.157$/ 550 Rejected
/^\[198\.51\.100\.157\]$/ 550 Rejected
/^[0-9.]+$/ 550 Rejected
/^[0-9]+(\.[0-9]+){3}$/ 550 Rejected
In short, this means that if the sending server tries to fool us by using one of our own hostnames or our IP address, the mail is bounced with a 550 error code.
root# rc-service postfix reload
Sender and recipient checking
We don’t want Postfix to accept mails for which the MAIL FROM or RCPT TO commands are not in the correct format.
Lines in bold and blue have been added:
# /etc/postfix/main.cf strict_rfc821_envelopes = yes smtpd_recipient_restrictions = permit_sasl_authenticated reject_non_fqdn_sender reject_non_fqdn_recipient reject_unauth_destination reject_unknown_sender_domain permit_mynetworks permit
root# rc-service postfix reload
More recipient checking
I have some email addresses that I created a long time ago, but that I no longer use, and they just collect spam these days. So I’d like to reject all mail that is sent to those addresses. To this end, I created a file containing all addresses I’d like to blindly reject.
# /etc/postfix/recipients nolongerused@example.com REJECT ancientaddress@example.net REJECT spammagnet@example.org REJECT
And then I built a Postfix lookup table from that file:
root# postmap /etc/postfix/recipients
To enable the check, I added a single line (bold and blue) to the Postfix config:
# /etc/postfix/main.cf
smtpd_recipient_restrictions =
permit_sasl_authenticated
reject_non_fqdn_sender
reject_non_fqdn_recipient
reject_unauth_destination
check_recipient_access hash:/etc/postfix/recipients,
reject_unknown_sender_domain
permit_mynetworks
permit
And I reloaded the Postfix configuration:
root# rc-service postfix reload
If I ever want to add addresses to this list, I add them to /etc/postfix/recipients, and I re-run the postmap command; I do not need to reload the Postfix config each time.
I could have done all this using the virtual database (see above). But the advantage of using a separate database is, that I can now run this check early in the process. No need to greylist these messages, and run them through a spam filter, if I know I’ll reject them anyway. Saves some resources.
Greylisting
Greylisting eliminates a considerable amount of spam. And it does so at low cost, because greylisting is light on resources. So this is a valuable technique to fight spam, especially if we use it before more costly techniques like a spam filter or Real-time Blackhole Lists.
You can have a look at Postgrey’s configuration in /etc/conf.d/postgrey, but there’s nothing that needs to be changed.
Postgrey runs as a daemon, so it must be added to the default runlevel:
root# rc-update add postgrey default root# rc-service postgrey start
And then it can be added to the Postfix configuration, so that Postfix will query the daemon (bold and blue line was added):
# /etc/postfix/main.cf
smtpd_recipient_restrictions =
permit_sasl_authenticated
reject_non_fqdn_sender
reject_non_fqdn_recipient
reject_unauth_destination
check_recipient_access hash:/etc/postfix/recipients
reject_unknown_sender_domain
permit_mynetworks
check_policy_service inet:127.0.0.1:10030
permit
root# rc-service postfix reload
Real-time Blackhole Lists
RBLs are lists of IP addresses known to belong to spammers. SMTP servers can check sender IP addresses against these lists by performing a DNS query.
You can add as many RBLs as you like, but be advised that each query takes time and resources.
The Wikipedia page on RBLs would be a good starting point to learn more about RBLs, and find more addresses.
Bold and blue lines were added:
# /etc/postfix/main.cf
smtpd_recipient_restrictions =
permit_sasl_authenticated
reject_non_fqdn_sender
reject_non_fqdn_recipient
reject_unauth_destination
check_recipient_access hash:/etc/postfix/recipients
reject_unknown_sender_domain
permit_mynetworks
check_policy_service inet:127.0.0.1:10030
reject_rbl_client sbl.spamhaus.org
reject_rbl_client xbl.spamhaus.org
reject_rbl_client dnsbl.sorbs.net
permit
root# rc-service postfix reload
Sender Policy Framework
SPF is not so much a technique to protect ourselves from receiving unwanted mail, as it is a technique to help others protect themselves from receiving unwanted mail sent in our name. And, obviously, we can protect ourselves from receiving unwanted mails by taking advantage of the SPF setup of others.
As I mentioned before, anyone can put any address as the sender address on any email.
SPF lets us create a DNS TXT record that defines which server(s) may be used to send mail for our domain. An example to make this more clear: if yellow.example.com is the only server we use to send mail onto the internet (the SMTP server we have configured in our email client), we create a TXT record saying so, allowing the receiver of an email which has our address as the sender address, to verify whether that email was sent from an SMTP server that we allow; if it wasn’t (our DNS record does not list the SMTP server that was used), the sender address is probably forged, and the mail is likely to contain spam or a virus.
The Postfix SPF policy daemon we’ve installed, tries to verify these DNS records for incoming mails. You will find a small configuration file at /etc/python-policyd-spf/policyd-spf.conf, but I didn’t need to change anything there.
For outgoing mails, all we need to do is add a correctly crafted TXT record to our own DNS. A sample record might look like this:
IN TXT "v=spf1 mx ip4:203.0.113.21 ip6:2001:db8:85a3:8d3:1319:8a2e:370:7348 -all"
This particular record tells us that we’re speaking SPF version 1 (v=spf1), and that 3 hosts may be used as the sending SMTP server for this domain:
- the host that is listed in the MX record for our domain (mx)
- the host with IPv4 address 203.0.113.21, which could be the SMTP server we must use when sending mails from work, for example
- the host with IPv6 address 2001:db8:85a3:8d3:1319:8a2e:370:7348, which could be a friend’s server we use as back up
Mails carrying a sender address in our domain, but coming from any other SMTP server, should be considered having a spoofed sender address, and rejected (-all).
An MTA receiving mail with userx@example.com as sender address may now check the DNS for example.com, and drop the mail if the sending SMTP server is not in the list we defined.
Obviously, this TXT record should be added to all 3 of our zones (example.com, example.net and example.org). Here is a simple but complete example:
; /var/bind/pri/example.net.zone
$ORIGIN example.net
$TTL 3h
@ IN SOA ns.example.com. sysadmin.example.com. (
2 ; Serial
3h ; Refresh
1h ; Retry
1w ; Expire
1h ; TTL
);
IN NS ns.example.com.
IN MX mail.example.com.
IN TXT "v=spf1 mx ip4:203.0.113.21 ip6:2001:db8:85a3:8d3:1319:8a2e:370:7348 -all"
example.net. IN A 198.51.100.157
www IN CNAME example.net.
No extra steps need to be taken to reconfigure our SMTP server or our email client: the SMTP server on the receiving end of our emails will, if correctly configured, ask our DNS server to send the SPF TXT record, and will process it if it exists.
Remember to increase the Serial for the DNS zones, and to reload the Bind config.
root# rc-service named reload
You can verify the record with this command:
root# dig @127.0.0.1 example.com TXT
And finally, to instruct Postfix to verify the SPF record on incoming mails, we add the bold and blue lines to the Postfix config:
# /etc/postfix/main.cf
smtpd_recipient_restrictions =
permit_sasl_authenticated,
reject_non_fqdn_sender,
reject_non_fqdn_recipient,
reject_unauth_destination,
check_recipient_access hash:/etc/postfix/recipients,
reject_unknown_sender_domain,
permit_mynetworks,
check_policy_service inet:127.0.0.1:10030,
reject_rbl_client sbl.spamhaus.org,
reject_rbl_client xbl.spamhaus.org,
reject_rbl_client dnsbl.sorbs.net,
check_policy_service unix:private/policyd-spf,
permit
policyd-spf_time_limit = 3600
And then, obviously, we restart Postfix.
DomainKeys Identified Mail
DKIM is comparable to SPF in the sense that it allows a recipient of an email to verify the sender domain by means of a DNS record. Furthermore it signs an email, allowing the recipient to verify that the mail has not been modified between dispatch and retrieval. The signature is added to the headers of the email, while the public key to verify the signature can be found in a DNS TXT record.
To generate the private and public keys, opendkim-genkey is used:
root# mkdir /etc/opendkim/keys root# cd /etc/opendkim/keys/ root# mkdir example.com example.net example.org root# opendkim-genkey --directory=./example.com/ --domain=example.com --selector=dkim root# opendkim-genkey --directory=./example.net/ --domain=example.net --selector=dkim root# opendkim-genkey --directory=./example.org/ --domain=example.org --selector=dkim
This creates the files dkim.private and dkim.txt in all 3 subdirectories, where dkim.private contains the private key, and dkim.txt contains the DNS TXT record that holds the public key.
The private keys must be owned by user milter, otherwise they cannot be loaded:
root# chown milter:milter */dkim.private
The problem here is that I don’t know for which DNS server this DNS record is generated, but Bind on my server wasn’t too happy with it.
The contents of dkim.txt looks somewhat like this:
dkim._domainkey IN TXT ( "v=DKIM1; k=rsa; "
"p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDCOQAKFvVZGLgBV9voysIxiHm4HO42z+YK8UfcZIhQK+uptyD49Bl1KZA41ijjfysHhav7IsVbk/dfWwP9kZM2fWJthmhMng9+IbncjYLRND9QSzkcqH+3ShW91HchnFadp4SsWE7s0r8mlBToRd1wIY4xa/yJ1JFEDbHgEp/3cQIDAQAB" ) ; ----- DKIM key dkim for example.com
(But not exactly, because you will have a different public key.)
Change that to this:
dkim._domainkey IN TXT "v=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDCOQAKFvVZGLgBV9voysIxiHm4HO42z+YK8UfcZIhQK+uptyD49Bl1KZA41ijjfysHhav7IsVbk/dfWwP9kZM2fWJthmhMng9+IbncjYLRND9QSzkcqH+3ShW91HchnFadp4SsWE7s0r8mlBToRd1wIY4xa/yJ1JFEDbHgEp/3cQIDAQAB"
So:
- everything on 1 line
- lose the brackets
- lose the idle quotes in the middle
- lose everything after the last quote
The result may then be copied into the zone file for this domain. Do so for each key/domain. And don’t forget to increment the Serial in the zone file, and to reload the named config afterwards. Here is an example with both the SPF and DKIM records:
; /var/bind/pri/example.net.zone
$ORIGIN example.net
$TTL 3h
@ IN SOA ns.example.com. sysadmin.example.com. (
3 ; Serial
3h ; Refresh
1h ; Retry
1w ; Expire
1h ; TTL
);
IN NS ns.example.com.
IN MX mail.example.com.
IN TXT "v=spf1 mx ip4:203.0.113.21 ip6:2001:db8:85a3:8d3:1319:8a2e:370:7348 -all"
dkim._domainkey IN TXT "v=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQDCOQAKFvVZGLgBV9voysIxiHm4HO42z+YK8UfcZIhQK+uptyD49Bl1KZA41ijjfysHhav7IsVbk/dfWwP9kZM2fWJthmhMng9+IbncjYLRND9QSzkcqH+3ShW91HchnFadp4SsWE7s0r8mlBToRd1wIY4xa/yJ1JFEDbHgEp/3cQIDAQAB"
example.net. IN A 198.51.100.157
www IN CNAME example.net.
To verify the record:
root# dig @127.0.0.1 dkim._domainkey.example.com TXT
If this works, we configure OpenDKIM itself (see `man opendkim.conf‘ for the meaning of these parameters):
# /etc/opendkim/opendkim.conf AutoRestart yes AutoRestartRate 10/1h Syslog yes SyslogSuccess yes UserID milter:milter PidFile /run/opendkim/opendkim.pid Socket inet:8891@localhost ReportAddress postmaster@ohreally.nl SendReports yes Mode sv Canonicalization relaxed/simple KeyTable file:/etc/opendkim/KeyTable SigningTable refile:/etc/opendkim/SigningTable
# /etc/opendkim/KeyTable dkim._domainkey.example.com example.com:dkim:/etc/opendkim/keys/example.com/dkim.private dkim._domainkey.example.net example.net:dkim:/etc/opendkim/keys/example.net/dkim.private dkim._domainkey.example.org example.org:dkim:/etc/opendkim/keys/example.org/dkim.private
# /etc/opendkim/SigningTable *@example.com dkim._domainkey.example.com *@example.net dkim._domainkey.example.net *@example.org dkim._domainkey.example.org
Then opendkim daemon may be added to the default runlevel, and started:
root# rc-update add opendkim default root# rc-service opendkim start
And finally we tell Postfix where to find the socket we just created:
# /etc/postfix/main.conf smtpd_milters = inet:localhost:8891 non_smtpd_milters = inet:localhost:8891
root# rc-service postfix reload
Send yourself an email, and look at its headers when it arrives.
Plug-ins/add-ons exist for email clients to verify DKIM signatures.
Spam filter
What I’ve described above, eliminates 90-95% of all the spam that is presented to my server. Obviously, spam is a very personal subject, as it depends at least in part of the mailing lists you’re subscribed to, the websites where you have accounts, and the amount of Viagra you order online. So your mileage may vary (I’ve always wanted to use that phrase 😉 ). But I think we can agree that we’ve done quite some spam filtering already, considering that we did it without a spam filter.
Now let’s send what’s left through Bogofilter, to filter out the final 5-10%.
Generally there are 2 points in the chain where you can send the mail through the spamfilter.
The first is the MTA (Postfix in our case). The advantage of this approach is that the MTA can reject spam at the door, letting the sender know that it’s useless to deliver spam to our server. The disadvantage is that all users share the same spam database, which means results are less personalized; additionally, this could be more expensive, because mails are tested against a database that contains tokens that are not relevant.
The other option is to let the LDA (Dovecot in our case) take care of the spam filtering. Disadvantage of this approach is that we cannot reject messages, but only discard. Advantages are that each user has their own, custom fit spam database, and that it’s easier to install and configure.
I decided to go for the latter approach.
We’ll do the filtering in 2 phases: first we’ll have Bogofilter add a header to the mail, indicating the probability of the mail being spam, and then we’ll use Sieve filters to decide where to send it.
Bogofilter needs a wordlist to compare the mails to. To create an empty wordlist, we’ll have to do this for each user (once):
root# mkdir -m 700 ~userx/.bogofilter root# echo "" | bogoutil -l ~userx/.bogofilter/wordlist.db root# chown -R userx:userx ~userx/.bogofilter
The phrase ~userx (’tilde-userx’) is a shortcut for writing /home/userx, but if you’ve changed the home directory to some other location, it still points to the correct directory.
Instead of sending the mail to Bogofilter directly, we’ll create a wrapper; this allows us to save the mail to a temporary file, which avoids forcing Bogofilter to keep the entire mail in memory while it is being processed.
First create a directory:
root# mkdir -p /usr/lib/dovecot/sieve/filter
And then save this as /usr/lib/dovecot/sieve/filter/sieve-bogofilter.sh:
#!/usr/bin/env bash
# Run a mail through Bogofilter.
# For details, see https://www.ohreally.nl/2018/11/19/mail-server/
######################################################################
# Copyright (c) 2018 Rob LA LAU <mailtuto@ohreally.nl>
# This work is licensed under the
# Creative Commons Attribution-ShareAlike 4.0 International License.
# To view a copy of this license, visit
# http://creativecommons.org/licenses/by-sa/4.0/
# or send a letter to
# Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.
######################################################################
######################################################################
# CONFIGURATION
# Directory for the temporary files.
TMPDIR="${HOME}/.bogotmp"
# These should be absolute, as there is no ${PATH}!
BOGO=/usr/bin/bogofilter
CAT=/bin/cat
MKDIR=/bin/mkdir
MKTEMP=/bin/mktemp
RM=/bin/rm
# END OF CONFIGURATION
######################################################################
# Create temporary files.
[ -d "${TMPDIR}" ] || "${MKDIR}" "${TMPDIR}" || exit 1
in=`"${MKTEMP}" --tmpdir="${TMPDIR}" --suffix=.in`
out=${in/%in/out}
# Redirect STDIN to a file.
"${CAT}" > "${in}"
# Check the file, creating a new file which includes the header.
# The Pigeonhole "spamtest" extension accepts no more than 4 digits after the decimal point.
"${BOGO}" --spamicity-formats="%0.4f, %0.4f, %0.4f" -p -I "${in}" -O "${out}"
# Output the new file.
"${CAT}" "${out}"
# Clean up.
"${RM}" "${in}" "${out}"
That script will add a header line to the mail, indicating the probability of it being spam. That is, if you make sure it’s executable:
root# chmod 755 /usr/lib/dovecot/sieve/filter/sieve-bogofilter.sh
Create a Sieve script to call the Bogofilter wrapper:
root# mkdir /usr/lib/dovecot/sieve/before
# /usr/lib/dovecot/sieve/before/bogofilter.sieve require "vnd.dovecot.filter"; filter "sieve-bogofilter.sh";
Then, enable the filter directory and call the script before any other scripts. At the same time, add the configuration we need to interpret the header that was added.
# etc/dovecot/conf.d/90-sieve-extprograms.conf
plugin {
sieve_filter_bin_dir = /usr/lib/dovecot/sieve/filter
}
# /etc/dovecot/conf.d/90-sieve.conf
plugin {
sieve_before = /usr/lib/dovecot/sieve/before/bogofilter.sieve
sieve_plugins = sieve_extprograms
sieve_extensions = +vnd.dovecot.filter +spamtest +spamtestplus
sieve_spamtest_status_type = score
sieve_spamtest_status_header = X-Bogosity: [[:alpha:]]+, tests=bogofilter, spamicity=([[:digit:]\.]+), version=[[:digit:]\.]+
sieve_spamtest_max_value = 1
}
You will need to compile the Sieve script into a binary. Normally the Sieve interpreter will do that for you, but it has no write permissions for this directory (and since it’s quite rare to create new scripts here, we won’t give it any).
We do this after editing the Dovecot configuration, because the vnd.dovecot.filter extension needs to be loaded.
root# sievec /usr/lib/dovecot/sieve/before/bogofilter.sieve
An alternative would be to call the script from the user’s Sieve scripts. On the one hand that would enable the more knowledgeable user to switch to some other spam filter (provided that an appropriate script is available in the sieve_filter_bin_dir), or to disable spam filtering all together. On the other hand, it would enable the less knowledgeable user to screw up the spam filtering when experimenting with their Sieve scripts, needing the help of a sysadmin to repair things.
For me, Bogofilter is the only installed spam filter (and currently I’m the only user), so this is a perfect solution. Even though I consider myself a rather knowledgeable user. 🙂
Anyway, to move the call to the user, all you need to do, is move the line ‘filter “sieve_bogofilter.sh”;‘ to the users’ Sieve scripts.
Reload the Dovecot configuration to activate the changes.
And finally we use the X-Bogosity header that was added, to file the mail into the correct folder. Which might look something like this:
require ["comparator-i;ascii-numeric", "fileinto", "mailbox", "relational", "spamtest"];
if spamtest :value "ge" :comparator "i;ascii-numeric" "10" {
discard;
}
elsif spamtest :value "ge" :comparator "i;ascii-numeric" "5" {
fileinto :create "ProbablySpam";
}
elsif spamtest :value "ge" :comparator "i;ascii-numeric" "2" {
fileinto :create "MaybeSpam";
}
The spamtest plugin translates the score given by Bogofilter (or any other spam filter) to a number between 0 — definitely not spam, or header line not found — to 10 — definitely spam — inclusive. And yes, this means that you could switch spam filters without users noticing; just change the sieve_before and the sieve_spamtest_status_header above.
Please do not be tempted to use the Sieve reject command for spam. Using reject will send a mail to the original sender of the mail, but since spammers often use forged sender addresses, your reject message will probably end up in the mailbox of somebody who had nothing to do with the spam you received. Read more about backscatter on Wikipedia.
Having spam rejected by Postfix is something entirely different, because in that case it is part of SMTP, where Postfix says to the sending server ‘No, I will not accept this mail.‘, resulting in the mail never entering the server. By the time the mail reaches the LDA, it has been accepted already, and by then the only way to ‘reject’ it, is to actually send a reply to the sender.
TL;DR:
The Sieve reject command should actually be called reply, and is not fit for rejecting spam.
Automated ham/spam (re-)classification
Well, we have a spam filter now. And it’s absolutely useless…
To determine whether a mail is spam, Bogofilter needs a list of words to compare the mail to. And currently, that list is still empty. Wouldn’t it be great if we had a mail folder where we could just drop mails to have them automatically classified as spam? Well, let’s create such mail folder.
First, for each user:
root# mkdir /srv/mail/userx/.Spam root# mkdir /srv/mail/userx/.Spam/cur root# mkdir /srv/mail/userx/.Spam/new root# mkdir /srv/mail/userx/.Spam/tmp
Mind you: the directory name for a mail folder starts with a dot! (Which is kind of why your mail client doesn’t allow you to use dots in mail folder names.)
If you are the only user, you could also just create a new toplevel folder named ‘Spam‘ in your mail client.
You may need to restart your mail client to make the new folder visible. If it still doesn’t show after a restart, your mail client is probably configured to only display subscribed folders; you can find more information about that in your mail client’s documentation.
So, these are now (or will be) the folders in our mail account:
- Spam
The folder we just created for mails of which we decide they are spam. The LDA will never deliver mail here; mails will ony enter manually. We will use this folder to train Bogofilter. - ProbablySpam
This folder will be created by our Sieve script for mails of which Bogofilter says they are probably spam. If they are, we move them to the Spam folder. If they are not, we move them to any other folder. - MaybeSpam
This folder will be created by our Sieve script for mails of which Bogofilter says they may be spam. If they are, we move them to the Spam folder. If they are not, we move them to any other folder. - other folders
For mails that are not spam. Mails that are not delivered to ProbablySpam or MaybeSpam will be delivered to one of the other folders, depending on our Sieve filters. If spam happens to fall through, we move it to the Spam folder.
Using ImapSieve we can attach hooks to all those folders for our ‘move’ actions.
These are the actions we need:
- anywhere → Spam
Train as spam. - Spam → anywhere
Un-train as spam, train as ham. - MaybeSpam or ProbablySpam → anywhere except Spam
Train as ham. - anywhere → Trash
No training.
(In a next version of this document I will add some more actions. But for now this will have to do, because I feel I’m reaching the end of this page, and I’m really eager to publish. Writing this took a lot more time than it took to implement it.)
First, we create another directory:
root# mkdir /usr/lib/dovecot/sieve/pipe
And then we save this as /usr/lib/dovecot/sieve/pipe/bogofilter-train.sh:
#!/usr/bin/env bash
# Train Bogofilter.
# For details, see https://www.ohreally.nl/2018/11/19/mail-server/
######################################################################
# Copyright (c) 2018 Rob LA LAU <mailtuto@ohreally.nl>
# This work is licensed under the
# Creative Commons Attribution-ShareAlike 4.0 International License.
# To view a copy of this license, visit
# http://creativecommons.org/licenses/by-sa/4.0/
# or send a letter to
# Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.
######################################################################
######################################################################
# CONFIGURATION
# These should be absolute, as there is no ${PATH}!
BOGO=/usr/bin/bogofilter
CAT=/bin/cat
# END OF CONFIGURATION
######################################################################
case $1 in
[sSyY]*)
# spam, SPAM, Spam, soYouThinkYouCanSpam, yes, YES, etc.
FLAG="-s"
;;
[hHnN]*)
# ham, HAM, Ham, no, NO, nHesitezPasAMEnvoyerUnAutreEmail, etc.
FLAG="-n"
;;
[rRuU]*)
# retrain, RETRAIN, Retrain, untrain, UNTRAIN, UpsIDidItAgain, etc.
FLAG="-Ns"
;;
*)
exit 1
;;
esac
# Fork (&), so Bogofilter doesn't block the copy.
"${CAT}" | "${BOGO}" ${FLAG} &
This script expects an email on standard input, and 1 command line argument, which may start with a case-insensitive ‘s’ or ‘y’ to indicate spam, a case-insensitive ‘h’ or ‘n’ to indicate non-spam, or a case-insensitive ‘r’ or ‘u’ to indicate that the message should be retrained, because it was moved from the Spam box to elsewhere. It will then redirect it’s STDIN to Bogofilter.
Obviously, it needs to be executable:
root# chmod 755 /usr/lib/dovecot/sieve/pipe/bogofilter-train.sh
Then the Sieve scripts that will call that shell script.
Another directory, why not.
root# mkdir /usr/lib/dovecot/sieve/imap-before
And the Sieve scripts:
# /usr/lib/dovecot/sieve/imap-before/train-spam.sieve # * -> Spam require ["copy", "imapsieve", "vnd.dovecot.pipe"]; pipe :copy "bogofilter-train.sh" ["spam"];
# /usr/lib/dovecot/sieve/imap-before/train-ham.sieve
# MaybeSpam|ProbablySpam -> !Spam
require ["copy", "environment", "imapsieve", "variables", "vnd.dovecot.pipe"];
if environment :matches "imap.mailbox" "*" {
set "box" "${0}";
}
if not string :is "${box}" ["Spam", "Trash"] {
pipe :copy "bogofilter-train.sh" ["ham"];
}
# /usr/lib/dovecot/sieve/imap-before/untrain-spam-train-ham.sieve
# Spam -> *
require ["copy", "environment", "imapsieve", "variables", "vnd.dovecot.pipe"];
if environment :matches "imap.mailbox" "*" {
set "box" "${0}";
}
if not string :is "${box}" ["Spam", "Trash"] {
pipe :copy "bogofilter-train.sh" ["retrain"];
}
And then finally the Dovecot config to hook these scripts to our actions:
# /etc/dovecot/conf.d/90-sieve.conf
plugin {
sieve_extensions = +vnd.dovecot.filter +spamtest +spamtestplus +vnd.dovecot.environment +vnd.dovecot.pipe
sieve_plugins = sieve_extprograms sieve_imapsieve
}
# /etc/dovecot/conf.d/90-sieve-extprograms.conf
plugin {
sieve_pipe_bin_dir = /usr/lib/dovecot/sieve/pipe
}
# /etc/dovecot/conf.d/91-sieve-bogofilter.conf
protocol imap {
mail_plugins = $mail_plugins imap_sieve
}
plugin {
imapsieve_mailbox1_name = Spam
imapsieve_mailbox1_causes = COPY
imapsieve_mailbox1_before = file:/usr/lib/dovecot/sieve/imap-before/train-spam.sieve
imapsieve_mailbox2_name = *
imapsieve_mailbox2_from = Spam
imapsieve_mailbox2_causes = COPY
imapsieve_mailbox2_before = file:/usr/lib/dovecot/sieve/imap-before/untrain-spam-train-ham.sieve
imapsieve_mailbox3_name = *
imapsieve_mailbox3_from = MaybeSpam
imapsieve_mailbox3_causes = COPY
imapsieve_mailbox3_before = file:/usr/lib/dovecot/sieve/imap-before/train-ham.sieve
imapsieve_mailbox4_name = *
imapsieve_mailbox4_from = ProbablySpam
imapsieve_mailbox4_causes = COPY
imapsieve_mailbox4_before = file:/usr/lib/dovecot/sieve/imap-before/train-ham.sieve
}
root# rc-service dovecot reload
Like the Sieve script that checks the mails, the scripts here should also be compiled. And again, we cannot do this before we’ve made the changes to the Dovecot config, because the extensions and plugins won’t be available, yet.
root# sievec /usr/lib/dovecot/sieve/imap-before/train-spam.sieve root# sievec /usr/lib/dovecot/sieve/imap-before/train-ham.sieve root# sievec /usr/lib/dovecot/sieve/imap-before/untrain-spam-train-ham.sieve
And now you can start training your spam filter by moving mails around.
When you start with this setup, you will see that all mails are delivered to ProbablySpam. The more you train Bogofilter, the less mails will end up in ProbablySpam and MaybeSpam.
The first preparation step, at the top of this page, was to store your incoming spam in a separate folder. You can now move all of that to the Spam folder.
If you are 100% sure that your INBOX (or any other of your folders) is free of spam, you can do some ham training like this:
userx$ bogofilter -n -B /srv/mail/userx/cur userx$ bogofilter -n -B /srv/mail/userx/.AnyOtherFolder/cur
If, as root, you would like to do this for 1 of your users, you would do something like
root# su -c "bogofilter -n -B /srv/mail/userx/.SomeFolder/cur" - userx
or
root# bogofilter -n -d ~userx/.bogofilter -B /srv/mail/userx/.SomeFolder/cur root# chown userx:userx ~userx/.bogofilter/wordlist.db
To train mails as spam instead of ham, you would use ‘-s‘ instead of ‘-n‘.
Actions that are currently missing or not yet investigated:
- If I move a mail MaybeSpam|ProbablySpam → * → Spam, it is first learned as ham and then as spam, but it is never unlearned as ham, so my database is of by 1.
- What happens when mail is saved by the mail client to ‘special’ mailboxes? Like Sent, Drafts, Archives, etc. I don’t think anything funny happens here, but I haven’t looked into it.
- Some mail clients allow to save contacts/calendars on the IMAP server. What happens there?
- Not an action, but still something to look into: I think all Dovecot configuration regarding Bogofilter could be moved to 1 single file, and there is no need to split it up into 90-sieve.conf, 90-sieve-extprograms.conf and 90-sieve-bogofilter.conf. I think all those plugin{} blocks end up merged, but I haven’t yet been able to find anything about this in the Dovecot documentation.
Virus scanner
ClamAV consists of 3 parts: clamd, which is the virus scanner, freshclam, which is the application that periodically updates the definitions, and clamav-milter, the interface between Postfix and ClamAV. All three are started at the same time with the clamd init-script. That is, if we enable all three:
# /etc/conf.d/clamd START_CLAMD=yes START_FRESHCLAM=yes START_MILTER=yes #...
Then the clamav-milter config; only showing changes:
# /etc/clamav-milter.conf MilterSocket inet:8892 OnInfected Reject AddHeader Replace
If you decide to leave OnInfected at Quarantine, you will find some info on processing the ‘hold’ queue on the Postfix website.
And start:
root# rc-service clamd start root# rc-service clamd restart root# rc-update add clamd default
Why the start and then the restart, you ask? Well, that’s because for me /usr/sbin/clamd refused to start the first time I tried. The log file said something about a file that could not be opened, but it didn’t say which file. So I figured it might be a file that freshclam should create, but hadn’t yet because it had just been started for the first time. And that’s why I restarted, and apparently I was right, because the second time it worked.
BTW: clamd takes quite some time to start, but in the end it will get there.
And to hook it into Postfix, we only need to add the milter to the Postfix config:
# /etc/postfix/main.cf
smtpd_milters =
inet:localhost:8891
inet:localhost:8892
root# rc-service postfix reload
And we’re done. Send yourself a mail to verify that the headers are correctly added. Send yourself a virus to verify that it never arrives.
Protecting the sender’s privacy
Email clients, as well as Postfix, add information to outgoing emails that may be considered private information. Email clients often add a header which contains their name (why?!) and Postfix includes the sender’s IP address in the first Received: header. Luckily, Postfix also has the capability to remove all this information from outgoing mails.
If you really want to see what this does, send yourself a mail now, and another mail after implementing this, and then compare the headers of those mails.
First, we create a file containing regular expressions for the information we wish to remove from the outgoing mails.
# /etc/postfix/outgoing_headers /^\s*Received:[^\n]*(.*)/ REPLACE Received: from authenticated-user$1 /^\s*User-Agent:/ IGNORE /^\s*X-Enigmail:/ IGNORE /^\s*X-Mailer:/ IGNORE /^\s*X-Originating-IP:/ IGNORE /^\s*X-Pgp-Agent:/ IGNORE /^\s*(Mime-Version:\s*[0-9\.]+)\s.+/ REPLACE $1
The first of those lines replaces the sender’s IP address (the first line of the Received: header) with the string authenticated-user; obviously, you can replace this string with anything you like ($1 copies the other lines of this header, so do not remove this). The other lines remove client information; the last will leave the line “Mime-Version: 1.0“, but will remove client info if there is any.
And with that file in place, we can tell the submission service to use this file for a cleanup service. First the service must be added to master.conf; it can be added at the end or in the middle, but be careful not to add it in the middle of another service.
# /etc/postfix/master.cf privclean unix n - - - 0 cleanup -o header_checks=pcre:/etc/postfix/outgoing_headers -o nested_header_checks=
The empty nested_header_checks option makes sure that headers for attached emails are not touched.
And finally, add this service to the submission service; do not add it to the general smtp service, unless you want to remove this information from incoming mails as well.
# /etc/postfix/master.cf
submission inet n - n - - smtpd
-o …
-o …
-o cleanup_service_name=privclean
Restart Postfix, and send a mail to yourself to make sure everything works; if it doesn’t, you probably made a typo in one of the regular expressions.
Protecting the recipient’s privacy
Many mail clients permit the sender of a mail to request confirmation of the delivery of mails. This functionality is called Delivery Status Notification, or simply DSN, and is part of SMTP. It could, however, be undesirable to allow the sender of a mail to register at which exact moment it was delivered to a certain mailbox. Luckily, Postfix allows the sysadmin to disable this functionality.
# /etc/postfix/main.cf smtpd_discard_ehlo_keywords = silent-discard dsn
With this line in place, successful mail delivery will no longer be reported, even if the sender requested it explicitly. If a mail is rejected by the server, this is still reported to the sender.
Mind you: the sender can potentially still receive reports from other servers the mail passes through, possibly even stating that the message was delivered to this server; but this server will no longer report that the message was delivered to a certain mailbox (which, from the sender’s point of view, might just as well mean that the server discarded the mail after having accepted it initially, e.g. due to a spam or other filter).
For more info, see Postfix DSN Support.
DSN is not the same as read confirmation, which is configured in the recipient’s mail client.
Client configuration
The client configuration has gotten a bit burried in the text above, so I’ll repeat it here.
For receiving mail:
- server name: imap.example.com
- protocol: IMAP
- port: 993
- connection security: SSL/TLS
- authentication method: normal password (sometimes called plaintext or cleartext)
- user name: userx
- password: OnlyYouKnowThis
And for sending mail:
- server name: smtp.example.com
- protocol: SMTP
- port: 587
- connection security: STARTTLS
- authentication method: normal password (sometimes called plaintext or cleartext)
- user name: userx
- password: OnlyYouKnowThis
Autoconfig and Autodiscover
But wouldn’t it be cool if, instead of sending all that info to your users, email clients could configure themselves? That would undoubtedly prevent quite a number of typos and questions.
Well, some email clients can, actually! Thunderbird invented the Autoconfig XML file, and Outlook invented the Autodiscover XML file (this is the internet, so let’s not agree to a single standard, please…). As far as I know, Kmail has adopted Autoconfig; I don’t know about other clients.
Both methods rely on the fact that the domain in question (example.com, in our case) has a web presence ( https://example.com ). For both methods it may be interesting to generate the XML file using server side scripting languages like PHP or Python, instead of serving static XML; this is left as an exercise for the reader.
The below examples are both hosted at example.com, and they both point at imap.example.com and smtp.example.com. However, they may be hosted at one domain, and point at another. For example, if users with an @example.org address should use example.com servers for their mail handling, the below examples are hosted on the example.org domain, but point at the example.com domain.
Autoconfig
In the document root of the website, create a directory /.well-know/autoconfig/mail. In that directory, save the following file as config-v1.1.xml. This means that the complete URL for this file is https://example.com/.well-known/autoconfig/mail/config-v1.1.xml. Replace everything that is in bold (and nothing else).
<?xml version="1.0" encoding="UTF-8"?> <clientConfig version="1.1"> <emailProvider id="example.com"> <domain>example.com</domain> <displayName>Example.com Mail</displayName> <displayShortName>Example</displayShortName> <incomingServer type="imap"> <hostname>imap.example.com</hostname> <port>993</port> <socketType>SSL</socketType> <authentication>password-cleartext</authentication> <username>%EMAILLOCALPART%</username> </incomingServer> <outgoingServer type="smtp"> <hostname>smtp.example.com</hostname> <port>587</port> <socketType>STARTTLS</socketType> <authentication>password-cleartext</authentication> <username>%EMAILLOCALPART%</username> </outgoingServer> </emailProvider> </clientConfig>
Now, if the user, when configuring Thunderbird, tells the mail client that his or her email address is userx@example.com, Thunderbird will automatically configure itself to connect to the correct server with the user name userx (this is what the %EMAILLOCALPART% string does). This way, the user only has to remember his or her primary email adress, and the password that goes with it.
The above is a very minimal example; go to MDN for documentation.
Autodiscover
In the document root of the website, create a directory /autodiscover. In that directory, save the following file as autodiscover.xml. This means that the complete URL for the file is https://example.com/autodiscover/autodiscover.xml. Replace everything that is in bold (and nothing else).
<?xml version="1.0" encoding="utf-8" ?>
<Autodiscover xmlns="http://schemas.microsoft.com/exchange/autodiscover/responseschema/2006">
<Response xmlns="http://schemas.microsoft.com/exchange/autodiscover/outlook/responseschema/2006a">
<Account>
<AccountType>email</AccountType>
<Action>settings</Action>
<Protocol>
<Type>IMAP</Type>
<Server>imap.example.com</Server>
<Port>993</Port>
<DomainRequired>off</DomainRequired>
<SPA>off</SPA>
<SSL>on</SSL>
<AuthRequired>on</AuthRequired>
</Protocol>
<Protocol>
<Type>SMTP</Type>
<Server>smtp.example.com</Server>
<Port>587</Port>
<DomainRequired>off</DomainRequired>
<SPA>off</SPA>
<SSL>on</SSL>
<AuthRequired>on</AuthRequired>
<UsePOPAuth>on</UsePOPAuth>
<SMTPLast>on</SMTPLast>
</Protocol>
</Account>
</Response>
</Autodiscover>
Now, if the user, when configuring Outlook, tells the mail client that his or her email address is userx@example.com, Outlook will automatically configure itself to connect to the correct server with the user name userx (this is what the tag DomainRequired does). This way, the user only has to remember his or her primary email adress, and the password that goes with it.
The above is a very minimal example; go to Microsoft Docs for documentation.
Webmail
What’s missing now for this otherwise rather complete, professional and future-proof mail system, is a webmail client. But since I really don’t need webmail, I’m going to leave that up to you, the reader; I know that several good webmail clients, that support Sieve message filtering, exist.
If you decide to install a webmail client, and to document the choice and installation, please let me know, and I’ll gladly link to your article. (The interesting part would probably be changing the SMTP/IMAP password from the webmail client.)
Message to users
Okay, this has nothing to do with any of the above. I just thought it was cool.
And you deserve a reward for scrolling, and hopefully reading, all the way down here.
All work and no play makes Jack a dull boy.
Let’s create yet another directory:
root# mkdir /usr/lib/dovecot/post-login
Save this script as /usr/lib/dovecot/post-login/motd.sh, and make sure it’s executable:
#!/usr/bin/env bash
# MOTD over IMAP.
# For details, see https://www.ohreally.nl/2018/11/19/mail-server/
######################################################################
# Copyright (c) 2018 Rob LA LAU <mailtuto@ohreally.nl>
# This work is licensed under the
# Creative Commons Attribution-ShareAlike 4.0 International License.
# To view a copy of this license, visit
# http://creativecommons.org/licenses/by-sa/4.0/
# or send a letter to
# Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.
######################################################################
######################################################################
# CONFIGURATION
# A text file.
MOTD=/etc/dovecot/motd
# END OF CONFIGURATION
######################################################################
[ -f "${MOTD}" ] && {
shopt -s extglob
motd=`/bin/cat "${MOTD}"`
# Replace any sequence of whitespace with a single space character.
# This puts everything on a single line.
motd=${motd//+([[:space:]])/ }
# Remove leading space.
motd=${motd# }
# Remove trailing space.
motd=${motd% }
# Send to client.
printf "* OK [ALERT] %s\r\n" "${motd}"
shopt -u extglob
}
exec "$@"
Add the script to the Dovecot config:
# /etc/dovecot/conf.d/20-imap-postlogin.conf
service imap {
executable = imap imap-postlogin
}
service imap-postlogin {
executable = script-login -d /usr/lib/dovecot/post-login/motd.sh
unix_listener imap-postlogin {
}
}
And reload the Dovecot config.
Now, if you ever have a message for all of your users, you just create a text file /etc/dovecot/motd, and it will automatically be displayed to the user when she logs in.
The mail server will be offline for maintenance from 2 January 11:00 until 30 December 15:00.
And when the message is no longer valid, you delete /etc/dovecot/motd (and leave the above script in place).
Mind you:
- The message should be on a single line; the above script takes care of that.
- I have not been able to find a maximum allowed length for the message, but you probably shouldn’t make it an entire book.
- It seems that not every email client supports this, even if the RFC says that it MUST be supported.
- `man motd’ will tell you more about the Message Of The Day.
- Post-login scripts are not just for alerts; there’s much more you can do.
- Use absolute paths to binaries, because ${PATH} is not available.
- Some of the more interesting variables that are available, are ${USER}, ${HOME}, ${UID}, ${GID}, ${IP}; redirect the output of /usr/bin/env to a file to see them all.
- Unfortunately this cannot be used to get input from the user; I was hoping to use this to allow the user to change their password in a more user friendly way…
- More info on the Dovecot wiki.
Expanding
The above configuration is for 1 user running 3 domains.
But it’s extremely simple to expand.
Adding domains
Adding a domain only takes 3 steps:
- Correctly set up DNS for the new domain.
- Add the domain to the virtual_alias_domains parameter in /etc/postfix/main.cf.
- Add the domain and its aliases to /etc/postfix/virtual.
Do not forget to rebuild the virtual database (`postmap /etc/postfix/virtual‘) and reload the named and postfix configurations.
Adding users
Adding a user takes a few more steps, but most of these could be scripted.
First, add some stuff to the skeleton directory; this only needs to be done once:
root# cd /etc/skel root# mkdir -m 700 .bogofilter root# echo "" | bogoutil -l .bogofilter/wordlist.db root# mkdir sieve root# nano sieve/main.sieve # You could, for example, copy the Sieve script # from the chapter 'Spam filter' above. # Type Ctrl-X to save and exit. root# ln -s sieve/main.sieve ./.dovecot.sieve
Each new user will now automatically get a working basic Sieve setup and an empty Bogofilter wordlist on account creation.
Okay, the new user.
Since we already have a user called userx, it makes sense to call this new user usery.
Create the user and create some directories; this part could be turned into a small script.
root# useradd -G mailusers usery root# mkdir /srv/mail/usery root# mkdir /srv/mail/usery/.Spam root# mkdir /srv/mail/usery/.Spam/cur root# mkdir /srv/mail/usery/.Spam/new root# mkdir /srv/mail/usery/.Spam/tmp root# chown -R usery:usery /srv/mail/usery root# chmod -R 700 /srv/mail/usery
Then, add this user’s mail aliases to /etc/postfix/virtual (remember to run `postmap‘ afterwards).
Add the username to the list of mail passwords:
root# echo "usery:" >> /etc/passwd.mail
(That’s 2 angle brackets! You will have a huge problem if you use only 1!)
And finally, the user should log in to the server using SSH, to set their mail password.
usery$ sudo mailpass
And this last bit really feels like a slack end, so again: if you decide to develop a small web interface for this, please let me know, so I can link to it.
And that’s it
Congratulations: your mail server is ready to be taken into production.
Changelog
- 2021-04-02
Disabled Delivery Status Notifications. - 2020-08-18
Small clarification for Autoconfig / Autodiscover. - 2020-08-17
Snake noticed that I had forgotten to add the call to policyd-spf to the Postfix configuration. - 2020-08-06
Added the cleanup service to the Postfix configuration, to remove private information from outgoing mails.
A big thank you to Apvc for contacting me with this idea. - 2020-08-06
Added Autoconfig and Autodiscover, to automatically configure mail clients.
Many thanks to Snake for bringing this up (almost 2 years ago, for which I apologise sincerely). - 2020-03-21
Removed append_at_myorigin and append_dot_mydomain from the Postfix configuration. The Postfix documentation explicitly states that these should not be changed. - 2020-03-21
Added the unix_listener to the Dovecot configuration that allows Postfix to use SASL. - 2020-03-17
Removed the disable_plaintext_auth setting from /etc/dovecot/conf.d/10-auth.conf. Since an SSL/TLS encrypted connection is required, this setting is superfluous (although it won’t harm). - 2019-10-12
Changed owner for DKIM private keys. - 2019-08-16
Changed cert.pem to fullchain.pem in the Dovecot and Postfix configurations, to prevent certain mail clients (Claws and Trojita, anyway) from complaining.
Thanks again to Jim Brown. - 2019-08-16
Corrected a typo Jim Brown spotted in the Postfix config. - 2019-08-15
Corrected the sed command in the mailpass script to delimit the regular expressions using pipe characters instead of slashes, since slashes may clash with slashes used in the unix/crypt B64 alphabet used by bcrypt.
Thanks to Jim Brown for noticing and reporting this. - 2019-06-02
Changed cert.pem to fullchain.pem in the Apache configuration, to prevent certain browsers (Midori, anyway) from complaining. - 2019-04-28
Changed directory /service to /srv to comply with the Filesystem Hierarchy Standard. - 2019-03-17
Small rewordings. - 2019-01-13
Rewording, refining and typos. - 2019-01-13
Added renew hook for Let’s Encrypt certificates. - 2019-01-01
Rewording and typos. - 2019-01-01
Fixed a bug in the mailpass script. - 2019-01-01
Added berkdb USE flag to Postfix installation; apparently this is no longer included automatically. - 2018-11-19
Initial publication.
About the author
Rob was a freelance Linux/*BSD sysadmin, software and web developer, and IT consultant for 15 years.
In 2014 he retired, to walk from his home in the Netherlands to Santiago de Compostela in Spain, and he has been drifting since, mainly in France.
Even though he swore he would never return to an office job, he would be open to small but interesting challenges that could be attacked from within a caravan somewhere in the south of France.