Self-Hosted Email: Running My Own Mail Server Again

Overview

Most self-hosting guides say not to run your own mail server. I did it anyway. Mail for my domain is now received, filtered, stored and sent from a container on my Proxmox cluster. Outbound mail goes directly to recipient servers, with no third-party smart host.

The goals were:

• One place for all mail: personal mailboxes and the alert mail from every home-lab host
• Modern authentication and transport security (SPF, DKIM, DMARC, DNSSEC, DANE, MTA-STS)
• Spam filtering that learns from what I file as junk
• Monitoring good enough that I hear about problems before my mail stops arriving

Before cutover, an external deliverability test scored 10/10 and Gmail showed SPF, DKIM and DMARC all passing.

All names and addresses in this post are anonymised. example.com stands in for my real domain.

Architecture

The core is one unprivileged Debian LXC container on the server VLAN. Its mail store is on a separate volume. The firewall forwards the standard mail ports to it and preserves the client’s source IP, which the spam checks depend on.

The software is all standard Debian packages:

• Postfix for SMTP, with postscreen in front
• Dovecot with Pigeonhole for IMAP, LMTP delivery, SASL auth, Sieve and quotas
• SpamAssassin with Bayes and TxRep, connected through spamass-milter
• OpenDKIM, OpenDMARC and policyd-spf
• Unbound as a local recursive, DNSSEC-validating resolver
• certbot with DNS-01 validation, nftables, fail2ban, etckeeper and unattended security upgrades
• Grafana Alloy shipping logs and metrics to my existing Loki/Prometheus stack

Three Lanes Into Postfix

I found it easiest to treat Postfix as three separate front doors, each with its own rules.

Inbound (port 25). postscreen scores each client against several DNS blocklists, using weights and an allowlist. Clients that pass go to smtpd, which applies HELO, sender and recipient checks. SPF hard fails are rejected. Any message claiming to come from my own domain is rejected on this port, because my own users always send through authenticated submission. The milters then verify DKIM, evaluate DMARC and score the message. Obvious spam is rejected during the SMTP conversation. Borderline mail is tagged, and a global Sieve script files it into Junk.

Submission (587/465). These ports require a login. Each login may send only from its own addresses. The client’s Received: header is stripped so my home IP layout doesn’t leak in headers. The message is then DKIM-signed and delivered directly.

LAN notifications. Every host in the lab (Proxmox nodes, the NAS, about twenty containers) needs to send alerts. They use a separate listener that accepts only LAN sources and only a short list of recipients. It rewrites the sender to [email protected] and DKIM-signs the result. A Sieve rule then files each host’s mail into its own Notifications/hostname folder automatically. This listener is never port-forwarded.

Authentication and Transport Security

The full set of email authentication standards is in place:

• SPF limited to the mail host
• DKIM with an RSA-2048 key, rotated yearly using a new selector
• DMARC with aggregate reporting to a dedicated mailbox
• DNSSEC on the zone, signed by Cloudflare with the DS record at the registry
• DANE: a TLSA 3 1 1 record pinning the certificate’s public key
• MTA-STS: the policy is served by a Cloudflare Worker, so nothing extra is hosted at home
• TLS-RPT, so other servers report TLS failures to me

Outbound Postfix uses DANE where the recipient publishes TLSA and opportunistic TLS everywhere else.

Two details made DANE practical to run:

1. certbot reuses the key on renewal. The public key hash stays the same, so the TLSA record doesn’t change every 60–90 days.
2. A small script compares the published TLSA record with the live certificate key. A monitoring alert fires if they ever disagree.

Webmail

Roundcube runs in a separate container. The mail server stays as small as possible, and webmail can be patched or rebuilt without touching it. Roundcube connects to the mail server over IMAPS and submission like any other client. Its address book reads from my existing LDAP directory. Remote access goes through an authenticating proxy at the edge and then the reverse proxy in the DMZ, so the webmail container is never exposed directly.

Monitoring

Every two minutes a small exporter writes mail-specific metrics to a textfile: queue depth, oldest deferred message, certificate expiry, blocklist status, whether DKIM verifies against DNS, and whether TLSA matches. Grafana alerts on these, on stopped services, and on hosts that stop reporting. A daily summary email shows rejects, new Junk, near misses and DMARC report results. That is where I check for false positives.

Tradeoffs

Advantages

• Full control over filtering, retention, aliases and folders
• Alerts from every host land in one place, signed and pre-sorted
• Modern transport security, much of which many hosted providers still don’t offer for custom domains
• Everything is standard Debian packages with configuration tracked in etckeeper

Limitations

• Residential IP reputation is the main delivery risk. A large receiver could decide to junk my mail at any time. I have a fallback plan to route outbound mail through an authenticated relay, but it isn’t enabled.
• A single node. Proxmox HA covers a hardware failure, but a long outage means senders eventually give up.
• More responsibility. Certificates, keys, blocklists and DNS all have to stay in step, and getting it wrong means mail bounces.
• IPv4 only, for now.

Lessons Learned

• Dovecot’s newer config syntax is a big change. Most examples online use the old format and don’t work. I put the whole configuration in one file so I could read it from top to bottom.
• One OpenDKIM instance couldn’t tell my lanes apart. OpenDKIM never asks Postfix which listener a message arrived on, so a combined sign-and-verify instance didn’t behave as expected. I now run two instances, one for verifying and one for signing. Each listener picks a socket, and that choice decides whether mail is signed.
• Some Postfix restrictions are silently ignored. reject_unauthenticated_sender_login_mismatch does nothing on a listener without SASL. My port 25 anti-spoofing relies on an explicit sender-access check instead.
• Read the documentation for flags carefully. In policyd-spf, TestOnly = 1 is the enforcing setting.
• Run your own resolver. Spamhaus refuses queries from large public resolvers, and DANE needs local DNSSEC validation. I also had to stop DHCP from overwriting the container’s resolver setting.
• Keep mail DNS records unproxied. Proxying the MX hostname through a CDN breaks SMTP and IMAP and hides the IP that SPF and reverse DNS depend on.
• Undo DNSSEC in the right order. If you ever turn it off, remove TLSA first, then the DS record, and wait before you disable signing. The reverse order breaks the whole domain for validating resolvers.
• Hosts with an empty relayhost fail quietly. They try to deliver directly, get rejected by the anti-spoofing rule, and their alerts stop arriving without any obvious error. A standard per-host Postfix snippet fixed this.
• Unprivileged containers can cause odd problems. PHP’s PCRE JIT couldn’t allocate executable memory, which broke Roundcube until I turned JIT off.
• Reverse proxies and SNI. The reverse proxy was sending the internal hostname as SNI, so the backend routed requests to the wrong virtual host and returned 421 errors. Turning off upstream SNI and setting the Host header explicitly fixed it.
• fail2ban’s stock Dovecot filter expects the old log format. I had to write a new filter before IMAP brute-force attempts were banned.
• Migrate gradually. Keeping the old provider running in parallel, with a daily sync and a scripted MX rollback, made cutover day uneventful.

Running your own mail is still more work than paying someone else to do it. Done carefully, it is also one of the most educational projects I’ve run in the home lab.