Ultimate Guide to VPS Email Server Setup with Postfix, Dovecot, and Roundcube

Introduction

Running your own email server gives you full control, but requires careful setup to avoid spam blacklists. This guide covers a complete mail stack on a Hostxpeed VPS: Postfix (SMTP), Dovecot (IMAP), Roundcube (webmail), with spam/virus filtering, and authentication (DKIM/SPF/DMARC).

Prerequisites and Hostname Configuration

A VPS with at least 2GB RAM (NVME-1 recommended). Domain name (e.g., example.com). Ensure reverse DNS (PTR record) matches your VPS hostname (request from Hostxpeed support). Set hostname: sudo hostnamectl set-hostname mail.example.com. Edit /etc/hosts: 127.0.1.1 mail.example.com mail. Open ports: 25 (SMTP), 465 (SMTP SSL), 587 (SMTP submission), 993 (IMAP SSL), 995 (POP3 SSL optional).

Step 1: Install Postfix (SMTP)

sudo apt update && sudo apt install postfix -y. During install, select "Internet Site", system mail name = example.com. Main config: /etc/postfix/main.cf. Edit: myhostname = mail.example.com, mydomain = example.com, myorigin = $mydomain, inet_interfaces = all, mydestination = $myhostname, localhost.$mydomain, localhost, $mydomain, home_mailbox = Maildir/. Then sudo systemctl restart postfix. Test: telnet localhost 25 (EHLO test). Allow relaying for authenticated users only later.

Step 2: Install Dovecot (IMAP/POP3)

sudo apt install dovecot-core dovecot-imapd dovecot-pop3d -y. Edit /etc/dovecot/dovecot.conf: protocols = imap pop3 lmtp. Edit /etc/dovecot/conf.d/10-mail.conf: mail_location = maildir:~/Maildir. Edit /etc/dovecot/conf.d/10-auth.conf: disable_plaintext_auth = yes, auth_mechanisms = plain login. Edit /etc/dovecot/conf.d/10-ssl.conf: ssl = required, ssl_cert =

Step 3: Create SSL Certificate for Mail

Use Let's Encrypt: sudo certbot certonly --standalone -d mail.example.com. The certificate is at /etc/letsencrypt/live/mail.example.com/fullchain.pem and privkey.pem. Then update Postfix: smtpd_tls_cert_file, smtpd_tls_key_file. Update Dovecot similarly. Set up auto-renewal. For multiple domains, use wildcard or separate certs.

Step 4: Configure Postfix Authentication (Dovecot SASL)

Edit /etc/postfix/main.cf: smtpd_sasl_type = dovecot, smtpd_sasl_path = private/auth, smtpd_sasl_auth_enable = yes, smtpd_tls_auth_only = yes. Then /etc/dovecot/conf.d/10-master.conf: service auth { unix_listener /var/spool/postfix/private/auth { mode = 0660, user = postfix, group = postfix } }. Also enable submission on port 587: in master.cf, uncomment submission lines with -o syslog_name=postfix/submission, -o smtpd_tls_security_level=encrypt, -o smtpd_sasl_auth_enable=yes. Restart both services.

Step 5: Create Email Users (Virtual Mailboxes)

Avoid system users. Use virtual mailboxes with MySQL backend. Install Postfix virtual: sudo apt install postfix-mysql. Create database: CREATE DATABASE maildb; GRANT SELECT ON maildb.* TO 'mailuser'@'localhost' IDENTIFIED BY 'password';. Create tables: domains, users, aliases. Configure Postfix to query these. This gets complex (full tutorial online). Simpler for small setups: use system users with sudo adduser user1 (but then each email user has shell login - security risk). Better use Dovecot virtual users with passwd-file. Example: /etc/dovecot/users file with username:{PLAIN}password.

Step 6: Install Roundcube Webmail

sudo apt install roundcube roundcube-mysql roundcube-plugins. During install, configure database. Configure Nginx to serve Roundcube: alias /var/lib/roundcube. Or use Apache if preferred. Access at http://mail.example.com/roundcube. Configure Roundcube to connect to local IMAP (localhost:143) and SMTP (localhost:587). Test login with created email user. Enable plugins (managesieve for filters, password for self-service).

Step 7: Spam and Virus Filtering (SpamAssassin + ClamAV)

sudo apt install spamassassin clamav-daemon. Configure Postfix to pipe through SpamAssassin: edit master.cf, add smtp inet ... -o content_filter=spamassassin. Then create spamassassin service. Also amavis (amavisd-new) is more integrated. Simpler: use Procmail with SpamAssassin. For high volume, consider Rspamd (better performance). Test with GTUBE pattern (spam test string). Add SpamAssassin to Dovecot sieve to move spam to Junk folder.

Step 8: DKIM, SPF, DMARC Setup

Install OpenDKIM: sudo apt install opendkim opendkim-tools. Generate key: opendkim-genkey -D /etc/dkimkeys/ -d example.com -s mail. Add to DNS: mail._domainkey TXT record with public key. Configure Postfix to sign outgoing mail. For SPF: add TXT record "v=spf1 mx a ip4:YOUR_VPS_IP -all". For DMARC: add _dmarc.example.com TXT "v=DMARC1; p=quarantine; rua=mailto:dmarc@example.com". Test using online tools (dkimvalidator.com). This prevents emails from going to spam.

Step 9: DNS Settings for Email

Ensure MX record points to mail.example.com (or your VPS IP). A record for mail.example.com. PTR (reverse DNS) set via Hostxpeed support (must match HELO hostname). Also add these to DNS: mail._domainkey TXT, SPF TXT, DMARC TXT. Check using dig: dig MX example.com, dig TXT mail._domainkey.example.com. Without proper DNS, recipients will mark email as spam or reject.

Step 10: Monitoring and Logging

Mail logs: /var/log/mail.log (or syslog). Monitor with logwatch: sudo apt install logwatch; sudo logwatch --service Postfix --detail High. Set up alerts for failed logins, delivery failures, high queue size. Use pflogsumm for daily summaries. For webmail, monitor Roundcube logs. Also check blacklists (mxtoolbox.com) monthly.

Common Issues and Solutions

Emails going to spam: check reverse DNS, DKIM signing, SPF, DMARC. Not receiving external mail: ensure firewall allows port 25 inbound, no block by ISP/Hostxpeed (Hostxpeed allows outgoing 25 but inbound? Usually allowed). Mail queue stuck: postqueue -p, then postsuper -d ALL (but investigate). Relaying denied: missing SASL authentication. Certificate errors: ensure SSL for all services, update hostname. High spam? Tune SpamAssassin thresholds.

Security Best Practices

Disable plaintext authentication (use STARTTLS). Enforce strong passwords. Rate limit SMTP auth attempts (fail2ban for postfix). Use postscreen to block spam early. Keep software updated. Monitor /var/log/mail.log for intrusions. Restrict mail user to virtual mailbox only (no shell). Use fail2ban for Dovecot and Postfix. Configure SPF reject at MTA level (check_policy_service).

Backup and Disaster Recovery

Backup /var/vmail (maildirs) and database (if used). Also backup /etc/postfix, /etc/dovecot, /etc/opendkim, SSL certs. Use rsync to remote. Test restore by setting up second VPS with same config, restoring data, and pointing MX to it. Practice quarterly.

Alternatives and Managed Email

Running your own email server is time-consuming. Consider transactional email services (SendGrid, Mailgun) for app emails, or Google Workspace/Microsoft 365 for business email. Use Hostxpeed only for incoming/outgoing of low volume (under 1k emails/day). For large volume (>10k/day), dedicated email service or high-spec VPS with MTA optimization (Postfix + Dovecot + Rspamd).

Conclusion

Setting up a full email server is achievable but requires careful configuration, especially DNS records. Hostxpeed VPS provides the resources and network (no port 25 blocks) to run email. However, consider managed email for business-critical communication. If proceeding, follow each step, test thoroughly, and monitor regularly. Use Dovecot and Postfix virtual user backend for scalability.