Consulting djbware Publications

Qmail Support

Note: This page is for historic reference only. Neither Qmail nor my patches for Qmail are under current development. A follow-up for Qmail is s/qmail which inherits all aspects of my development for Qmail (and much more).

Qmail

Qmail is a UNIX-based fast, secure, and uncompromised SMTP MTA.

System resources are moderate even under high E-Mail load and volume.

Qmail was developed by Dan J. Bernstein, and is well recommended internationally. In 2008 Dan Bernstein placed Qmail in the public domain.

D.J. Bernstein is Research Professor in the Department of Mathematics, Statistics, and Computer Science at the University of Illinois at Chicago and Professor at the Faculteit Wiskunde & Informatica at the Technical University of Eindhoven. Please have at look at his remarks on:

Qmail 1.03 was released in June 1998 and lacks support for some ESMTP enhancement which have been published since then, in particular SMTP Authentication and SMTPS/STARTTLS. However, some of the ESMTP enhancements are of very little use, in today's email world even counter-productive, and often weakly documented, as discussed in my

which was originally planned as part of my Qmail book. In addition, I have set up a particular email discussion list to deal with the following issues:

You can participate in those discussions subscribing to the email list, which is run by EZMLM, of course.

Note: The email list has been moved from 'qmail' to 'sqmail'; my new development.

Support

Since 1998 I install and maintain Qmail and in addition Qmail's E-Mail list manager EZMLM on commercial and non-commercial base.

High volume (> 1 mio emails/day), high efficient, and low latency (< 10 sec/email) mail servers can be achieved with Qmail and my recommended extensions on current hardware. This allows in addition a high efficient spam and virus protection with little headache and administration efforts.

Development

Qmail's home page as maintained by Russel Nelson Qmail Homepage is unfortunately out-of-date. Over the past ten years not only the public recognition of email (and its abuse) has been changed, but in addition the requirements for a SMTP MTA. However, by means of my extensions for Qmail, I not only keep track of current developments, but in addition try to incorporate the requirements for large email sites.

Currently under investigation & development:

Some major paradigm changes need to be considered:

My plans are to support fully IPv6 and TLS throughout. ucspi-tcp6 together with it's secure companion ucspi-ssl has already been finished; next is djbdnscurve6, completed by s/qmail which I consider as successor of vanilla Qmail.

Currently not supported:

Some of the major patches/extensions are available for vanilla Qmail:

Patches, mostly for qmail-smtpd:

SMTP HELO/EHLO Greeting delay

A flexible greetdelay approach is available in ucspi-tcp6. Within rblsmtpd a defined waiting delay can be defined after the SMTP client has initiated the SMTP session and before qmail-smtpd answers with "220 ESMTP". The delay is control as part of the rblsmtpd's options or via the environment variable GREETDELAY which is typically set by means of tcpserver's rule-set file. A value of GREETDELAY="30" will delay the qmail-smtpd's greeting for 30 seconds.

Time enough to discourage a typical Spambot's SMTP client from continuing the SMTP session (the session will be aborted by the client). Setting of GREETDELAY does not harm 'regular' mail clients, but will (currently) reduce Spam traffic by about 50%.

It has been argued, that the sleeping qmail-smtpd blocks local resources and it should be better to extend the algorithm to send the client a 55x reply (in case of protocol violations). However, one should keep in mind, that not only the resources on the mail server are used, but in addition those of the spam mail sender are blocked for that particular session. Thus, the GREETDELAY will not only save you for spam mails, but unlike Greylisting and/or filtering a la SpamAssassin, this is the only mean to really reduce the overall amount of spam because the timeslot required for the spam sender to deliver messages (whether successfully or unsuccessfully) is raised from typically one second to (<=) GREETDELAY seconds.

On the 2007th "Frühjahrsfachtagung" of the German Unix User Group (GUUG) in Berlin, I have given a brief presentation (in english) about pros'n'cons of GREETDELAY.

Note: In case you run redundant mail servers, you have to install the patch on every system.

A typical result looks like:

@4000000047c732fd3b15ec54 sslserver: pid 23324 from 24.107.74.16 @4000000047c732fd3b15fbf4 sslserver: ok 23324 mail.fehcom.net:217.172.188.134:25 24-107-74-16.dhcp.stls.mo.charter.com:24.107.74.16::2899 @4000000047c732fd3b160f7c rblsmtpd: 24.107.74.16 pid 23324: greetdelay: 50

In case you are nosey about the efficiency, you can download the little script greetdelay.ksh. This script only works in conjunction with the patched rblsmtpd and SPAMCONTROL > 2.5.

After invoking the script like

./greetdelay.ksh -v /var/log/qmail-smtpd/current

you get the following output (sample from today):

Greetdelay discouraged: @400000004964a20729258414 rblsmtpd: 208.116.174.226 pid 12246: greetdelay: 50 [typicalcocktail.com] Greetdelay discouraged: @400000004964a49602a67d5c rblsmtpd: 189.70.9.178 pid 12290: greetdelay: 180 [18970009178.user.veloxzone.com.br] Greetdelay discouraged: @400000004964abec38c07334 rblsmtpd: 81.214.243.141 pid 12596: greetdelay: 80 [dsl.dynamic81214243141.ttnet.net.tr] Greetdelay discouraged: @400000004964ac0213d02e84 rblsmtpd: 208.116.174.226 pid 12602: greetdelay: 50 [typicalcocktail.com] Greetdelay discouraged: @400000004964ac7e016cefd4 rblsmtpd: 66.180.215.253 pid 12690: greetdelay: 50 [immortelles.net] Greetdelay discouraged: @400000004964af750ff89d8c rblsmtpd: 189.18.243.251 pid 12736: greetdelay: 180 [189-18-243-251.dsl.telesp.net.br] Greetdelay discouraged: @400000004964b6eb1d2c410c rblsmtpd: 208.116.203.40 pid 13067: greetdelay: 50 [bourbonmisto.com Greetdelay discouraged: @400000004964bda005330c0c rblsmtpd: 208.116.203.40 pid 13280: greetdelay: 50 [bourbonmisto.com] Greetdelay discouraged: @400000004964c04517d61aac rblsmtpd: 208.116.174.226 pid 13407: greetdelay: 50 [typicalcocktail.com] Connections [We Jan 7 16:15:31 CET 2009]: 75 -- Greetdelayed: 39 -- SMTP Sessions: 36 [9 secs]

You may use the options '-v' for verbose, '-t' to be verbose and to resolve the tai64 timestamp. Further, the input files can be provided 'stacked', e.g. '"@* current"' to analyse all multilog generated files. The speed of the analysis depends of course on the size of your log files and the DNS resolution.

RECIPIENTS extension (0.7)

qmail-smtpd accepts messages if the SMTP domain part of recipient address ("RCPT TO: ") matches an entry in control/rcpthosts or control/morercpthosts.cdb. The existence of a mailbox/maildir for the corresponding SMTP recipient is checked later in the delivery chain. In case no Mailbox/Maildir exists, the message is bounced back to the SMTP sender ("MAIL From: "). For normal SMTP mail traffic thats fine as long as the rate of undeliverable messages dont exceed 10% and the sender is 'legitmate'; ie. exists.
Todays situation is different: Spam and Virus attacks with forged/faked sender addresses to a bunch of random recipient addresses yield a undeliverable rate up to 90%. Worse: The generated bounces will never reach the sender and a double-bounce is eventually send to the postmaster.

There is an ongoing need to facilitate the existence of the Recipient's mailbox (and thus the "RCPT TO:" address) during the SMTP dialogue. For large sites, this is a complex task, since often multi-tier systems are involved, and the required efficiency can only be guaranteed at the borderline (= Internet) MTA.

From the usage and administrative point-of-view, for any email user one has to setup up:

The different authentication methods may require different security frameworks, which depend in particular on the client. Not only that this raises the need for a central repository of the mail user's attributes for administration purposes, in addition the email system should be able to use the repository. While my RECIPIENTS extension makes qmail-smtpd aware of acceptable recipients, the knowledge of those recipients has to be kept in a local database.

The RECIPIENTS extension (0.7.2) (MD5: a39b001c4c049893049cc15677d9dab0) drops this requirement and provides a checkpassword compliant API to allow practically any source to be queried, preferable a LDAP directory (as well as Microsoft's ADS), though a SQL database may be fine as well (feel free to write your own PAM!).

The following PAMs are supplied:

Features:

None-RELAYLCLIENT mode: The recipient check is done for addresses accepted per control/rcpthosts or control/morerecpthosts.cdb, i.e. in a none-RELAYCLIENT case.

Virtual domain support: The lookup is triggered per domain; allowing different sources to consult with the additional capability of redundant severs, wilddomains and fall-thru mode for not-listed domains.

VERP support: Enabled by default. Recipient addresses included in the lookup database starting with the local part local-@domain are automatically recognized as VERP addresses.

Compatibility: RECIPIENTS 0.7 is backward-compatible with 0.4 (except for the wilddomain support). Probably, with some effort, all known recipient verifications methods known for qmail-smtpd (ie. validrcptto) can be incorporated easily.

The legacy, fastforward compatible cdb approach is supported unaltered. Thus, the contributions of some users, to build the lists of E-Mail addresses from local information, i.e. vpopmail's accounts are still valid.

Scripts to generate recipients.cdb:

Note: These scripts are included unaltered without a particular 'fitness' check.
Please check yourself, if they fulfill your requirements.

Warlord (1.4.0)

These days, MTA's are flooded with virus E-Mails in high frequency, resulting in considerable delays in E-Mail communication. Out-of-band E-Mail Anti-Virus processing is cost-intensive and bounces are meaningless and even counter-productive for forged "MAIL FROM:" addresses.

Based on the idea of Russell Nelson's (and Charles Cazabon's) so-called "qmail-smtpd-viruscan-1.1.patch" to filter Windows executables attached as BASE64 encoded MIME parts in the E-Mail, I created a "Warlord" (MD5:5f77272038774d1b1a3cf164dcb682a5) enhancement for qmail-smtpd.

"Warlord" enables a wire-speed filtering of E-Mails containing BASE64 encoded attachments with about 99,5% efficiency. The Warlord algorithm employs

Note: The WARLORD algorithm has been made "public domain" in the Unix magazine iX (issue November 2004) and is therefore not patent-able.

QHPSI (0.2.0)

The Qmail High Performance Scanner Interface (MD5: Sfabf359926197ccc0e18498ba3a7b6e2) is a new approach to an old problem:

While typically Virus Scanning is facilitated by means of more or less complex scripts (i.e. qmail-scanner, AMaViS), calling in a second or third stage one or more Virus Scanners, todays requirements are towards more efficiency. With QHPSI, you can directly invoke a Virus Scanner to scan messages in the Queue. No need for "staging areas", no need for external programs like reformime or ripMIME. Simply call your Virus Scanner for qmail-smtpd from the associated tcpserver cdb.

The QHPSI works exceptionally well with Clam AV's clamd/clamdscan. Simply use in your cdb :allow,QHPSI="clamdscan" and you are done. The entire power of QHPSI is exploited in SPAMCONTROL.

Version 0.9.x.y of ClamAV includes a bug, which prevents clamscan/clamdscan to log to STDERR. Here, you find a patch for clamav-0.9x.y (borrowing some code from 0.7x).

And I've implemented a new approach to cope with email sender (originator) verification:

'Mail From:' Address Verification (MAV)

There is an ongoing discussion, how to verify that a particular email sender (originator) is authorized to use a specific MTA (Mail Transfer Agent):

are the most prominent candidates, though only DKIM is deployed more commonly.
However, in any case, the burden to verify the sender is put on the recipient MTA's shoulder.

Reversely, my proposal 'Mail From:' Address Verification tries to actively label the email to originate authoritatively from a certain MDA (identified per IP/Domain name) or an (authenticated) User (or both).

Unlike eg. SPF, there is no heavy-weight patching of Qmail necessary. By means of the simple plug-in MAV 0.20 (MD5: 9f1b33a4ab14ea03caa0a1d94d3605fa) qmail-smtpd can be made verification aware for local addresses.

Those extensions in combination are part of SPAMCONTROL:

SPAMCONTROL

SPAMCONTROL is a major extension for Qmail, in particular for qmail-smtpd to permit an advanced controlling of incoming emails with respect to the SMTP envelope and the data.

SPAMCONTROL provides a full featured SMTP AUTH and ESMTP STARTTLS support for qmail-smtpd and qmail-remote as well for qmail-pop3d based on Scott Gifford's extension and is highly integrated with ucspi-ssl (> 0.8).

A qmail-queue.scan script is available to act as front-end for qmail-queue allowing to run Spam and Virus checking on a RAM-Disk. Further, SPAMCONTROL includes (almost) all the Qmail enhancements presented above.

The current version of SPAMCONTROL is suited for the AMD64 architecture and compiles with MacOS X/BSD's clang.

Yet more Qmail add-ons:

Any service -- and in particular -- mission critical services like email shall be setup such, to work in an un-attended, error-free manner. Qmail together with supervise and multilog provides this kind of service. However, monitoring and reporting is important as well.

qmail-mrtg

The log information written by qmail-send, qmail-smtpd, and qmail-popup/qmail-pop3d (as patched with SPAMCONTROL) are written by qmail-mrtg in a format to be used and displayed by MRTG.

qmail-mrtg 3.02 (md5: 24c9fe643d0bcd09b3a3d6786bc0974c) is now available in version 3.0 and can be used to analyse multilog's log output for:

qmail-mrtg follows a modular concept and can easily be extended to include other information. qmail-mrtg allows to customize the feeding period such, only the most recent logfile - current - can be read even for very busy servers. qmail-mrtg can read simultaneously multiple logfiles produced by several instances of qmail-smtpd, i.e. running on different ports.
The current version 3.0.2 fixes some counting bugs.

Here you find a living sample.

Note: In the forthcoming release of s/qmail qmail-mrtg will be integrated by default.