FEHCom
Consulting Qmail TIPN Publications
Spamcontrol QMVC Newanalyse Documentation

Qmail Support

Qmail

is a UNIX-based fast and secure SMTP MTA.

System resources are moderate even under high E-Mail load and volume.

Qmail was developed by Dan J. Bernstein, and is well recommended internationally. In 2008 Dan Bernstein placed Qmail in the public domain.

D.J. Bernstein is Associate Professor in the Department of Mathematics, Statistics, and Computer Science at the University of Illinois at Chicago. Please have at look at his remarks on:

Qmail 1.03 was released in June 1998 and lacks support for some ESMTP enhancement which have been published since then, in particular SMTP Authentication and SMTPS/STARTTLS. However, some of the ESMTP enhancements are of very little use, in today's email world even counter-productive, and often weakly documented, as discussed in my

which was originally planned as part of my Qmail book. In addition, I have set up a particular email discussion list to deal with the following issues:

  • SPAMCONTROL: release announcements, additional usage information, forthcoming developments
  • QMVC: release announcements, usage, additional templates.
  • NEWANALYSE: release information, particular customization.
  • Other topics: New viruses / WARLORD, SMTP Authentication, STARTTLS usage, MAV, documentation.

You can participate in those discussions subscribing to the email list, which is run by EZMLM, of course.

Support

Since 1998 I install and maintain Qmail and in addition Qmail's E-Mail list manager EZMLM on commercial and non-commercial base.

High volume (> 1 mio emails/day), high efficient, and low latency (< 10 sec/email) mail servers can be achieved with Qmail and my recommended extensions on current hardware. This allows in addition a high efficient spam and virus protection with little headache and administration efforts.

Development

An active community supports Qmail, as maintained by Russel Nelson's excellent Qmail Homepage, mentioning in addition my own developments. Over the past ten years not only the public recognition of email (and its abuse) has been changed, but in addition the requirements for a SMTP MTA. By means of my Qmail extension SPAMCONTROL, I not only keep track of current developments, but in addition try to incorporate the requirements of large email sites.

Currently not supported:

  • STARTTLS with qmail-remote: Due in SPAMCONTROL 2.6 (under development)
  • CRAM-MD5 for qmail-remote: Due in SPAMCONTROL 2.7
  • Greylisting: Currently no plans yet (rblsmtpd ?)
  • DKIM: Currently no plans (rblsmtpd ?)
  • IPv6: SPAMCONTROL 3.0.
  • Bounce Address Tag Validation (BATV): No plans.

Some of the major patches are available for vanilla Qmail:


Patches, mostly for qmail-smtpd:

SMTP HELO/EHLO Greeting delay

The qmail-smtpd.c_greetdelay patch adds a defined delay after the SMTP client has initiated the SMTP session and before qmail-smtpd answers with "220 ESMTP". The delay is control via the environment variable GREETDELAY, which is typically set by means of tcpserver's rule-set file. A value of GREETDELAY="30" will delay the qmail-smtpd's greeting for 30 seconds.

Time enough to discourage a typical Spambot's SMTP client from continuing the SMTP session (the session will be aborted by the client). Setting of GREETDELAY does not harm 'regular' mail clients, but will (currently) reduce Spam traffic by about 50%.

It has been argued, that the sleeping qmail-smtpd blocks local resources and it should be better to extend the algorithm to send the client a 55x reply (in case of protocol violations). However, one should keep in mind, that not only the resources on the mail server are used, but in addition those of the spam mail sender are blocked for that particular session. Thus, the GREETDELAY will not only save you for spam mails, but unlike Greylisting and/or filtering a la SpamAssassin, this is the only mean to really reduce the overall amount of spam because the timeslot required for the spam sender to deliver messages (whether successfully or unsuccessfully) is raised from typically one second to (<=) GREETDELAY seconds.

On the 2007th "Frühjahrsfachtagung" of the German Unix User Group (GUUG) in Berlin, I have given a brief presentation (in english) about pros'n'cons of GREETDELAY.

Note: In case you run redundant mail servers, you have to install the patch on every system.

Usage hint: As it has been pointed out by Matthew R. Dempsky, you can achieve the basic GREETDELAY functionality even without patching qmail-smtpd by means of the following call:

tcpserver 0 25 sh -c "sleep $GREETDELAY; exec qmail-smtpd"

rblsmtpd Patches

Dan Bernstein has designed rblsmtpd (as part of it's ucspi-tcp-0.88) as front-end to qmail-smtpd. Since vanilla qmail-smtpd does not include any DNS client capabilities, he decided to include the RBL DNS lookup (based on the incoming IP of the SMTP client) into a 'slim' forefront processor: rblsmtpd.

In fact, DJB decouples with this approach a 'heavy-weighted' SMTP interface with a light-weight DNS front-end (currently about 250 lines of code). One of the bottlenecks of this architecture is, that any decision -- whether to accept or to deny a SMTP connection -- is done on a 'binary' base. Currently, there is no way how the information gathered by rblsmtpd is forwarded to qmail-smtpd, though rblsmtpd employs the environment variable RBLSMTPD. However, this kind of information proliferation is required to allow rblsmtpd not only to do simple RBL lookup but rather to enable it to serve for DKIM and SPF queries, which needs some communication with the SMTP back-end (qmail-smtpd).

Of course, it is a natural choice to include GREETDELAY into rblsmtpd (Greetdelay MD5: 39ec9d9290da1fd9921104cb18ee2e73). This patch

  • removes the dependency of rblsmtpd w.r.t the default RBL rbl.maps.vix.com.
  • introduces a new flag "-w delay" to include a default delay -- and --
  • includes the new flag "-W" which fetches the value of 'delay' from the populated GREETDELAY environment variable (set by tcpserver or sslserver).
  • adds a particular line to the rblsmtpd output to tell the current GREETDELAY value for this connection.
@4000000047c732fd3b15ec54 sslserver: pid 23324 from 24.107.74.16
@4000000047c732fd3b15fbf4 sslserver: ok 23324 mail.fehcom.net:217.172.188.134:25 24-107-74-16.dhcp.stls.mo.charter.com:24.107.74.16::2899
@4000000047c732fd3b160f7c rblsmtpd: 24.107.74.16 pid 23324: greetdelay: 50

In case you are nosey about the efficiency, you can download the little script greetdelay.ksh. This script only works in conjunction with the patched rblsmtpd and SPAMCONTROL 2.5.
After invoking the script like

./greetdelay.ksh -v /var/log/qmail-smtpd/current
you get the following output (sample from today):
Greetdelay discouraged: @400000004964a20729258414 rblsmtpd: 208.116.174.226 pid 12246: greetdelay: 50 [typicalcocktail.com]
Greetdelay discouraged: @400000004964a49602a67d5c rblsmtpd: 189.70.9.178 pid 12290: greetdelay: 180 [18970009178.user.veloxzone.com.br]
Greetdelay discouraged: @400000004964abec38c07334 rblsmtpd: 81.214.243.141 pid 12596: greetdelay: 80 [dsl.dynamic81214243141.ttnet.net.tr]
Greetdelay discouraged: @400000004964ac0213d02e84 rblsmtpd: 208.116.174.226 pid 12602: greetdelay: 50 [typicalcocktail.com]
Greetdelay discouraged: @400000004964ac7e016cefd4 rblsmtpd: 66.180.215.253 pid 12690: greetdelay: 50 [immortelles.net]
Greetdelay discouraged: @400000004964af750ff89d8c rblsmtpd: 189.18.243.251 pid 12736: greetdelay: 180 [189-18-243-251.dsl.telesp.net.br]
Greetdelay discouraged: @400000004964b6eb1d2c410c rblsmtpd: 208.116.203.40 pid 13067: greetdelay: 50 [bourbonmisto.com]
Greetdelay discouraged: @400000004964bda005330c0c rblsmtpd: 208.116.203.40 pid 13280: greetdelay: 50 [bourbonmisto.com]
Greetdelay discouraged: @400000004964c04517d61aac rblsmtpd: 208.116.174.226 pid 13407: greetdelay: 50 [typicalcocktail.com]
Connections [We Jan 7 16:15:31 CET 2009]: 75 -- Greetdelayed: 39 -- SMTP Sessions: 36 [9 secs]

You may use the options '-v' for verbose, '-t' to be verbose and to resolve the tai64 timestamp. Further, the input files can be provided 'stacked', e.g. '"@* current"' to analyse all multilog generated files. The speed of the analysis depends of course on the size of your log files and the DNS resolution.

The sample above shows the reduction after the DNSBL lookup (Spamhaus) and the efficiency is in my case around 50%.

Note: Some versions of SPAMCONTROL use GREETDELAY as well, thus employing this patch would result in a 'double delay'.


RECIPIENTS extension (0.6)

qmail-smtpd accepts messages if the SMTP domain part of recipient address ("RCPT TO: ") matches an entry in control/rcpthosts or control/morercpthosts.cdb. The existence of a mailbox/maildir for the corresponding SMTP recipient is checked later in the delivery chain. In case no Mailbox/Maildir exists, the message is bounced back to the SMTP sender ("MAIL From: "). For normal SMTP mail traffic thats fine as long as the rate of undeliverable messages dont exceed 10% and the sender is 'legitmate'; ie. exists.
Todays situation is different: Spam and Virus attacks with forged/faked sender addresses to a bunch of random recipient addresses yield a undeliverable rate up to 90%. Worse: The generated bounces will never reach the sender and a double-bounce is eventually send to the postmaster.

There is an ongoing need to facilitate the existence of the Recipient's mailbox (and thus the "RCPT TO:" address) during the SMTP dialogue. For large sites, this is a complex task, since often multi-tier systems are involved, and the required efficiency can only be guaranteed at the borderline (= Internet) MTA.

From the usage and administrative point-of-view, for any email user one has to setup up:

  • a valid email address
  • a userid + password for SMTP Authentication
  • a userid + password for POP3 Authentication
  • a userid + password for IMAP4 Authentication
  • a userid + password for Web access
  • The different authentication methods may require different security frameworks, which depend in particular on the client. Not only that this raises the need for a central repository of the mail user's attributes for administration purposes, in addition the email system should be able to use the repository. While my RECIPIENTS extension makes qmail-smtpd aware of acceptable recipients, the knowledge of those recipients has to be kept in a local database.

    The RECIPIENTS extension (0.6.6) (MD5: b75db208ac6a327c3fa468064df14e92) drops this requirement and provides a checkpassword compliant API to allow practically any source to be queried, preferable a LDAP directory (as well as Microsoft's ADS), though a SQL database may be fine as well. A ldap_pam.pl (here: version 0.9.2) is included in the patch.

    None-RELAYLCLIENT mode: The recipient check is done for addresses accepted per control/rcpthosts or control/morerecpthosts.cdb, i.e. in a none-RELAYCLIENT case.

    Virtual domain support: The lookup is triggered per domain; allowing different sources to consult with the additional capability of redundant severs, wilddomains and fall-thru mode for not-listed domains.

    VERP support: Enabled by default. Version 0.6 has a different (now default) notion of VERP addresses. Recipient addresses included in the lookup database starting with the local part local-@domain are automatically recognized as VERP addresses.

    Compatibility: RECIPIENTS 0.6 is backward-compatible with 0.4 (except for the wilddomain support). Probably, with some effort, all known recipient verifications methods known for qmail-smtpd (ie. validrcptto) can be incorporated easily.

    RECIPIENTS 0.5 (0.5.21) can be used as a lagacy alternative (MD5: a9feb59be62560fd83253d47246ea70).

    The legacy, fastforward compatible cdb approach is still supported unaltered. Thus, the contributions of some users, to build the lists of E-Mail addresses from local information, i.e. vpopmail's accounts is still valid:

    Scripts to generate recipients.cdb:

    Note: These scripts are included unaltered without a particular 'fitness' check.
    Please check yourself, if they fulfill your requirements.


    Warlord (1.4.0)

    These days, MTA's are flooded with virus E-Mails in high frequency, resulting in considerable delays in E-Mail communication. Out-of-band E-Mail Anti-Virus processing is cost-intensive and bounces are meaningless and even counter-productive for forged "MAIL FROM:" addresses.

    Based on the idea of Russell Nelson's (and Charles Cazabon's) so-called "qmail-smtpd-viruscan-1.1.patch" to filter Windows executables attached as BASE64 encoded MIME parts in the E-Mail, I created a "Warlord" (MD5:5f77272038774d1b1a3cf164dcb682a5) enhancement for qmail-smtpd.

    "Warlord" enables a wire-speed filtering of E-Mails containing BASE64 encoded attachments with about 99,5% efficiency. The Warlord algorithm employs

    • a robust MIME type identification. Particular MIME types can be added on-the-fly in a badmimetypes.cdb and the badmimetype check is enabled, invoking the environment variable BADMIMETYPE,
    • a high efficient and unique Loader type recognition. Specific Loader types (i.e. for the Windows OS) can be included in a badloadertypes.cdb and the badlodertype check is employed, if the environment variable BADLOADERTYPE is set with a characteristic pattern (i.e. BADLOADERTYPE='M'),
    • additional capability to reject a BASE64 encoded attachment, in case it includes white spaces (spaces or tabs) which is typically the case for malware. In order to employ the check, use BADMIMETYPE='!' .

    Note: The WARLORD algorithm has been made "public domain" in the Unix magazine iX (issue November 2004) and is therefore not patent-able.


    QHPSI (0.2.0)

    The Qmail High Performance Scanner Interface (MD5: Sfabf359926197ccc0e18498ba3a7b6e2) is a new approach to an old problem:
    While typically Virus Scanning is facilitated by means of more or less complex scripts (i.e. qmail-scanner, AMaViS), calling in a second or third stage one or more Virus Scanners, todays requirements are towards more efficiency. With QHPSI, you can directly invoke a Virus Scanner to scan messages in the Queue. No need for "staging areas", no need for external programs like reformime or ripMIME. Simply call your Virus Scanner for qmail-smtpd from the associated tcpserver cdb.

    The QHPSI works exceptionally well with Clam AV's clamd/clamdscan. Simply use in your tcpserver's cdb :allow,QHPSI="clamdscan" and you are done. The entire power of QHPSI is exploited in SPAMCONTROL.

    Version 0.9.x.y of ClamAV includes a bug, which prevents clamscan/clamdscan to log to STDERR. Here, you find a patch for clamav-0.9x.y (borrowing some code from 0.7x).


    And I've implemented a new approach to cope with email sender (originator) verification:

    'Mail From:' Address Verification (MAV)

    There is an ongoing discussion, how to verify that a particular email sender (originator) is authorized to use a specific MTA (Mail Transfer Agent):

    are the most prominent candidates, though only SPF is deployed more commonly.
    However, in any case, the burden to verify the sender is put on the recipient MTA's shoulder.

    Reversely, my proposal 'Mail From:' Address Verification tries to actively label the email to originate authoritatively from a certain MDA (identified per IP/Domain name) or an (authenticated) User (or both).

    Unlike eg. SPF there is no heavy-weight patching of Qmail necessary. By means of the simple plug-in MAV 0.19 (MD5: c1ef96d37a060024776ab69b87d01296) qmail-smtpd can be made verification aware for local addresses.


    Those extensions in combination are part of SPAMCONTROL:

    SPAMCONTROL

    SPAMCONTROL is an add-on (patch) for mainly qmail-smtpd for advanced control of incoming emails with respect to the SMTP envelope and the data.

    Version 2.5 includes significant enhancements in particular for qmail-remote:

  • QMTP client based on qmtproutes.
  • Possibility, to raise serveral qmail instances to allow in particular a dedicated delivery based on smtproutes and qmtproutes.
  • Define a particular host for qmail-send generated bounces.
  • SPAMCONTROL provides ESMTP STARTTLS support for qmail-smtpd based on Scott Gifford's extension for SuperScript's sslserver as part of ucspi-ssl.

    In addition, a script is available to act as front-end for qmail-queue allowing to run Spam and Virus checking on a RAM-Disk. Further, SPAMCONTROL includes all the Qmail enhancements presented above.


    Yet more Qmail add-ons:

    QMVC

    QMVC is a Korn-Shell Plug-In for qmail-local to enable filtering and Virus scanning of locally delivered E-Mails.

    Unlike AMaVis or Qmail-Scanner, it is entirely designed for Qmail and does not require additional patches.

    Newanalyse

    Newanalyse is a set of small programs helping the sysadmin in his task of processing and automatically rotating multilog generated logfiles.


    Documentation

    Unfortunately, my forthcoming book about Qmail, Daemontools, UCSPI, and DJBDNS in German language has been canceled by the publisher.

    However, I will continue in that project (though with less intensity) and put my material on my Qmail Documentation web page.

    [Impressum]

    [FEHCom]

    [top]