|
|
SMTP Authentication [Tutorial](last edit: 2010-02-03 -- this page exists since 2003) Consequences for the ESMTP procedure Authentication and Transport Layer Security Setting up qmail for SMTP Authentication IntroductionSMTP Authentication is a scheme which was introduced in 1999 by J. Myers of Netscape Communications and finally released as RFC 2554 ("SMTP Service Extension for Authentication"). It is partly based on the SMTP Service Extensions as defined in RFC 1869. Most modern SMTP implementations support SMTP Authentication, whereas Qmail 1.03 does not (without a patch). On the other hand, a lot of Mail User Agents (MUAs) - which include a SMTP Client - make SMTP Authentication available (e.g. Outlook, Eudora, Netscape, Mozilla, The Bat! ....). SMTP Authentication is advertised by the SMTP Authentication server, requires a client to authenticate, while finally both parties have to mutually accept and support the chosen authentication procedure. Originally invented as a Host-to-Host protocol, with SMTP Authentication, a User has to identify itself and after successful authentication, reception/transmission of his/her emails is granted. RFC 2554 does not explicitly state, what advantages/benefits a user has being SMTP authenticated, except that optionally a "security layer" for subsequent protocol interactions may be chosen. However, in common sense, an authenticated user is allowed for email transmission not only to the target system (the SMTP server) but rather anywhere. In Qmail terminology, this is equivalent to a 'relayclient'. SMTP Authentication takes some ideas of the Simple Authentication and Security Layer (SASL) and does not fit well into the SMTP scheme, as will be outlined in this document. Request For CommentsIn order to understand SMTP Authentication, one has to work through several RFC, which seem to be unrelated in the first place. On the other hand, the most recent SMTP RFC 5321 and it's predecessor RFC 2821 (by John Klensin) now at least mentions the existance of SMTP extensions and - by the same token - requiring the 'EHLO' command introducing a SMTP transaction. Even after all those years, it really would be time, to have more coherent SMTP RFCs; see also the comments of Dan Bernstein about the " Klensin RFC". (E)SMTP is not that such a difficult protocol to not at least cover the basics in one document - while removing obsolete commands like VRFY and EXPN.
Consequences for the ESMTP procedureWhile SMTP Authentication has been introduced solely as a service extension, it actually touches the (E)SMTP protocol substantially, which is not yet fully documented/discussed.
ESMTP session stateWhile within SMTP basically only a transaction is usefully defined, we now need to care about an ESMTP session:
The current Klensin ESMTP draft RFC 5321 takes partially care of this. Obviously, Klensin did not read his own RFC carefully, because it mixes in the attached sample (taken almost unaltered from RFC 821) happily the terminology 'transaction' and 'session' (Appendix D.1.). However, the conceptual change is more severe. The problem here becomes virulent in case of a ESMTP reply code. Does the server's response now belong to a transaction, or the entire session ? One particular problem is the ESMTP error code 552:
Clearly, the first case is mailbox (and thus transaction) specific, while the second case is a policy limit as discussed further in RFC 5321:
It is obvious that the current scheme of (E)SMTP command related reply codes without telling whether it belongs to the session or a transction needs more refinement. Let's hope for RFC 10821. Mail Submission [RFC 4409]While the standard SMTP port 25 is used for unrestricted email reception, in particular DSL and cable providers would like to setup their MTAs for their customers on a different port and requiring ESMTP Authentication. According to RFC 4409, the mail submission port defaults to 587. A MTA listening on that port will demand a successful SMTP authentication prior of accepting the MAIL FROM: command; otherwise an error is issued:
Apart from that behaviour, an ESMTP MTA listing on the submission port is required to only implement (offer) a subset of ESMTP commands. This effectively separates the tasks of a (E)SMTP server to accept
Authentication FrameworkIt seems to be clear by know, that SMTP Authentication depends upon a patchwork of mechanisms/methods/procedures scattered over a wide range of RFC. Now, we have to go on and discuss the SMTP Authentication framework and will realize, that things are even more complicated. Server AnnouncementWe take a sample from RFC 2554. "S:" denotes the SMTP Server and "C:" the SMTP Client.
Here, RFC 2554 uses multiple values for the keyword AUTH as ESMTP command, which is allow by RFC 1869, however broke the parsing of several ESMTP client implementations. One work around is, to add artificially a "=" (equal sign) between the AUTH keyword and the value, eg. AUTH=LOGIN. AUTH mechanismsThere are three authentication mechanisms widely used for SMTP Authentication. In the documentation coming with the qmail-smtp-auth-patch by Krzysztof Dabrowski, an overview of MUAs and their AUTH mechanisms is provided (which I updated):
AUTH LOGINThe most common 'AUTH LOGIN' mechanism looks like this
From all the ESMTP Authentication mechanisms the offered, the client selects 'auth login'. The ESMTP server issues then a '334 VXNlcm5hbWU6' where 'VXNlcm5hbWU6' is a BASE64 encoded string 'Username:'. The client provides the BASE64 encoded user name and the sever responses with the request for the 'Password:' ('334 UGFzc3dvcmQ6'). In the sample above, random input is given and the server finally rejects the authentication request. AUTH PLAINAccording to IANA's documentation, the PLAIN Authentication is defined in RFC 2245 "Anonymous SASL Mechanism". However, a more useful explanation of the PLAIN Authentication can be found in RFC 2595 "Using TLS with IMAP, POP3 and ACAP" (chapter 6):
In other words, the correct form of the AUTH PLAIN value is 'authid\0userid\0passwd' where '\0' is the null byte. Some ESMTP AUTH PLAIN implementations don't follow that procedure completely. We see that in the trace using Netscape's 4.8 MUA connecting to a modified Qmail 1.03 to do PLAIN authentication:
In this sample, the user name was 'test' and the password 'testpass'. Here, the Netscape client immediately blasts the authentication information to the server (including the artificial authorization identity 'test') without waiting for the server to announce his SMTP Auth capabilites. AUTH CRAM-MD5While for AUTH PLAIN and LOGIN clear user names and password are transmitted, things go significantly more secure with the CRAM-MD5 authentication mechanism. As already mentioned in it's name, CRAM-MD5 combines a Challenge/Response mechanism to exchange information and a (cryptographic) Message Digest 5 algorithm to encrypt important information. I use an example based on a posting of Markus Stumpf to the Qmail mailing list. A typical ESMTP AUTH CRAM-MD5 dialog starts like this:
Unlike AUTH LOGIN, the server's response is now a one-time BASE64 encoded 'challenge'. The challenge 'PDI0NjA5LjEwNDc5MTQwNDZAcG9wbWFpbC5TcGFjZS5OZXQ+' translates to '<24609.1047914046@popmail.Space.Net>'. The leading and trailing brackets ('<', '>') are mandatory, as well the portion of the challenge which provides the hostname after the '@'. '24609.1047914046' is a random string, typically build from the 'pid' and the current time stamp to make that challenge unique. The server's reponse includes both the username and
the digest. While the user name is transmitted in clear
text (but of course BASE64 encoded), the server's challenge is
used by the client to generate a 'digest' from the challenge
and the password (which is commonly called 'secret' or
'shared secret' in this context) and reads as: tim b913a602c7eda7a495b4e6e7334d3890The 'shared secret' following the username with an additional space is computed employing the following MD5 hashing algorithm:
If both the ESMTP server and the client 'share' the same challenge and secret, the user may now be authenticated successfully by means of the transmitted and BASE 64 encoded 'user name' and 'digest'. AUTH parameter as part of the 'MAIL FROM:' commandAccording to RFC 2554, authentication information can optionally provided as ESMTP AUTH parameter with a single value in the 'MAIL FROM:' command. The ESMTP AUTH parameter has to be used in the following way:
Here, the AUTH value has to be encoded inside an "xtext" as described in RFC 1891 "SMTP Service Extension for Delivery Status Notifications". RFC 2554 discusses the use of the optional AUTH parameter to the 'MAIL FROM:' command in the context of a "trusted environment to communicate the authentication of individual messages". It actually requires the proliferation of the AUTH information to another MTA (Mail Transfer Agent; eg. email gateway) as AUTH parameter when relaying the message to any server which supports the AUTH extension. In case the authentication is to weak, the Server should set 'AUTH=<>' as parameter to the 'MAIL FROM:' command. I am not aware, that any MUA implementation using the latter scheme however, some MTA (eg. Postfix) support it. Qmail
1.03, and in particular qmail-smtpd has no understanding
of any parameters in the 'MAIL FROM:' command; it lacks
a qualified ESMTP support in that respect. This holds in addition
for the ESMTP 'SIZE' announcement (RFC1870),
which was partially recovered by Chris Harris'
SIZE extension. Authentication StateAs outlined, RFC 2554 allows two distinct usages of the ESMTP AUTH extension:
Clearly, this has a significant impact on the authentication state itself. The first approach is actually equivalent with an authenticated SMTP session, while the second is effectively the authentication of the provided 'MAIL FROM:' sender and serves as 'informational' data. Unfortunately, RFC 2554 does not give any hints what an "authenticated" state really means. There is a common sense, that an authenticated user is allowed for unrestricted relaying. In case the authentication information is transmitted as extension to the 'MAIL FROM:' command, one may treat that equivalently with having an additional 'tcpremoteinfo' - usually provided by means of the 'ident' protocol. Authentication AbortsThe Client may cancel the authentication request, sending simply a '*' to the server. The server must reject the AUTH procedure and replying the SMTP protocol error '501'. However, the server has to cache the authentication method in order to preserve the state. Authentication FailuresThe server may reject the AUTH request by the client with the following response codes:
After a failed ESMTP request, the server has to reset it's state tables and the client may either provide the correct information, or may chose a different authentication mechanism, or may go on in un-authenticated state. Authentication proliferationIn general, SMTP Authentication allows a one-hop User-to-MTA authentication. An interesting case is to discuss Authentication proliferation. Let's first define what we are talking about: Typically, a User receives emails by means of the protocols POP3 or IMAP4. For sending, a useful approach would be, that the User - the email originator - sets up an email client (ie. Outlook) for SMTP Authentication and first connects to the Principal-MTA. Here, the user-id and password is stored; which is typically the same as the one used for the POP3/IMAP4 account. In this case, the Principal-MTA acts as SMTP-Relay. Now, we have User-to-MTA Authentication.It may be necessary to obey SMTP Authentication to the recipient's MTA or a further internal SMTP-Gateway, which connects to the Internet. Thus, we are talking about User-to-Principal-MTA-to-MTA SMTP traffic with the requirement of an authenticated communication chain. What shall this be good for? We have seen, that SMTP Authentication serves mainly to allow unrestricted relaying. With an End-to-End authentication, two additional aims could be achieved:
The latter is a requirement for the first, since it enables to reject emails with forged/spoofed "Return-Path" addresses. In order to maintain an authentication chain for the
User's MUA, not only the user-id and password has to be proliferated,
but rather in addition the "Return-Path" address. In
this respect, the Mail From: <Return-Path> acts
as authorization information. Authentication information in the email "Received:" header [RFC 3848]One - actually inadequate - attempt in this direction is to add authentication information into the email header, which is required by RFC 3848. The standard SMTP Authentication patches for qmail-smtpd incude the authenticated user equivalent to the tcpremoteinfo in the Received header:
Though the information erwin@fehcom.net@81.173.228.159 is rather precise, it lacks the knowledge, how it is derived. RFC 3848 requires a different notation, which is incorporated in my most recent SMTP authentication patches for qmail:
The keyword EMSTPA denotes "ESMTP Authentication"
and thus the information presented can be clearly interpreted.
However, the quality of this information can not be trusted,
if it does not originate from the last receiving host. Authentication and Transport Layer Security [RFC 4954]RFC 4945 is very strict about the use of unprotected Userids/Passwords during the SMTP Auth dialoge: If an implementation supports SASL mechanisms that are vulnerable to passive eavesdropping attacks (such as [PLAIN]), then the implementation MUST support at least one configuration where these SASL mechanisms are not advertised or used without the presence of an external security layer such as [TLS].Essentially, this REQUIRES from any ESMTP client and server:
If an SMTP client is willing to use SASL PLAIN over TLS to authenticate to the SMTP server, the client verifies the server certificate according to the rules of [X509]. If the server has not provided any certificate, or if the certificate verification fails, the client MUST NOT attempt to authenticate using the SASL PLAIN mechanism. See RFC 4945: 14. Additional Requirements When Using SASL PLAIN over TLS
Actually, I don't have the faintest idea, why this very strict
recommendation (which demands a server validation by means of
DNS) is part of this standard and is not expressed/referenced elsewhere. SummaryWe have seen by now:
ImplementationThere exist two major implementation concepts to be used in conjunction with SMTP Authentication:
Using Cyrus SASL authentication is done against the SASL database 'sasldb'. Entries (ie. the user base) there in are modified by means of the command 'saslpasswd'. The Cyrus SASL library supports different authentication methods, like LOGIN, CRAM-MD5, and others. In particular, a PAM may be referenced as external authentication method. The Pluggable Authentication Module (which actually
never matured as RFC) is a more general framework where the user
lookup is done against an arbitrary external module - the PAM.
The basic idea is, to transmit authentication information from
the network (ie. via qmail-smtpd) to the PAM. The PAM
checks the validity of the authentication information on it's
own behalf and and exits either with return code '0' in case
of successful authentication or with '1' (or non-zero value else),
if the authentication failed for some reason. Checkpassword InterfaceAs a generalization of the PLAIN authentication method, Dan Bernstein has defined a checkpassword interface to be used in particular for the combination qmail-pop3d and the auxiliary PAM checkpassword. Typically, a User receives emails by means of the protocols
POP3 or IMAP4.
For sending, a useful approach would be, that the User - the
email originator - sets up an email client (ie. Outlook) for
SMTP Authentication and first connects to the Principal-MTA.
Here, the user-id and password is stored; which is typically
the same as the one used for the POP3/IMAP4 account. In this
case, the Principal-MTA acts as SMTP-Relay. Now, we have User-to-MTA
Authentication. What shall this be good for? We have seen, that SMTP Authentication serves mainly to allow unrestricted relaying. With an End-to-End authentication, two additional aims could be achieved:
The latter is a requirement for the first, since it enables to reject emails with forged/spoofed "Return-Path" addresses. In order to maintain an authentication chain for the
User's MUA, not only the user-id and password has to be proliferated,
but rather in addition the "Return-Path" address. In
this respect, the Mail From: <Return-Path> acts
as authorization information. Authentication information in the email "Received:" header [RFC 3848]One - actually inadequate - attempt in this direction is to add authentication information into the email header, which is required by RFC 3848. The standard SMTP Authentication patches for qmail-smtpd incude the authenticated user equivalent to the tcpremoteinfo in the Received header:
Though the information erwin@fehcom.net@81.173.228.159 is rather precise, it lacks the knowledge, how it is derived. RFC 3848 requires a different notation, which is incorporated in my most recent SMTP authentication patches for qmail:
The keyword EMSTPA denotes "ESMTP Authentication"
and thus the information presented can be clearly interpreted.
However, the quality of this information can neither be verified
nor estimated, if it does not originate from the last receiving
host. ImplementationThere exist two major implementation concepts to be used in conjunction with SMTP Authentication:
Using Cyrus SASL authentication is done against the SASL database 'sasldb'. Entries (ie. the user base) there in are modified by means of the command 'saslpasswd'. The Cyrus SASL library supports different authentication methods, like LOGIN, CRAM-MD5, and others. In particular, a PAM may be referenced as external authentication method. The Pluggable Authentication Module (which actually
never matured as RFC) is a more general framework where the user
lookup is done against an arbitrary external module - the PAM.
The basic idea is, to transmit authentication information from
the network (ie. via qmail-smtpd) to the PAM. The PAM
checks the validity of the authentication information on it's
own behalf and and exits either with return code '0' in case
of successful authentication or with '1' (or non-zero value else),
if the authentication failed for some reason. Checkpassword InterfaceAs a generalization of the PLAIN authentication method, Dan Bernstein has defined a checkpassword interface to be used in particular for the combination qmail-pop3d and the auxiliary PAM checkpassword.
The advantage of the checkpassword interface is to be simply applicable for most authentication methods like CRAM-MD5 and for instance the POP3 APOP mechanism. In case of CRAM-MD5, the checkpassword string is:
Though Bernstein's checkpassword program is only suited for a local user lookup (via /etc/passwd or shadow passwd) and therefore requires to run under root, it's interface definition is widely deployed eg. in Vpopmail's vchkpwd. It should be noted, that checkpassword itself calls another (child-) program, typically qmail-pop3d. For SMTP Authentication this becomes obsolete, however the child program has to be supplied; otherwise the user validation will fail. A common choice is the program true (available as /bin/true or /usr/bin/true) which exits always '0'. qmail-smtpdAccording to the Russell Nelson's web site www.qmail.org, there are several SMTP Authentication patches available to qmail-smtpd:
qmail-remoteThe choices become very slim regarding a SMTP Authentication for qmail-remote:
Both version employ the ESMTP AUTH parameter as part of the 'MAIL FROM:' command; as has been discussed above; though with a wrong syntax and the email address instead the user name. Again, there is no good understanding what an "authentication state" may be and how to glue SMTP Authentication for (emails received by) qmail-smtpd and (send by) qmail-remote. The concept introduced in RFC 2554 may me useful for monolithic SMTP implementations like sendmail, but is very hard to sustain in cases where multiple tasks/users are involved. Qmail Authentication patchThis very unsatisfactory situation for Qmail can be relaxed
employing my combined Qmail
Authentication patch.
Combining authentication for qmail-smtpd and qmail-remote the user's authentication information can be preserved to some extend, if Qmail is acting as a relay. User DatabaseThere is very little common understanding, where to place the user data base for SMTP Authentication and how to construct it.
Apart from those details, the SMTP Auth user database could be a "local" database (Oracle, Mysql, Postgres) or could be "remotely" accessible by means of a LDAP lookup against a "centralized" database. However, the main task is to maintain a consistent user/password database for email:
ConclusionWhat is SMTP Authentication good for? The main reason is to
allow unrestricted relaying of emails for particular Users. Thus, SMTP Authentication complements/substitutes other administrative means to enable a controlled usage of the email system. Other means are for instance:
Most of those tools based on the knowledge of the IP/FQDN of the peer host, or - like my SPAMCONTROL patch - employ checks on the SMTP envelope information. Mostly, checks on the IP/FQDN/SMTP envelope have precedence over SMTP Authentication. Therefore, SMTP Authentication is an additional approach based on a User identification/authentication and is particularly well suited to support roaming Users. Patches & Programsqmail-authentication-0.6.9
qmail-smtpd-auth-0.5.9
qmail-smtpd-auth-0.4.3 - Updated and bug-fixed version of Krysztof Dabrowski's SMTP-Auth patch (MD5:f2653126515ca3ae26ff7d016a70663b). cmd5checkpw-0.30 - Adopted version of Krysztof Dabrowski's cmd5checkpw; the user base resides in /var/qmail/users/authuser. vchkpw.c.diff - against Vpopmail's 5.3.27 vchkpw to comply with the checkpassword interface for C/R requests. Base64-1.3 - a Base 64 converter for Unix (taken from John Walker). SPAMCONTROL - allows additional logging of qmail-smtpd sessions and the relevant SMTP Auth parameters. AddendumInter7 has incorporated the above patch for vchkpw into the current Vpopmail 5.4.x. Most of the current "big" qmail patches (e.g.. Bill Shupp's 'Qmail Toaster') include my SMTP Authentication patch, but not netqmail. Setting up qmail for SMTP AuthenticationFirst, patch Qmail 1.03 (or netqmail-1.0.6) with one of the Auth patches as provided above. Second, you need a PAM to allow authentication against a certain database. For small environments, cmd5checkpw-0.30 would be a useful choice, however for larger sites one of the following PAMs are more useful.
Third, you have to setup qmail-smtpd to accept SMTP Authentication.
Two drawbacks of the mentioned solutions exist: However, my SPAMCONTROL recognizes the environment variable SMTPAUTH, which can be used in the following way:TestingAs discussed, successful SMTP Authentication depends on the smooth interaction of three parties:
Apart from customization mistakes, in case of problems it is necessary to determine the chosen Auth mechanism (as discussed before) and to trace the (E)SMTP session. Dan Bernstein's recordio (part of his UCSPI) can be used in conjunction with a modified run script for eg. qmail-smtpd:
For testing purposes, this run script should be invoked in the foreground and the tracing apears on the TTY while a SMTP client is connecting to the server. The sample above can be used to trace SMTP Authentication against Vpopmail's vchkpw. In order to decode the BASE64 strings, one can use the base64 converter. It's important to understand, that for a correct decoding the trailing "\0" has to be included. Lets assume the username is "test" and the password is "testpass". In order to verify the en/decoding one should proceed as follows:
Thus, the user name "test" translates to "dGVzdA=="
and the corresponding password "testpass" becomes "dGVzdHBhc3M=". Note: It is important to use printf from the bash, thus no CR/LF characters are added which will happen employing echo instead of printf. |
- Supports in addition SMTP (client) authentication
for qmail-remote; complies to RFC 3848 and RFC 4409 (MD5:
9d75f4bf5bea6556497dafcc8df7feca).