SMTP and Transport Layer Security (TLS) [Tutorial]


2014-12-01 - 3rd revision

Chapter 1: Introduction

Chapter 2: Origin and Scope of the TLS Protocol

Chapter 3: Protocol and Companions: Request for Comments

Chapter 4: Transport Layer Security Framework

Chapter 5: Cryptographic Ingredients

Chapter 6: X.509 Certificates

Chapter 7: UCSPI-SSL

Chapter 8: Qmail and TLS

Chapter 9: Setting up qmail for Transport Layer Security

Chapter 10: Lose ends upon TLS cryptography

Chapter 1: Introduction

According to the volume of today's emails send and (perhaps) received, we can be make the following assumptions:

  1. [Undesired] Spam emails
  2. [Public] Emails from disclosed mailing lists
  3. [Confidential] Personal emails
  4. [Protocol] SMTP notifications (bounces, NDR = None Delivery Reports)

In our today's understanding personal emails need special care:

Figure 1: IT security's CIA paradigms

By construction, neither the MTA protocol SMTP nor the MDA protocols POP3 and/or IMAP4 provide this level of care and are simple 'plain' transport protocols. Without authentication SMTP is not even aware of an authenticated email originator.

In order to achiev confidentially, two different approaches are generally used:

Encryption with [S/MIME, PGP] can only be achived in case the originator as has access to the public key of the recipient which is on a large scale still difficult and only solved (to my oppion) in the PGP Pretty Good Privacy case.

Transport Layer Security TLS however, may work on both, the MTA and MDA level. It does not require any action from the user except from perhaps enabling the check-button 'use TLS encryption'. On the other side, the level of protection/confidentially/encryption is not easily visible for the communicating partners.

TLS touches the SMTP protocol significant - as SMTP Authentication does. Apart from it's transport layer encryption, TLS can be additional used for the following purposes:

Chapter 2: Origin and Scope of the TLS Protocol

Our current TLS protocol is the successor of the former SSL Secure Socket Layer protocol, whose implementation was mainly written by Eric A. Young for Netscape named SSLeay back in 1994.

At that time, cryptographers developed hybride encryption techniques based on

  1. Asymmetric Public Key/Private Key encryption like Diffie-Hellman and RSA to allow the exchange/secret agreement on a
  2. shared secret key for fast symmetric en-/decryption of bulk data enhancing the security with the additional capability
  3. of a keyed message integrity check (keyed HMAC).

Though not necessarly depending on that, in practice TLS utilizes heavily X.509 certificates which bind

Typically, TLS support is part of the operating system based on one of the following products, which come with a BSD, GPL, or commercial license:

Programming in 'C', developers need to build there software against one of those libraries. Since they always include complex crypto-libs and compression-libs, not only the 'safety' of the depending programs, but in addition the speed (and system resources) are critically impacted.

Often (and also in case of Qmail TLS and by most other MTAs) the OpenSSL framework is used. It has a good reputation regarding speed, certainly not the best concerning system usage, but posseses a well known history of bugs. Further, the documentation is still rather weak and some functions are not documented at all (sample: SSL_CTX_set_trust).

OpenSSL is considered to be the reference implementation for TLS though it includes the older SSL mechanisms for compatibility as well. OpenSSL posses a rich set of functionalities:

This list is not exhaustive, however.

Chapter 3: TLS Protocol and Companions: Request for Comments

The OpenSSL implementation has heavily influced the outline of the TLS RFCs which I reference for completeness:

Major # Minor # Name RFCs Comments
2 0 SSLv2    
3 0 SSLv3    
3 1 TLS 1.0 2246; 3268, 3749, 3943 AES encryption added; new compression method(s)
3 2 TLS 1.1 4346; 4279, 4366, 4492, 5077 Additional handshake methods, PSK usage defined; ECC added, stateless Session Resumption
3 3 TLS 1.2 5246; 5288, 5289, 5489, 5746, 5878, 6066, 6176 IDEA + DES deprecated + PRF function; ECCDH and GCM
DTLS 6375; 6083, 6520 Datagram RFCs + Heartbeat extension

Personally, I find the TLS RFCs hard to understand. In particular, because they don't provide any sort of state diagram for the TLS session (I will discuss this later) but rather use programming samples, derived directly from the OpenSSL source.

Though SSL and TLS are commonly understood as the same standard, there is however a noticable difference among both, which needs particular attention:

  1. TLS uses a common Pseudo Random Function PRF to calcuate the hashes for the KeyMaterial.
  2. TLS client message Certificate Verify does not include the MasterSecret in order to calculate the hash value.
  3. The hash value of the message Finished differs in construction.
  4. Variable padding of the message content in case of TLS.
  5. Different notion of client certificates.
  6. Substantially more Alarm messages for TLS.

Up to version 1.1 TLS was required to be backword compatible with SSLv2. From a security point of view, TLS did not provide better security than SSLv2 because it was always possible to allow a version roll-back even in the current session: session renegotiation. TLS 1.2 broke with this tradition; but typically needs to be recompiled without compatibility support.

There are also other cryptographic relevant RFCs for TLS. I just mention the following:

On the other hand, TLS depends strongly on X.509 certificates and the PKI Public Key Infrastructure, which are layed out in others sets of RFCs:

Chapter 4: Transport Layer Security Framework

I have already mentioned some protocol elements of the TLS framework like messages and procedures. In fact, TLS is rather complex and it couples

Since computing power for en/de-cryption is not any longer a significant concern, we can assume that

Another important aspect of the TLS protocol is due to it's asymmetrical behavior between the client and the server. The server is responsible for authentication and thus to presents his X.509 certificate and actively depends on the PKI framework, while the client acts normally explicitely anonymous. The validation and verification of a server certificate is relative new in Web browsers (as TLS clients) and still lacks a common user interface (and understanding). Thus, many other TLS clients (i.e. email clients) acting in a none-interactive way have no glues how to handle TLS ALARM and authentication failure messages.

The TLS protocol is situated entirely on top of the Transmission Control Protocol TCP and realizes additionally transport and session services, independently from TCP (see figure below).

Figure 2: Layer architecture of the TLS protocol

In this sense, TLS violates the OSI seven-layer model from a structural point of view. Even more severe, the OSI concept of Service Access Points SAP is not used. It was assumed, that TLS acts 'transparently' between TCP and the application layer. Given the fact, that a potential TLS user needs to know the security context, this design decision - which has a birthmark of SSL - does not fit well into our current understanding. Severe security leakages are opened up, since the application and the user does not receive a qualified feedback about the current security context from TLS.

With RFC 6375 TLS has been enhanced to support UDP as well. In order to mitigate the none-state behavior of UDP additionally a new support protocol was supplied: The Heartbeat (RFC 6520).
Of course changing a vehicle from a four-wheel system to a five-wheel system is not a trivial task and messed up entirely in OpenSSL due to it's broken memory manager. This gave rise to the so-called 'Heartbleed-bug' which compromised a lot TLS-protected internet sites. In particular -- since no counter-measure was available -- the bug could actively used to copy chunks of TLS memory from the server to malicious client 'steeling' potentially even the private key of the server.

It is not clear until now, why the IETF allowed this kind of (not validated) change to an Internet core protocol and why the Heartbeat extension was not disabled by default for TLS connections over TCP, where it doesn't make sense at all. Thus, it is better to keep the spare tire in the trunk and rather not mount it while driving.

4.1. Communication Security State

To my knowledge, there is no common understanding about a potiential communication security state. However, according to TLS the so-called Cipher Suite introduces partially such a concept.

I have already introduced the CIA security paradigm. Let's try to map that into to TLS world:

4.2. The 'Cipher Suite'

As we have seen, TLS provides in each aspect a set of technical solutions and propagates the chosen methods as 'Cipher Suite' coupling the following cryptographic and transmission elements (figure 3):

  1. Key exchange between partners
  2. Authentication (at least of the server)
  3. Symmetrical De/Encryption of message
  4. (Perhaps) Compression
  5. Message Authentication and Integrity

Figure 3: Interpretation of the TLS Cipher Suite

and is typically represented and transmitted in cleartext in a dash-format as: DHE-RSA-AES256-SHA or initially in an underscore format: TLS_RSA_AES256_MD5 depending on the TLS framework in use. The structure and the (valid) elements of Cipher Suite are identifed in RFCs and each combination is enumerated.

The final Cipher Suite is the result of the negotiation among both communication partners and can be influenced to some extend externally as parameter prior setting up the TLS connection. Though conceptually - coupling the transport and the session layers of TLS to be considered as 'Service Access Point' - the current choice of it is available only within the closed TLS connection. However, Apache has made the information available to the outside world by means of the MOD_SSL environment parameters, which are interrogatable by the application.

It needs to be mentioned, that the Change Cipher Spec protocol as part of TLS allows to change those Cipher Suites during the on-going session without explicit notification.

4.3. TLS Applications

Once a TLS connection is established, there could be up to 255 applications to use this 'session'. In practice, however, it is only one. The reason is, that an application either requests a TLS connection as client or is in listening mode.

The listening mode (server mode) can be established in three ways:

  1. The application is linked with the OpenSSL (or other implementation) library, thus 'speaks' the TLS protocol. In order to interprete the TLS messages correctly, a separate port has to used and commonly deployed (eg. HTTPS: port='443' instead '80').
  2. The application facilitates an underlying transport service which does in turn provides the TLS service transparently to the application:
  3. We use a 'trick' to switch to the TLS service while the application has already been started in native (unencrypted) mode: STARTTLS or STLS:
    • SMTP: RFC 3207 'SMTP Service Extension for Secure SMTP over Transport Layer Security'
    • POP3: RFC 5034 'The Post Office Protocol (POP3)'
    • IMAP4: RFC 2595 'Using TLS with IMAP, POP3 and ACAP'
    • LDAP: RFC 2830 'Lightweight Directory Access Protocol (v3): Extension for Transport Layer Security'
    The application needs to be explicitly enhanced to support STARTTLS/STLS which only has been defined for the very few protocols mentioned above. We discuss STARTTLS in case of SMTP in more detail below.

Most of the TLS-aware appplications follow the first approach, thus use directly e.g. the OpenSSL programming interface (API). For this purpose, OpenSSL provides special network functions (named BIO* routines) which can be called from the application directly, instead of the standard socket routines.

With respect to security, a few important aspects are touched:

  1. The whole TLS mechanism, including X.509 certificate treatment, takes place in the (user) context of the application. While this is fair for a TLS client application, this might be unacceptable for a TLS server. Thus, there is no isolation between the TLS context and the application itself. Given the fact that a TLS server application might posses (we see that later) several identities, it is is not easy to realize the required privacy with this approach.
  2. Since the implementation of in particular the OpenSSL libraries are changing in time (FreeBSD 8.1 still uses OpenSSL 0.9.8n instead of the most up-to-date 1.0.x library), we need to recompile and relink our application to cope with those changes. This is in particular required in case a security issue is identified in a certain version, which unfortunately happens.

4.4. TLS Session Layer and the Handshake

The Handshake is certainly the most complex part of the TLS implementation. It comprises of the

During the TLS Handshake the following tasks are commonly realized:

  1. Session management: TLS Session initialisation and establishment, session resumption and Cipher Suite (re-)negotiation.
  2. Key management: RSA/DH key negotation, choice of symmetrical en/decryption method and HMAC protocol, including key length for any of those.
    The actual keys are derived from a common KeyMaterial which is specific for the TLS connection.
  3. Authentication management: Choice of method (RSA/DSS) and key length, optional client authentication, optional additional external verification (eg. FQDN lookup).
  4. X.509 certificate management: Reading and parsing X.509 certificates and reading (perhaps keyword protected) private key, validating and verifying the partner certificate and it's legitimation with optional CRL evaluation and optional OCSP lookup.
  5. Identity management: 'Server Name Indication' according to RFC 4366, extended X.509 KeyUsage and others.

Neither the TLS RFCs, nor the available textbooks (see reference below) distinguish clearly among those tasks, while either delegating those to be the responsibility of the PKI or subsume those to be simply 'session' related. Of course, the notion of those tasks change in time (and are definitively different w.r.t. SSL 2.0) and depend on the application. Further, the tasks of the Session management differ significantly between client and server, which has in particular a severe performance impact.

Handshake Protocol

The Handshake protocol uses a set of Hello messages between both partners which initiate the TLS session, and after successful key exchange/negotiation and authentication finally establishes the session. Further, during the handshake the ChangeCipherSpec protocol is invoked to setup the Cipher Suite.

The following Handshake messages are defined, typically transmitted set-wise in TCP segments (figure 4):

Figure 4: Exchange of TLS handshake messages on top of TCP

Once the security context is defined, the Handshake protocol resumes and finally a SessionID is assigned which can used for a later (fast) Session resumption.

Session Management

Session Management can be considered as dynamical part of the Handshake protocol. It uses internally Key Management, Authentication management and X.509 certificate management and externally the ChangeCipherSpec protocol and in case of errors the Alarm protocol.

The German Wikipedia provides a nice (and correct) diagram which shows the dynamical flow of the TLS handshake including client authentication by means of a required X.509 certificate.

Session Parameter:

In addition to the connection related KeyMaterial (from which the MasterSecret is derived) the Session parameters contain the following information:

Stateful Session Resumption:

Stateful Session Resumption [servers side] only relies on the fact that the client in his Hello message a valid SessionID from a previous session. It depends on the setting of the flag 'is resumable'. Of course, the server needs to save the state of the previous session, in particluar the KeyMaterial for that connection. Using the stored MasterSecret the keys are recomputed together with new random variables for the new session.

Stateful Session Resumption is client triggered, thus the client can skip server authentication and directly move to the established session state.

Stateless Session Resumption:

Stateless Session Resumption [client side] is an extension to the generic TLS protocol and layed out in RFC 5077 'Stateless TLS Session Resumption'. Of course, it needs to be supported by the TLS server and client as well.

Basic building block is a Session Ticket which represents the session state including the following information calculated by the server:

Since this Session Ticket repesents the session state and is send to the client, it posses confidentially. Therefore, it is encryted in 128 bit AES using CBC mode and additionally MAC protected (HMAC-SHA-256).

In order to process the Session Ticket, additional information is required alongside with it:

This cryptographically secured ticket is only calculated by the server in case the client indicates the use of Session Ticket extension in the Client Hello message.

It should be noted, that the content of the Session Ticket is 'opaque' for the client, i.e. has no particular meaning. However, by means of the key reference the server is able to recognize the session and to recover the former session state without keeping it in store.
Further, upon setting up the new connection, the old Session Ticket is expired, and a new ticket (with the current connection parameters) is generated by the server and provided to the client within the message New Session Ticket.

Session Initiation:

By means of the discusses Session Resumption we now need to distinguish:

  1. [Full Handshake] The client has no Session Ticket or provided SessionID length is '0':
    Now, the usual session handling takes place, involving key management and authentication.
  2. [Abbreviated Handshake] The client has a Session Ticket or re-uses the previous SessionID:
    In case the TLS can be resumed, no further authentication is required (and thus avoiding expensive public key/private key calculation), key management is reduced to generate new session keys and the old security context can be re-established.

Session Establishment:

Session Establishment requires additional actions typically involving:

Once this is done, the Session Keys are used to en/de-crypt the following communication, thus the 'tunnel is closed'.

Session Termination:

Session Termination is facilitated by the Alert message Close Notify message. However, for many, in particular STARTTLS based communication, this is theory only, since they work with a mix of TLS and TCP connections.

Key Management

While the Session Management determines the dynamic behaviour of the handshake, the Key Management provides the algorithms and is responsible for it's cryptographical strength working 'under the hood' it realizes a solution for two tasks:

  1. How to generate the required keys, thus they can not easily guessed or intercepted ?
  2. How to synchronize the keys between the communication partners; avoiding the insecure transmission of keys ?

The strength of the TLS protocol regarding security results from the fact, that all relevant transport keys are derived uniformly and commonly for a particular session among the communicating partners from some cryptographic material. The better the KeyMaterial the better the cryptographic strength.
On the other hand, the (potential) knowledge of a (comprised) session key for one session does not necessarily impact the security of recent and/or future session. This behavior is called Perfect Forward Secrecy PFS, which is an important concept.

Key Generation:

TLS (version 1.2) provides now a common and established method to calculate the key material and obtain from here the required keys.

Since TLS supports a bunch of cryptographical algorithms with different length keys, version 1.2 of the protocol now finally uses a generic Pseudo Random Function PRF to compute the KeyMaterial in an iterative way. Both, the client and the server use for the calculation hash values starting the with the same input seed and thus obtain finally the same KeyMaterial.

In this sense, the KeyMaterial is never exchanged between the communication parterns, but rather uniformly and confidentially computed at each side. In order to make sure that both partners posses the same information, the procedure is devided into several steps:

  1. PreMasterSecret:

    RSA key exchange: The client generates the inital seed with 48 byte containing 2 bytes of the TLS version and 46 byte random number to be send to the server while encrypted with the server's public key.

    DHE key exchange: Starting from the DH domain parameters, the commonly computed DH key is used; however stripped from potentially '0's.

    Note: This is the only place in the TLS protocol, where the private key/public key algorithms are used solely for communication purpose.

  2. MasterSecret:

    The PreMasterSecret, the server random and the client random are concatinated, 'salted' with the characters 'A', 'BB', 'CCC' as labels, and SHA-1 hashed. Together with the PreMasterSecret those values are MD5 hashed and eventually build the 3*32 byte fixed-length MasterSecret.

  3. KeyMaterial:

    The PRF function uses now the MasterSecret, a seed which is required to be independently random generated from the client and the server, and one or optionally more fixed label(s):

    = MD5(MasterSecret + SHA(A + MasterSecret + ClientHello.random + ServerHello.random))
    + MD5(MasterSecret + SHA(BB + MasterSecret + ClientHello.random + ServerHello.random))
    + MD5(MasterSecret + SHA(CCC + MasterSecret + ClientHello.random + ServerHello.random))
    + ...

    The number of bytes of the KeyMaterial depends on the length of the requested keys: The longer the keys, the more often the PRF has to be invoked.

  4. Thus, the TLS implementation uses a deterministic approach to compute the KeyMaterial; the only secret component is the PreMasterSecret, which depends on the algorithm used for the key exchange.

  5. Keys:

    The commonly derived KeyMaterial is cut into chunks of bytes which are used as keys:

    1. chunk: Secret for the HMAC calculation (variable length):
      • Client MAC key
      • Server MAC key
    2. chunk: Secret for the symmetrical end/de-cryption (variable length):
      • Client Session key (Client Write)
      • Server Session key (Server Write)
    3. chunk: Initialisation Vector IV in case of CBC encryption (optional, variable length):
      • Client IV
      • Server IV

Figure 5: Generating the KeyMaterial via the PRF

Essentially, the Key Management has to provide sets of pseudo-random bytes which are used to derive the keys. It is assumed, that - given a random seed - hashing provides by means of it's avalanche effect enough entropie.

Perfect Forward Secrecy PFS:

Thus, once the TLS session is established, the transmitted data are en/de-crypted with different keys for the client and server. The same holds for calculating the MACs. Even if these keys may be stolen and thus the current TLS can be encrypted by a third party, it is (should) not be possible to use this information as a valid guess for other sessions. However, this depends strongly on the randomness of the initial sequences, which are critcal in this sense.

On the other side, upon session closer, the KeyMaterial needs to the wiped out from the memory. However, there are some C compiler on the market, which don't overwrite the (contiguous) memory even if instructed to do so. This is one fundamental security breach. Another problem occours between memory allocation and de-allocation, thus when the KeyMaterial is written or read. The involved program needs to take care, that any access to the KeyMaterial has to be programed as Critical and needs to be protected against race conditions, swapping and others run-time dependencies.

ChangeCipherSpec Protocol
Alert Protocol

4.5. The Record Layer

Connection states:

Chapter 6.1 in RFC 5246 introduces "Connection States" and describes those as "operating environment of the TLS Record Protocol"!

Pending states:
Initially, both client and server are in pending state exchanging Hello messages to initiate the session.

Initial Current states:
Since no security-context (this is the chosen Cipher Suite) is chosen yet, the entire communication taking place by means of the TLS Record protocol is none-encrypted, none-secured, and none-compressed. All relevant informations, including X.509 certificates and acceptable Cipher Suites are transmitted in cleartext over the network (and are interrogatable eg. by means of WireShark).

Current states:
Once the authentication of the partern (typically only the server) is verified, the ChangeCipherSpec protocol is used to setup the security context while chosing the Cipher Suite. At that point, the connection enters a 'current state'. The description of those tasks in the current RFCs (see table above) is done in programming language terms and it lacks (from my understanding) a qualified state scheme (compared to eg. the TCP protocol).

The Record Layer is actually the workhorse of TLS since does not only serves as data container but rather has to operate on the data prior of transmission; thus (figure 6b)

  1. fragment the appliation data into chunks (segments)
  2. provide the integrity by means of calculating the keyed MAC
  3. perhaps compress the data
  4. filling up the data for block-encryption with some padding
  5. encrypt the resulting data and
  6. finally encapsulate the data into the Record Layer Frame while
  7. adding the required Record Layer Header.

It is important to mention, that the Record Layer Frame is used even at the beginning of the TLS session; but now the data are Null-encrypted (figure 6a).

Figure 6: Tasks of the Record Layer protocol

Chapter 5: Cryptographic Ingredients

After this lengthy and theoretical discussion, let's focus on the most important cryptographical ingredients (and leaving the potential vulnerability due the complex setting out of scope for the minute). We need to consider:

  1. The handshake and thus how the sessions keys are mutually agreed upon.
  2. Verifying the server's authentity and it's legitimation.
  3. The message transport and how it is en/decrypted.
  4. Securing the integrity of the messages.

5.1. Key Exchange during the handshake

The TLS handshake implements the (complex) key exchange protocol. According to figure 7, the key exchange can be subdivided into the logical steps:

While it is believed, that the RSA method - factorisation into large prime numbers - will be broken by quantum computers, today's focus is Diffie-Hellman and here in particular Elliptic Curve Cryptography ECC.

Figure 7: Overview of asymmetrical encryption methods; El Gamal is not supported by TLS

RSA Key Exchange (RSA)

RSA is the Acronym of the names of Rivest, Shamir, Adleman who invented this classical asymmetrical encryption method. Unlike Diffie-Hellman, the algorithm relies on a 'real' 'private' and 'public' key. Thus, RSA can be used for three distinct cryptographic purposes:

  1. Encryption and Decryption of data, messages respectively.
  2. Signing of messages.
  3. Verification of signed messages.

The first task can be accomplished by means of ephemeral i.e. temporary, one-the-fly generated RSA keys (see figure 7), while the latter requires static keys, where in particular the public key is included in the server's X.509 certificate.
However, static RSA keys used for encryption/decryption purposes do not comply well with the aims PFS but are considerable cheaper for computation.

Let's briefly walk thru the RSA algorithm:

  1. The server choses two prim numbers p and q with considerable different (bit-)lengths and calculates n = p*q (the modulus) as well as the 'Euler' number φ(n) = (p-1)*(q-1).
  2. The next step is to find an odd number e by the server's which is 'relative' prime to φ(n) used as exponent.
    Both numbers n and e comprise the public key P(n,e) of the server - which can be published or send to the client.
  3. The sever's private key - we call it d - is computed by means of the Euclidean algorithm d = e-1 mod(φ(n)) resulting in the identity: d * e = mod(φ(n)).
  4. Once the client posses the server's public key a message N can be encrypted as C:   C = Ne mod(n).
  5. The server is now able to decrypt the message by means of: N = Cd mod(n).

The critical part of the algorithm is to find 'qualified' prime numbers p and q in the first place. Of course, the probability for the prime number's evaluation shall be uniformly distributed among all natural numbers. However, this assumption seems to be broken, as Bernstein and colleagues have shown recently. For very large numbers, a strict proof regarding their primality would be too computing-intensive so rather a statistical approach is used. However, the RSA mechanism works even if the found numbers are 'almost' prime only.

Mostly, the RSA algorithm is used with static keys. The 'ephemeral' variant was used in conjunction with the 'exported' version of TLS, where the RSA key was restricted to 512 byte only. For static RSA, the respective values of the public key can be found in the X.509 certificate as shown below. The Subject's public key is signed of course by the Issuer -- again with RSA.

The dynamical behavior of the RSA key exchange uses the following stages:

  1. The client sends a TLS Client Hello message to the server including a random byte sequence and a list of supported ciphers.
  2. The server responds with a Server Hello (figure 8a)
    • providing the server's X.509 cert (and perhaps the entire key chain) which can be used to proof the server's identity and of course including it's public key
    • a random byte sequence and
    • the chosen TLS cipher suite.
  3. Based on the exchanged random byte sequences the client generates the PreMasterSecret, encrypts it with the server's public key prior of sending it to the server by means of the message Client Key Exchange (figure 8b).
Figure 8a: Server Hello Message
Figure 8b: Client Key Exchange
Key Exchange using DH with Discrete Logarithm (DH)

Unlike the RSA cryptography, where only the server is involved, in DH both the client and the server need to compute some temporary, ephemeral parameters. DH key exchange includes the following steps:

  1. Before the server has been started, a so-called DH (initial) parameter file including the generator g and a large prime p is generated and peristantly stored and used for all later connections.
  2. Upon session initialisation, the server choses additionally a random value b (kept secret and being smaller than p) and calculates B = gb mod p sending it to the client (as 'pubkey') in addition with the public parameters g and p.
  3. Following the recipient of those values, the client randomly choses on-the-fly a large (secret) integer a < p-2 computing A = ga mod p ('pubkey') and sends it to the server.
  4. The common Key Exchange Key KexK is now KA = Ba mod p to be equal KB = Ab mod p.

In Diffie-Hellman, the values KA = KB - having a bit-length of p - are considered to be the PreMasterSecret for the following calculation of the MasterSecret.

We call this sequence Ephemeral Diffie-Hellman key exchange EDH, DHE respectively.

The security of the entire DH key exchanged is closely related to the randomness of the chosen parameters a by the server and b by the client. The bit-length of the DH domain parameters however, restrict the 'length-space'. A DH parameter of 512 bit corresponds to an upper decimal value of about 10154; while a 1024 bit key pushes the range up to 10308, which is *significantly* larger.

In plain DH there is no-such-thing like a 'private' and/or 'public' key, but rather a mutual agreement on the chosen parameters. However, the server typically posses a private/public key-pair which is used to authenticate the transmitted DH 'pubkey'.
The server's DH params g and p together with it's X.509 certificate are public and transmitted un-encrypted to the clients. The X.509 certificate can be used by the client to verify the server's authentity, since the DH params are additionally signed with the server's private key as well and additionally send to the client (as can be depict from the following Wireshark trace in figures 8c and 8d).

Figure 8c: DHE Server Key Exchange
Figure 8d: DHE Client Key Exchange

In case the server does not sign the DH params, we call this Anonymous Diffie-Hellman ADH. If so, the entire method is potentially subject of a Man-in-the-Middle MitM attack: A possible eavesdropper Eve can easily switch himself into the communcation between typically Alice and Bob and contemplate the entire handshake while modifying it on both sides and of cause reading the clear-text messages.

Key Exchange using DH with Elliptic Curves (ECC)

Elliptical curves are currently the best mean to provide confidentially for TLS.

Currently, within the IETF there is an intense discussion which curves and which parameters are best. Certainly, Dan Bernstein's Curve25519 is the most prominent one. Questions arise, which parms are best with respect to speed and least vulnerable against timing attacks.

State after the Key Exchange

At the end of the Key Exchange we have learned a lot about our partner - and in case we didn't like it - refused the further TLS connection:

The information labeled as [OPTION] depend of course on the TLS implementation and depend entirely on the X.509 certificate's content. We have learned however, that this [OPTION]al information may be forged and thus may lack reliability.

Furthermore, we have agreed with our partner upon the following attributes of our forthcoming communication:

The chosen paramaters are the Cipher Suite as we already know.

The Cipher Suite is public and essentially determined by the client. However, the cryptographically important information are sealed among the communication partners and are effectively the Master Secret from which the KeyMaterial ist derived deterministically.

5.2. Server Authentication and Legitimation

While authentication is based on the existence and (fitting) of the server's public and private key, legitimation is subject of an additional protocol and/or mechanism, known as Public Key Infrastructure PKI.

PKI is one of the heritages of TLS and developed side-by-side. In order to understand legitimation, we have to have a closer look into X.509 certificates, the information they provide, and how they are deployed. This is subject of the next chapter.

5.3. Message Encryption during transport

The TLS handhake - as discussed - takes places between the communication peers and is private. Thus, in fact never and never the cryptographic relevant information is leaked to anybody else and only interceptable if the server's private key is compromised.

TLS employs two different methods to encrypt the data (figure 9):

Often RC4 is used, since it requires less computing power. However, the security of this method - using a 56 bit key - depends not only on the IV but also on the generation of the following keys which might be biased.

According to my own analysis, DES ist not secure, due to the small key, and 3DES (using the scheme EDE Encryption-Decryption-Encryption with two keys only) is more resource-heavy than AES, unless the operation takes place in a dedicated part of the CPU.

Figure 9: Symmetrial encryption and their frameworks

From the crypt-analysis we know, that the One Time Pad - having the same length as the original message - is unbreakable (once the shared secret is unknown).

While stream ciphers act as 'pseudo OTP' generating the required cipher stream deterministically on-the-fly, the situation for block ciphers is different and two weaknesses arise:

5.4. Ensuring Message Integrity

Keyed Massage Authentication

Each TLS Record Layer Frame ist secured with a keyed Message Authentication Code MAC. Thus - taken from the KeyMaterial - each Service Data Unit SDU is 'salted' with a known MAC key and finally is subject of the hash function; typically (still) MD5 or Secure Hash SHA with it's different length. Thus, not only the integrity of the indivdual frame is guaranteed, but in addition it's authentity (figure 6a).

Cipher Block Chaining

Cipher Block Chaining 'entangles' frame n with the previous frame n-1 by means of an XOR operation. But what to do with the first block ? It needs - comparable with the stream ciphers - an Inititialisation Vector (figure 10)!

Figure 10: Example of Cipher Block Chaining with DES encryption

Now, the entire (encrypted) message can only decrypted, if all TLS Record Layer Frames are correctly received. However, there is a price to pay: The first frame is subject of the IV only, which might result in the BEAST attack; however does only work with TLS < 1.2.

Chapter 6: X.509 Certificates

At that point TLS could live - not considering the vulnerability of DH due to MitM attacks - without something we call Public Key Infrastructure PKI depending in practice on X.509 certificates. However, during the pioneering days of asymmetric cryptography authentication of the communication partner was a too tempting goal not be realize by the very same token.

During the last 20 years X.509 certificates were believed to be magical wand in order to secure Internet communication and is a conerstone of for your today's commercial use of the Internet. Not only 'big business' depends on cryptography (and laid down in the ANSI X9.63 standards) but it is additional attractive for crime and surveillance ....

6.1. X.509 Certificate as data container

Let's have a look into a X.509 certificate (figure 11). Basically, a certificate (cert) carries four indispensible pieces of information:

  1. The name (= identity) if the owner, called Subject of the cert and expressed as Distinguished Name DN.
  2. The public key of the Subject.
  3. The name of the Issuer who has signed this certs; again expressed as Distinguished Name.
  4. The Signature of the Issuer for this cert providing some kind of legitimation.

In addition, there are some other technical information included:

Furthermore, a certificate does usually indicate, what is it's purported usage:

Apart from these attributes, today significant information is provided by means of so-called Extended Key Usage EKU flags. For instance, a cert may be used for code-signing. An iOS or Android developer needs to sign it's software in order be be acceptable for the Operating System prior of installation.

If you are familiar how to break down the family of Dinosaurs into their zoological names, you probably don't have difficulties following the ASN.1 naming conventions for certs ... which is a pain in the neck for most programmers and leaves unfortunately a lot of ambiguities both in creating and also in interpreting the cert's attributes.

Figure 11: Basic ingredients of a X.509v3 certificate

6.2. Presentations of X.509 certificates and Key files

The structure (and the content) of X.509 certs are well described, however the presentation depends somewhat on the purpose. Typcially, we use certs in the following forms:

OpenSSL allows to transform and to use the certs in any format. The table below shows one certificate I ship with UCSPI-SSl:

openssl x509 -in ::1.pem -text -noout

Version: 3 (0x2)
Serial Number:
Signature Algorithm: sha1WithRSAEncryption
Issuer: C=US, ST=New York, L=Brooklyn, O=ucspi-ssl research laboratory, CN=ucspi-ssl research ca
   Not Before: Sep 27 19:03:54 2012 GMT
   Not After : Sep 25 19:03:54 2022 GMT
Subject: C=US, ST=New York, L=Brooklyn, O=ucspi-ssl research laboratory, CN=::1 Subject Public Key Info:
   Public Key Algorithm: rsaEncryption
   RSA Public Key: (1024 bit)
    Modulus (2048 bit):
    Exponent: 65537 (0x10001)
X509v3 extensions:
  X509v3 Basic Constraints: critical
  X509v3 Key Usage:
    Digital Signature, Key Encipherment
  X509v3 Extended Key Usage:
   TLS Web Server Authentication
  X509v3 Subject Alternative Name:
    IP Address:0:0:0:0:0:0:0:1, DNS:localhost
  X509v3 Authority Key Identifier:
    DirName:/C=US/ST=New York/L=Brooklyn/O=ucspi-ssl research laboratory/CN=ucspi-ssl research ca

Signature Algorithm: sha1WithRSAEncryption

openssl x509 -in ::1.pem -text


cat ::1.key


Though we used the openssl x509 routines to display the cert; since it is a .pem file we could simply use cat instead.

We notice, that the meta-information coming with the cert, like it's fingerprint, is evaluated by openssl independent of it's representation and will result always in the same values.

6.3. Authentication, Validation, Legitimation

Let's try to sort out some often used terms and explain those:

Authentication Generally spoken, a X.509 certificate binds an identity (named by it's Distinguished Name DN) with it's Public Key.
Authentication results from the possesion of the fitting private key by the identity and it's challenge by means of typically a NONCE message.
Validation I will call the checking of the syntactical correctness and (internal) consistancy of a (complex) X.509 certificate validation.
Each X.509 certificate needs to have a digital signature provided by the issuer. The issure's DN and the signature are mandatory parts of the certificate. The signature algorithm includes a hash value calculated either by means of MD5 or SHA/SHA256 and encrypted with with the issuer's private key.

A X.509 certificate includes a lot of other information to be subject of validation:
It's fingerprint, the purported usage, the validity period and others (extended usage).
Verification The X.509 certificate provides some information regarding the identity of the sender:
  • The Subject given as Distinguished Name perhaps including the FQDN or the email address.
  • The Alternative Subject Name (occassionally abbreviated SAN) carrying the FQDN, the IPv4 and/or IPv6 address of it's owner (for networking components).
This information can be cross-checked and verified against the (external) DNS and/or IP information.
Legitimation Finally, some X.509 certificates need to be trusted. Trust is provided by means of the knowledge of superior root certificate and is to some extend a question of 'policy'. Some so-called root certificates need to be present locally a 'trust-store' belonging to the application or the OS.
External 'trust stores' may be inquired by means of the Online Certificate Status Protocol OCSP in case the respective URL is part of the X.509 certificate itself.
'Trust' is a transitive attribute spanning a series of signed certificates. By means of a Certificate chain all intermediate X.509 certificates can be provided in a series.

6.4. 'root' Certificates, Signatures, and the Trust Chain

X.509 certificates which have the attribute CA:True can be used to sign subordinated certificates:

Figure 12: 'foul' trust chain

Though in general, the cert's signature includes enough (unique) crpytographic material, by means of the weak MD5 algorihm it is possible to generate rogue CA certificats with the same MD5 hash sum, but different content.

6.5. Certificate 'Usage'

Focusing the discussion on the 'usage' of the certificate, we have to consider

Since we are talking about 'asymmetrical' en/decryption, we always have to keep in mind the 'who'. In general, we have to distinguish between

  1. Standard certificates CA:False, and
  2. 'root' certificates CA:True.

The main difference is, that standard certificates require for the 'how' always a fitting private key during the transaction; while for a root certificate the private key is generally 'off line' (figure 13).

Figure 13a: Usage of a 'root' certificate for signing
Figure 13b: Purported usage of standards certificates

The X509v3 extensions provide the framework for the proliferation of the cert's usage. Here we have:

6.6. Certificate owner's identity

Until recently, the identity or the certificate's owner was solely report by means of the Distinguished Name and here in particular (figure 11) the so-called Canonical Name CN which should equal the hostname in case of the remote computer system, or the email address in case of a personal certificate.

Given a X509v3 certificate, the Subject Alternative Name SAN may carry additional information. Here, multiple attributes are possible, e.g. commonly the IPv4 and IPv6 address alongside with the DNS FQDN. We can use the provided values to verify those against the evaluated DNS information per peer. Once the information in the certificate is correctly embedded, matching the purported values with the received one greatly reduces the risk of MitM attacks and interception.

Note: OpenSSL allows to use 'compactified' IPv6 addresses upon generation of the X509v3 certificate and expands those accordingly.

Chapter 7: UCSPI-SSL

The concept of D.J. Bernsteins's UCSPI-TCP has been picked by Superscript's William Baxter to provide the same type of client/server communication but now based on OpenSSL's TLS services instead the standard TCP sockets called UCSPI-SSL.

While OpenSSL also include sets of network I/O calls named *BIO routines UCSPI-SSL sill uses the low-level routines for communication. UCSPI-SSL comes with two main programs:

In fact, sslserver and sslclient provide the same UI (options and flags) as their brothers and in particular sslserver reads the same cdb for a controlled access of remote clients. Thus, UCSPI-TCP and UCSP-SSL are installed side-2-side on most systems because UCSPI-SSL does not provide capabilites to generate the cdb and does not come with extra services like for instance rblsmtpd.

7.1. Recent Developments of UCSPI-SSL

The original UCPSI-SSL software was written and released in D.J. Bernstein's slashpacket format back in 2001 by William Baxter at Superscript.

Scott Gifford subsequently enhanced the original implementation to allow a 'delayed' negotation of the TLS session, which is required by STARTTLS and STLS.

Since the development of UCSPI-SSL was stalled in 2008, and I was on the way to include TLS support into qmail-remote, while SPAMCONTROL's qmail-smtpd was already successfully STARTTLS-enabled in 2006 (version 2.4), I took the chance to integrate all the required patches to UCSPI-SSL und create the follow-up version 0.8. Currently, UCSPI-SSL 0.9 is in beta, which includes now full support for IPv6.

7.2. UCSPI-SSL working model

The original implementation of UCSPI-SSL behaved 100% comparable to UCSPI-TCP except for it's cryptographic operations using the OpenSSL framework. Mainly two programs are in use:

  1. sslserver providing a 1:1 implementation of tcpserver capabilities.
  2. sslclient again, beeing the cryptographic twin of tcpclient.

In addition, some 'helper' programs exist, like sslhandle and https@. Further, William Baxter included sslperl into UCSPI-SSL.

7.3. The sslserver

Access to the Key Store and Trust Store, where the certs and the key-files live is realized in a 'sandbox' by means of a suid to a low-privileged system user having access to the sensitive information. UCSPI-SSL does not provide store for Certificate Revocation List CRL and does not come with a OCSP client to lookup the cert online according to the information provided in the cert.

Figure 14a: Standard use of sslserver for TLS encryption on port 465
Figure 14b: 'Delayed' TLS negotiation by means of Scott Gifford's StartTLS option

From a functional point of view, these two deficiencies are not relevant for in particular the typical use case of sslserver. However - more relevant - in my implementation of UCSPI-SSL I added Certificate Chain Support.

Currently, there are two other implementation features missing:

Both missing features concern the performance and perhaps scalability of UCSPI-SSL, which however was developed with a small memory footprint in mind.

Standard sslserver use:

Figure 14a shows the 'old' behavior of sslserver acting as complement for tcpserver while reading from and writing to the network the SSL sockets and feeding the (unencrypted) values to the application by means of the file descriptors FD 0 and FD 1.
From the user's persepective, the only change is the additional Key Store and Trust Store to be included using the environment variables CADIR, CAFILE, CERTCHAINFILE, KEYFILE and additionally the Diffie Hellman domain parameters DHPARAM. Unlike other implementions, there is no explicit central store to be read. Access to those files is granted setting up Unix read permissions for a particular user SSL_USER beeing part of SSL_GROUP. While reading the stores, the effictive user-id is switched to the provided values and never exposed to the user of the current sslserver session.

We also realize the casual tcpserver capabilities, reading the cdb and exporting the read environment variables to the target application. Additionally, an IDENT lookup is performed (uncommon now) and not only the IP address of the client is provided to the application, but by means of Dan Bernstein's DNS libs, the FQDN is evaluated and serves as useful information too.

Note: In the current UCSPI-SSL implementation, I included CIDR support for both IPv4 and IPv6 connections.

Let's have a look at the four relevant moments in time as shown in figure 14a:

  1. T=a: sslserver is listing to a specifc port an a specific IP address of the server (or all). It receives a TCP SYN from the client and starts to work. Upon start it has read the Key Store and the Trust Store while loading the certificates and keys into it's memory.
  2. T=b: Depending on the provided options for IDENT and DNS lookup, sslserver tries to gather the information from the remote site; the DNS respectiveley. This information (together with the IP address of the connection) is used to lookup the cdb. In general, accept or deny rules are given.
  3. T=c: in case of an accept the next program is spawned, according to the call of sslserver. In the sample qmail-smtpd is called next, but in reality it might be rblsmtpd or perhaps greylistd in the first place while feeding the decrypted information read from the network on FD 0 to the called routine.
    Invoking sslserver with the option -e, the TCP environment variables are stored in the environment; with additionally the TLS informations are expressed as mod_ssl.
  4. T=d: The child program now can send it's response via FD 1 to sslserver, which will encrypt the information and send it to the remote client.

It is important to understand that the entire procedure realized by sslserver is completely opaque, that means invisible for the called program, unless it has a particular built-in knowledge.

sslserver using the StartTLS option:

Employing STARTTLS however, the different behaviour of sslserver is shown figure 14b:

  1. sslserver is instructed by means of the '-n' (not yet) option to accept unencrypted connections in the first place; typically listing for SMTP traffic on port 25.
  2. Both sslserver and the called client program - in our case qmail-smtpd - communicate by means of the file descriptors FD 0 and FD 1 in the usual way, though these file descriptors or soleley reserverd for un-encrypted commuication.
  3. The STARTTLS behavior needs to be triggered for the client application qmail-smtpd (or qmail-pop3d providing the STLS command) by means of the environment variable UCSPITLS="". Subsequently, qmail-smtpd offers the STARTTLS ESMTP verb to the client.
  4. Once the SMTP client honors STARTTLS, which is recognized by qmail-smtpd, now three additional file descriptors are chosen by sslserver and used by qmail-smtpd:
    • SSLCTLD a control socket used to trigger the STARTTLS operation.
    • SSLREAD the socket to read the encrypted information.
    • SSLWRITE the socket to write the encrypted information.
  5. While sslserver is responsible to care about the SSL context and setting up the connection while reading from the Key Store and Trust Store in a setuid environment, and - upon connection - promotes the mod_ssl TLS environment variables to the called program. Thus, qmail-smtpd is not directly involved in any cryptographic operation and 'knows' about the TLS environment only by means of the received environment variables.
Setting up sslserver

In order to call sslserver and it's client program, we need to become familiar with sslserver's

sslserver command line syntax:

sslserver requires upon start at least the local IP(v4/v6) address and target port to bind with, and the (child) application to run. In case '0' is provided for the local IP address, sslserver binds to all available local (unicast) addresses of the machine. Usually, the port is given explicitely, though you can chose a port for /etc/services like SMTPS.

Figure 15 shows the rich variety of options permitted by sslserver. Typically used are the following:

Logging of sslserver's actions (not very prosaic) is done by means of the option -v.

Figure 15: sslserver command line options and arguments

sslserver's environment variables:

Until now, sslserver seems to have no ideas about the required Key Store and Trust Store and how to read it's own certificate .... However, if we look at the required environment variables, we find the following sets:

The reading and processing of those files can be modified by means of some additional environment variables:

Aport from those file locations, the behavior of sslserver can be influenced by the following environment variables:

Since these parameters are given per environment variable, their setting might not to only to be 'global' for the entire sslserver process but rather may depend on the connecting client using the cdb mechanism. For instance, we may issue different server certificates based on the incoming IP address.

Requesting client certificates:

The TLS protocol is asymmetrical: Only the server needs to authenticate while the client stays anonymous. This is the reason, while typically upon - lets say a HTTP connection - you have to authenticate additonally by means of userid and password.

However, TLS allows mutual authentication of both parties. But there is a price to pay:

In practice, the server's administrator has setup an own Certificate Authority which is used to provide the personalized client certs which are roled-out together with the matching Keyfile to the clients. The server now uses the CA cert (often a self-signed root certificate) to tell the clients:

      "If you like to authenticate with your certificate, it has to match the CA cert which 
      I provide to you in the first place!"

Thus, there are these steps to consider for client certificate verification:

  1. A CA cert needs to be created to sign the client's certs and it has to be deployed to the server as 'client CA cert' CCAFILE in our case.
  2. The client cert alongside with the matching private Keyfile (and perhaps in addition the CA cert for the server has to be deployed to the clients and registered into their Key Store and Trust Store respectively.
  3. The server has to be instructed to request a valid cert from the client and by the same token to promote all it's know CCACERTs which is triggered in case of sslserver by means of the global option -z - or much more useful - setting the environment variable CCAVERIFY perhaps for specific IP addresses.

7.4. The sslclient

UCSPI-SSL's sslclient ist used as base for the auxiliar programs sslcat and https@. Unlike sslserver it uses file descriptor 6 ('&6') for reading and file descriptor 7 ('&7') for writing to the child. Further, it reads the certs and it's keyfile (which my be password protected protected) by means of arguments:

Figure 16: sslclient command line options and arguments

The best way to illustrate how sslclient can be used connecting to public servers can be picked up from the https@ script:

#!/bin/sh # WARNING: This file was auto-generated. Do not edit! if [ "${1}" = "-4" -o "${1}" = "-6" ] then vers=${1} shift else vers="-6" fi host=${1-0} path=${2-} port=${3-443} args="" if [ $# -gt 3 ] then shift; shift; shift args="$@" fi echo "GET /$path HTTP/1.0 Host: $host:$port " | /usr/local/bin/sslclient -XRHl0 "$vers" $args -- "$host" "$port" sh -c ' /usr/local/bin/addcr >&7 exec /usr/local/bin/delcr <&6 ' | awk '/^$/ { body=1; next } { if (body) print }'
Listing 1: The https@ script

Apart from setting some arguments to be transfered to sslclient in the first piece of the script, we recognize, that - following the 'echo' - the input to the 'echo' is redirected to file descriptor '&7' and the result is taken from '&6' to be subject of the programs delcr and awk. It is important to realize, that the both programs addcr and delcr are supplied by UCSPI-TCP(6); thus the script won't run without their help. In the current version of the script, I added -X as an argument for sslclient. Otherwise, the sslclient would ask for the (Web) server's certificate and fail, since no CA cert is provided or perhaps available via the Trust Store.

7.5. The Key Store and Trust Store

Within UCSPI-SSL there are some standard locations fo certs: /usr/local/ssl/certs which is defined in conf-cadir. For the Diffie-Hellman parameter file, the default location /usr/local/ssl/pem/dh1024.pem is used.

In general, we need to take care about:

In general, all these files shall be put in the same directory with as little as possible permissions while readable by SSL_USER and evaluated in the chroot environment as defined by SSL_CHROOT. This of course holds for the Key Store while the certs and the DH parameter file are by definition public.

For standard sslserver operations, we don't need a particular Trust Store. However in case we are using sslclient, the situation is different. Comparable with Web browsers, a Trust Store has to be provided. For this purpose, we use the CADIR capabilities:

Let's give it a try with the certs provided in UCSPI-SSL:

openssl x509 -subject_hash -in rootCA.pem -noout
openssl x509 -issuer_hash -in ::1.pem -noout

Thus, the generic file name of my rootCA.pem in the CADIR would be 1346b848.0, where the '0' indicates that this is the first cert with this hash (others may flollow, if newly generated).
The very advantage of this method is, that the name of the CA cert stays independent from it's i.e. Serial Number.

Chapter 8: Qmail and TLS

Currently, to my knowledge, the following different approaches have been developed to integrated TLS security with qmail:

For the further discussion, I would like to concentrate on my own implementation.

8.1. qmail-smtpd

Using Scott Gifford's approach (figure 14b), qmail-smtpd is completely decoupled from the TLS network interaction by means of sslserver as it is by using tcpserver. There are just three implications:

qmail-smtpd's log information:

qmail-smtpd writes a qualified log file displaying the Cipher in use and perhaps the received DN in case a client certificate is required by sslserver.

2013-09-11 15:22:22.639093500 qmail-smtpd: pid 27929 Accept::TLS::ADH-AES256-SHA P:ESMTPS != 'none'

The 'Received:' Header line:

Received: (qmail 27930 invoked from network); 11 Sep 2013 13:22:22 -0000
Received: from ( de/crypted with TLSv1: ADH-AES256-SHA [256/256]
 DN=none by with ESMTPS; 11 Sep 2013 13:22:22 -0000
Received: from ( []) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits))
 (Client CN "OTRS Test", Issuer "Trusted Introducer (TI) Client CA - G001" (not verified))
 by (Postfix) with ESMTPS id A2380257819E; Wed, 11 Sep 2013 14:05:42 +0200 (CEST)

We recognize, that the first 'Reiceived:' line is written by qmail-send while the second one is included by qmail-smtpd. We see, that the transmission protocol now is ESMTPS. DN=none is alwas included unless the client is required to issue a valid certificate (as discussed above), which DN is displayed.
The last, very lengthy 'Received:' line is done by Postfix (for comparison).

Postfix prefers to use ADH Anonymous Diffie-Hellman upon sending.

Summary for qmail-smtpd TLS capabilities:

By means of sslserver and in particluar the connection dependent environment variables carrying the certificate information, a very flexible TLS setting is possible for qmail-smtpd:

Note: Currently qmail-smtpd does not use client certificate verification in order to flag those clients as RELAYCLIENT. However, this might change in the future.

8.2. qmail-remote

It took me some time how to realize a consistant way to include TLS into qmail-remote in a complete and extensible way. It is easy to provide simple TLS encryption into qmail-remote but is is not trivial to provide authentication as well, since it is not clear in the first place: What or Whom to authenticates ? Thus, we need to solve the problem for qmail-remote to present a valid certificate to the server on demand. Without this capability, client TLS is simply incomplete.

In the qmail system, qmail-remote runs a user qmailr for the entire delivery. The only information which is given for the sender is the 'Mail From:' address, the return-path respectively. Fortunately, as gift of Spamcontrol's qmail-smtpd, we can couple a user identity to the 'Mail From:' address: Mail From: Address Verification MAV.
Thus, it is possible - to some extend - to support for TLS:

Since the cert we need to presented on demand may include within the Subject Alternative Name a DNS name or an IP address - which has to be unique for the sender - we may be forced to use a dedicated IP sending address per domain.

The other problem to solve is the degree of 'trust' we are willing to accept for the remote side, the server respectively. For a given target address (or domain) we need to be able to adjust the credentials required for a particular TLS connection. This is, of course, closely related with the verification of the received server certificate.

Unlike qmail-smtpd, we need to add basic TLS capabilities into qmail-remote since it is acting as network client and facilitates socket calls, which are subject of replacement/enhancement with TLS socket operations.

Within qmail-remote, the control of the TLS session is based on two control files:

control/tlsdestinations is by far the most complex control file. Fortunately, for most qmail sites it includes only one line:


That's it ! It simply tells:

  Upon establishing an ESMTP session with the remote MTA use StartTLS whenever it is announced.
  Demand a cert, validate it, but don't check for potential credentials of the received cert.

In essence, qmail-remote supports:

Most of these operations use explicitely or implicitely the routines provided by UCSPI-SSL.

Last but not least, within the qmail-send logs, TLS transmitted emails are indicated:

2013-09-06 10:02:40.907859500 delivery 110: success:

8.3. qmail-pop3d

The concept to include TLS capability for qmail-pop3d is the same as for qmail-smtpd; except it is not qmail-pop3d which is involded in the first place but rather qmail-popup. Again, the environment variable UCSPITLS is used to trigger TLS.

Unlike qmail-smtpd, there is no fine-grain usage of this variable and there is no additional information to track or to report. Thus, TLS is mainly used here for transport encryption. However, since qmail-pop3d (and it's helpers) are based as well on sslserver, refined usage would be possible, but is currently not supported.

Thus, qmail-pop3d supports two typical ways for TLS encryption:

In addition, since in Spamcontrol 2.7, I provided a logging for qmail-popup and the result of the connections can clearly spotted:

2013-09-11 21:30:59.050975500 qmail-popup: pid 7523 Accept::AUTH::User P:POP3S ?= 'erwin'

Here, P:POP3S indicates the TLS secured connection. If this connection would not be encrypted by TLS, everybody would be able to read my password by means of the POP3 'User' mechanism in clear text.

Chapter 9: Setting up qmail for Transport Layer Security

In order to setup qmail-smtpd and/or qmail-pop3d services, the following files have to be available:

  1. The Diffie-Hellman parameter file, DHFILE.
  2. The X.509 certificate for the servers CERTFILE.
  3. The matching KEYFILE for this certificate, usuallly without an passphrase.

These files, typically together with a configuration file are stored in a dedicated directory, which can be used for all services. Since there the private key of the server is deployed, this directory needs to be specifically secured against espionage.

A typical choice would be /var/qmail/ssl. This directory should belong to root and only root should have permissions to read the files here:

chmod og-rwx /var/qmail/ssl

The required certificates, keyfiles, and the dhparam file can be generated herein by means of the OpenSSL framework. For test purposes however, these files can be taken from the UCSPI-SSL installation directory.

9.1. The TLS environment for sslserver

A generic - minimal - environment file (let's call it tls.env, which will populate sslserver's UCSPI-SSL environment variables looks this way:

# SSL_USER=qmaild SSL_GROUP=nofiles SSL_CHROOT="/var/qmail/ssl" # #CERTCHAINFILE="/var/qmail/ssl/certs.pem" CERTFILE="/var/qmail/ssl/mycert.pem" KEYFILE="/var/qmail/ssl/mykey.pem" DHFILE="/var/qmail/ssl/dh1024.pem" CIPHERS="TLSv1+HIGH:!SSLv2:!MD5" #CAFILE="/var/qmail/ssl/ccacert.pem" #CCAFILE="/var/qmail/ssl/fehnetCAcert.pem" #CADIR="/var/qmail/ssl" #VERIFYDEPTH="3" # SSL_UID=`id -u $SSL_USER` if [ $? -ne 0 ] ; then echo "No such user '$SSL_USER'" >&2 ; exit; fi SSL_GID=`id -g $SSL_USER` if [ $? -ne 0 ] ; then echo "No such group '$SSL_GROUP'" >&2 ; exit; fi export SSL_UID SSL_GID SSL_CHROOT CERTFILE KEYFILE DHFILE CIPHERS #export CAFILE CCAFILE CERTCHAINFILE VERIFYDEPTH CADIR
Listing 2: The SSL 'environment' file

It might be necessary to have different files to set up a specific environment, e.g. smtps.env, starttls.env. Remember, that these parameters may be subject of change in the later execution chain; for instance can be modified while reading the cdb.

The environment file needs to be called from every run script providing TLS services (figure 17):

Figure 17: Including the environment setting script to be used by sslserver

As can be seen, there are several ways, how and what to include in the environment file. For instance, you may like to use Bernstein's envdir scheme instead.

9.2. qmail-smtpd and SMTPS

Once the TLS environment file is populated, the next step is to setup the ./run script for the serivce.


That's my very generic run script for providing SMTPS services:

#!/bin/sh QMAILDUID=`id -u qmaild` QMAILDGID=`id -g qmaild` HOSTNAME="" port SMTPAUTH="!" export UCSPITLS="" . /var/qmail/ssl/smtps.env exec sslserver -4 -sev -Rp -l $HOSTNAME -c 20 \ -x /var/qmail/etc/tcpd.smtpd.cdb \ -u $QMAILDUID -g $QMAILDGID 0 smtps \ /var/qmail/bin/qmail-smtpd /usr/local/bin/checkvpw true maildir 2>&1
Listing 3: The qmail-smtpsd run script

It is important not to call sslserver with the -n flag!
Since my provider does not yet support IPv6, I've called sslserver with the -4 option. Also, for using SMTPS I require ESMTP authentication: SMTPAUTH="!".

Here, I use a somehow more restrictive smtps.env file with some special ciphers.

9.3. qmail-smtpd and StartTLS

Typically, the run script for the public offered port 25 (SMTPS) is little more complicated since we do more checking for the remote client:


That's my run script for providing SMTP services:

#!/bin/sh QMAILDUID=`id -u qmaild` QMAILDGID=`id -g qmaild` HOSTNAME="" export SMTPAUTH="" export UCSPITLS="" . /var/qmail/ssl/env exec sslserver -4 -sevn -Rp -l $HOSTNAME -c 120 \ -x /var/qmail/etc/tcpd.smtpd.cdb \ -u $QMAILDUID -g $QMAILDGID 0 smtp \ rblsmtpd -W -r \ /var/qmail/bin/qmail-smtpd /usr/local/bin/checkvpw true maildir 2>&1
Listing 4: The qmail-smtpd run script

Please recognize the call of sslserver with the options -sevn.
Further details of the setting (e.g. for rblsmtpd) are explained elsewhere.

9.4. qmail-pop3d

Since we worked thru the SMTP/SMTPS configuration already, the POP3/POP3S settings are no riddle anymore:


#!/bin/sh HOSTNAME="" export UCSPITLS="" . /var/qmail/ssl/env exec sslserver -4 -sevn -Rp -l $HOSTNAME -c10 \ -x /var/qmail/etc/tcpd.pop3d.cdb \ 0 pop3 \ /var/qmail/bin/qmail-popup $HOSTNAME /usr/local/bin/checkvpw \ /var/qmail/bin/qmail-pop3d Maildir 2>&1 5>&1
Listing 5: The qmail-pop3d run script

Here, I use in addition a tcpd.pop3d.cdb because some US and Chinese folks try to read my mailboxes; which i don't like too much.


#!/bin/sh HOSTNAME="" export UCSPITLS="" . /var/qmail/ssl/env exec sslserver -4 -sev -Rp -l $HOSTNAME -c10 \ 0 pop3s \ /var/qmail/bin/qmail-popup $HOSTNAME /usr/local/bin/checkvpw \ /var/qmail/bin/qmail-pop3d Maildir 2>&1 5>&1
Listing 6: The qmail-pop3sd run script

9.5. qmail-remote

Though the TLS client, in a qmail system qmail-remote hat at first glance a relative easy task to provide TLS encryption and security, however once we not only require encryption, but in addition sufficient credentials and identification from the server things become quite complex again; not to consider client-side authentication by means of a valid X.509 certificate.

In my qmail-remote TLS implementation I tried to realize easy and complicated settings with just two control files:

Indeed, there are some more control files to consider:

Controlling outgoing TLS connections:

The control file tlsdomains allows the following information to be included:


All but the entry (before the colon ':') are optional. Let's discuss three simple cases:

1. *: Switch to StartTLS once the server annouces it.
2. -*:||!kRSA:aNULL Don't check the server's cert ('-*') and don't announce RSA key exchange ('!kRSA') but rather request ADH exchange.
3. ! Even if host '' announces StartTTLs, use an un-encrypted connection only ('!' means not).

Now consider a few cases, where you would like to verify the server's cert. Of course, here you need to have the corresponding CA cert:

4.||!SSLv2:HIGH Verify the cert for '' against the CA cert provided as /etc/ssl/cafile; don't accept SSLv2 connections and request 'HIGH' encryption standards.
5.|3:465 For every host of domain '' check it's cert against the CAs available in /etc/ssl/certdir and use a Verifydepth of at most '3'; use SMTPS on port 465.
6.|4 For the host '' verify it's cert and demand matching of cert with evaluated 'hostname'; check the cert against the CA cert named /etc/ssl/cacert and allow a Verifydepth of 4.
7.|:26|| For the host '' verify it's cert against the CA cert /etc/ssl/partnerca using a connection to port 26 but only, if the domain in the 'Mail From:' address is

The tlsdestination control files uses thus the special characters '*' (all), '!' (not), '-' (no cert required), '=' (verify cert against hostname). While typically a complete hostname is given left from the colon ':' (which is the domain part of the 'Rcpt To:' address), a whole domain can be specified if the provided name starts with a dot '.'.

In addition, we can bind the TLS settings with the domain part of the 'Mail From:' address.

Providing Certs to the requesting Server:

Occassionally, it is necessary to show a valid X.509 cert upon request: client authentication.

In order to support this behaviour, the control file domaincerts can be used:


Since qmail-remote needs to read not only this file, but also the certs, and keyfile, this information is sensitive and should be protected by means of file permissions. In particular, if the keyfile is not protected by the a password it should be solely readable be qmailr and placed in a well-suited directory.

certfile and keyfile contain the complete path of the file. In case domain equals '*' the provided certificate is potentially used for all outgoing connections. This means, the cert is attached to the entire qmail-remote instance, the sending MTA respectively and thus does not depend on the sending domain, as provided usually according to the 'Rcpt To:' information.

Chapter 10: Lose ends upon TLS cryptography

NSA interception

In spite of the current NSA surveillance affair, the believe in public key cryptography is somewhat undermined. Some cryptographic functions TLS offers are directly influenced by the NSA.

Quoting Eric Rescorla from his book SSL and TLS (page 105):

      "FORTEZZA was originally designed by the NSA to provide strong cryptography while allowing
      the NSA to intercept communications. The conflict between these goals was resolved by incorporating a
      key escrow feature into the device. Each card had its own key, which was escrowed with the NSA.
      Thus, when the NSA wished to decrypt a communication, it could recover the key for the card used to
      encrypt it, but the encryption would be secure against all other attacks."

FROTEZZA was specified in SSLv3 but removed in TLS.

Cipher Strength

With UCSP-SSL and my implementation into qmail you can control the level of security by means of the Cipher Suite as shown in the samples. To some extend, you can also be protected against roll-back attacks. However, in case the OpenSSL library has some specific bugs or perhaps 'forged' cryptographical routines, there is little I can do about that. Please watch the isssued CVEs perhaps reinstall OpenSSL, bind UCSPI-SSL with the new libs and reinstall.
From my point of view, the RSA key exchange is too weak these days, Diffie-Hellman is by far better. Using ECC, the standard Elliptical Curves and there parametrization with TLS are at least guided by the NSA while beeing standardised by NIST.

Broken CA trust system

On the other hand - again in my oppinion - the entire trust system, PKI is depending upon, is broken. It irrelevant, if the certificate is signed by RSA, DSA or ECDSA; once you control the Trust Center, you rule the game. The Dutch CA DigiNotar (among others as well) was comprised to do so.

Peer X.509 certificates still do work, though.

State of a TSL connection

Not providing the security state to the upper layer is a considerable design flaw. The problem is double-present in case a TLS ALRM message needs to get issued. In case this happens, (hopefully) the application is getting terminated but without any reason, the user is informed for. This is bad by design.

IDNs, the hostname, and the SAN

Unlike the old days; domain names in the DNS may carry 'umlauts', hebrew, arabic and other characters: International Domain Names IDN. To code those characters PunyCode has been developed.
I'm not sure how to support this in X509 certificates and in particular how to match those.
Parsing and comparing is a poisoned operation.

Authentication and Authorization by means of DANE

The DNS-Based Authentication of Named Entries (DANE) (RFC 6698) is a hot topic as candidate to replace the Public Key Infrastructure (PKI). With DANE, the server's public key is stored in the DNS Zone file of the provider of the server itself. This ensures a-priori authorization of the public key and authentication can be guaranteed if the transmission of the public key by means of DNS is uncompromised. This may be subject of DNSSEC or CurveDNS.

Further information about TLS can be found here: RFC Sourcebook.

Disclaimer: This page may contain outdated, or even misleading statements. Help me to improve it! This is work in progress.