FAQ.qmvc


1. Controlling the filter
2. Infected E-Mails and Anti Virus Scanners
3. Incident Messages
4. Setting up Response Teams
5. Special environments
6. NullSender-Mails
7. Errors in parsing Mails
8. Load and performance
9. Multilingual support
10. QMVC's Analysis tools

1. Controlling the filter

1.1. How to filter Bad Subjects?

In the control file "filters" the option "subject=y" has to be set.

This is the default.

In the control file "badsubjects" reasonable text-strings have to be defined. By default, the strings "ILOVEYOU" and "I-LOVE-YOU" are included.

Every E-mail containing any of those strings will be blocked.

1.2. How to filter Bad Body Texts?

In the control file "filters" the option "body=y" has to be set.

Attention! The default is "body=n".

In the control file "badbodytexts" reasonable text-strings have to be defined. Be careful! Any part of the E-Mail body, including PDF documents, and e.g. Word files will be parsed for this string.

1.3. How to filter Bad Mime Types?

In the control file "filters" the option "mimetype=y" has to be set.

This is the default.

In the control file "badmimetypes" those types have to be included which can be identified by means of your "mailcap" file (default location /usr/local/etc/mailcap) and/or your "magic" file (default locations /etc/magic or /usr/local/share/magic).

Since qmvc employs a string compare, enter the description of the MIME type to be filtered very precisely according to what is given here, e.g. "MPEG video stream data" (put that string without quotes into badmimetypes) will filter MPEG video attachments.

Therefore, the results are a little "fuzzy" and depend on your environment. See man-page of "file" for locations of the "mailcap" file.

1.4. How to filter Bad File Names?

In the control file "filters" the option "filename=y" has to be set.

This is the default.

In the control file "badfilename" you may define sets of bad file names; either explicitly (e.g. "link.vbs") or wild carded (e.g. \.vbs$or \.exe$).

1.5. How to filter Bad Files in Archives?

Neither the Bad Mime Type nor the Bad File Name filter work on archived/zipped/tared files.

The main reason is, not to disturb communication, but rather to reduce the risk of potential dangerous attachments like ".exe", ".vbs" on vulnerable clients (e.g. MS Outlook) which will be loaded and executed without user notification and presumably will be infected.

1.6. Are the filters case sensitive?

The case of the filter words is only relevant for the BADSUBJECT and the BADBODYTEXT.

Whether or not the filter work case sensitive depends whether the Option "ignorecase" is set in the "filters" control file.

1.7. Do the BAD* filter support Regular Expressions?

The BADSUBJECT, BADBODYTEXT, and BADMIMETYPE filter employ the grep command to compare strings. Thus, the ability for a RegEx comparison depends on your grep command. Typically, the standard grep supports POSIX RegEx expressions which are summerized in man qmvc-filters. If you need exted grep capabilies you can modify the "alias" for grep in the qmvc script accordingly.

The BADFILENAME algorithm uses the find command with RegEx capabilities.

1.8. When will new filters be active?

It is the concept of QMVC, that all filters will be active on-the-fly. After you have changed a filter and saved it to disk, the next call of qmvc will read it and follow the new instructions.

 

2. Infected E-Mails and Anti Virus Scanners

2.1. How do they work?

The Anti Virus Scan Engines usually depend on the fact, that the E-Mail is already disentangled into individual files - the attachments - which is facilitated in qmvc by means of reformime.

Thus, the AV Scan Engines act as file scanners, rather then E-Mail scanners. In addition, most AV Scan Engines do a "uncompression" of files. This means, if they recognize an archive (zip, cab, tar, arj ....) they scan the contents of the de-compressed archive.

The AV Scan Engines generate logs of their actions and - usually - return with different Return Codes if a Virus is detected or not.

2.2. How are the AV Scanners integrated?

AV Scanners are integrated by means of an external Plug-In. The Return Codes of AV Scanners and their results are read and returned to the main program (qmvc).

2.3. How to update AV pattern files?

The use of the script vupdate is depreciated. Use your AV Scanners update facility instead.

2.4. What is Virus Bombing?

Virus Bombing is a criminal act - though not prosecuted yet - scanning public Web-Pages for E-Mail addresses and sending to those accounts E-Mails with Viruses in high frequency.

While QMVC helps to block those E-Mails, it is more efficient to block those Originators by means of Qmail's "badmailfrom" or the additional features of my SPAMCONTROL patch.

2.5. How to deal with new Viruses?

New Viruses come out all the time. The latest and most spectacular was "NIMDA" (or "Code Blue"). While it takes time for the AV vendors to update their products, to produce, and to deploy the re-newed pattern files, QMVC enables you to stop new Viruses instantly by means of their typical pattern in E-Mails, e.g. "README.EXE" for NIMDA using the "Bad File Names" mechanisms.

2.6. What about BinHex and UUDECODED files?

qmvc decodes MIME and UUDECODED E-Mails correctly, though there is no support for BinHex Attachments. This has two consequences:

  1. The Bad* filter mechanism for file (mime) types and filenames does not work for BinHex attached files in the E-Mail.
  2. BinHex encoded Attachments may not be correctly scanned for viruses (however some AV engines recognize BinHex encoded files and are able to scan those).

2.7. How can I test qmvc without having Virus Mails?

In the ./doc directory some testmails with the EICAR.COM test pattern are included which may used to test the virus scanners and qmvc in general. Feel free to change the testmails, (Subject:, Body, etc.) to your needs. Feed the testmails through qmvc:

cat testmail | /usr/local/bin/qmvc -u

and watch the results in the logs. In order to get correct results w.r.t. Sender and Recipient you have to set (temporarily) "evalheaderaddress=yes" in ./control/options. The EICAR.COM test pattern might not to be recognized by all AV scanners.

2.8. Now to exclude certain file types for AV scanning?

In the control file "options" set "skipnoscantypes=y".

Now, all file types as identified by the "file" command and included in the control file "noscantypes" are excluded for Virus checking. In particular, you may want to skip plain ASCII or ISO text and multimedia files.

File types of other formats (eg. executables) are labeled "positive" and - if encountered - the AV Scan Engines are loaded.

 

3. Incident and Notification Messages

3.1. What are incident messages and who receives them?

In general, incident messages for blocked E-Mails (Badmails/Virusmails) are generated and send to E-Mail addresses for teams which are called "AV Guards" and "BM Guards" defined in the control files "avguards" and "bmguards". Those people act as "postmasters" for Badmails/Virusmails and may be contacted in case of inquiries and questions. Further, they have to stay up-to-date with the "virus development" and should have the ability to define and set new filters in QMVC.

Incident messages serve as a "on-line" notification of qmvc's activity they show some details about the filtered E-Mail for further investigation. Starting with QMVC 1.2.0, the generation of incident messages can be disabled, in order to stop flooding the mailboxes of the responsible persons and since QMVC provides now some good off-line tools to investigate it's activity (Virulator).

3.2. Should I send notification messages to Originator?

That's a matter of your company policy. In fact, you may wish to send a notification to somebody who "accidentally" sends a Virus-Mail. However, most common, people do send Virus-Mails on purpose. Furthermore, those folks usually fake their Originator address, thus they never receive incident reports. In opposite, it may be useful to send Badmail notification Mails to the Originator. Rule of thumb:

If you are an ISP and use QMVC on your Internet E-Mail Gateway with Qmail

  • don't send Virus notification Mails to the Originator
  • you may send Badmail notification Mails to the Originator if this agree's with your customer's policy.
  • If you use QMVC on a UNIX system where you have individual accounts

  • don't send Virus notification Mails to the Originator
  • don't send Badmail notification Mails to the Originator.
  • Warning: Notification Messages to the Originator may lead to bounces and unexpectable results and even may target the wrong persons/organizations.

    3.3. Should I send notification messages to Recipient?

    Again, that's up to your company policy. However, this is less critical rather then sending incident messages to the Originator. If the local E-Mail account is not a real person, rather a public address, this may lead to an frequently filled mailboxes, in particular in cases of "virus bombs".

    3.4. Response E-Mail addresses

    The control file "respmail" is used to define response E-Mail addresses which are valid Internet SMTP addresses. Thus, within the control files "bmguards" and "bmguards" you may include your company's internal addresses; within "respmail" official addresses have to be declared. Those are propagated - if applicable - to Senders/Recipients of incident messages.

     

    4. Setting up Response Teams

    4.1. Role of QMVC

    qmvc acts as a local originator of incident E-Mails. The name of qmvc is defined in the control file "me", the hostname in "defaulthost". The qmvc user is equivalent to Qmail's alias user with same permissions. qmvc never receives mail, therefore within a .qmail-qmvc file every incoming mail for qmvc is trashed.

    4.2. Role of QMVC Administrator

    The QMVC administrator is responsible frequently checking qmvc's log files, integrating and checking the Virus Pattern updates and watching the behavior of the system. Perhaps he acts in addition as "postmaster" and "root" on that systems and receives E-Mails send to those accounts.

    4.3. The Anti-Virus Guards

    The AV Guards have to watch the current developments of Virus deployment in the Internet by means of the know organizations (e.g. CERT). The have to contact the vendors of the AV Scan Engines in use with QMVC and have to stay on-line with current developments.

    4.4. The Badmail Guards

    More or less same as 4.3 but in addition they have to consult customers and users (in case QMVC is used for an ISP) and have to guarantee the companies policy w.r.t. Badmails.

    4.5. Forwarding Bad- and Virus-Mails

    Under certain conditions, it may be useful not (only) to have a local copy of the Badmail/Virusmails but rather to forward them for later investigation and analysis - or simply - to dump them.

    Therefore, you may use (with QMVC > 1.3.x) the control file "options" key "forwardbadmails=y" and "forwardvirusmails=y" and additionally include E-Mail addresses in the control files "bmforward" and "vmforward".

    However, you may consider the following:

  • E-Mails are private. To read those E-Mails without permission does not only violate their privacy but also may break laws.
  • Forward those E-Mails to accounts/systems not having an active E-Mail/Virus-Scanner (and in particular not to an account controlled by QMVC).
  • Don't forward infected Mails to an account based on a Microsoft OS (except for MS XENIX).
  • It may be wise to define on the Qmail/QMVC MTA local accounts just for this purpose and make those accessible for the Badmail and Anti-Virus Guards.
  •  

    5. Special environments

    5.1. What is a QMVC "profile"?

    We call the set of QMVC configuration files in the control directory a "profile". The default profile is located in the directory "/var/qmvc/control".

    An "extended profile" consists of a individual control directory and individual template directory; default "/var/qmvc/tpl".

    5.2. Machine dependent setup of QMVC

    In this case, the call of qmvc is included in ".qmail-default" in /var/qmail/alias.

    /var/qmail/alias/.qmail-default
    | /usr/local/bin/qmvc

    and perhaps on every local account as per ~/.qmail.

    The temporary files and the logs are hosted in the QMVC home directory, usually /var/qmvc/tmp/ and /var/qmvc/log/.

    5.3. User specific setup

    QMVC guarantees privacy of your E-Mail scanning if you include the call of qmvc with option '-u' in your local .qmail and .qmail-default file. By means of the flag '-u' (user) qmvc will look for the directories "~/qmvc/control" and "~/qmvc/tpl" to use a local (extended) profile.

    ~/.qmail :
    |/usr/local/bin/qmvc -u

    Furthermore processing and logging of qmvc's actions will be placed in the user's home directory.

    5.4. QMVC and Qmail's Virtual Domains

    Under Qmail, virtual domains are included in the control file "virtualdomains" and treated like locals. For E-Mails targeted to that virtual domain, Qmail prepends the original Recipient with a string (usually an account name) and further delivery is handled by Qmail's VERP mechanism.

    qmvc recognizes the prepended account name using the option "-v-". Incident E-Mails to the (foreseen) Recipient are scheduled accordingly. You may want to setup QMVC individually for each user (= virtual domain) with their own profiles, templates, temporary space (with quotas) and own log files.

    Include the call to qmvc with the '-uv-' flag in the user-specific .qmail files:

    ~/.qmail-default
    | /usr/local/bin/qmvc -uv-

    In addition, generate a specific "~/qmvc/control" and perhaps "~/qmvc/tpl" directory. Those definitions take precedence over the default ones.

    5.5. Using QMVC with fetchmail

    It is useful to enable the option "evalheaderaddress=y" which will use the Sender ("From:") and Recipient ("To:") address from the E-Mail's Header instead of the probably meaningless SMTP-Envelope.

    5.6. Adding a customized Header-Field in the scanned E-Mail

    Unlike other AV tools, qmvc does not change the E-Mail Header and does not allow to include a certain Header-Field like "X-Scanned: QMVC xxxx". Why not? Because it is nonsense:

  • Nobody can guarantee that the received E-Mail is real virus-free.
  • The (local) Recipient believes the E-Mail is virus-free and does not not obey the simplest security rules.
  • The (remote) Recipient will be informed about the tools/programs you use to check E-Mails; thus - doing reverse engineering - he/she is able to build an E-Mail such, it will use the shortages of the product, resulting in unidentified viruses or a DOS attack on the system.
  • 5.7. Filtering outgoing messages

    In order to filter outgoing messages by QMVC you have to set up two Qmail instances. Let's assume your domain is "yourdomain.com".

  • Modify in the qmail-1.03 source directory conf-qmail, chose a name like qmail2 for the base directory and install.
  • You are free to bind the new qmail instance to a separate IP address or perhaps an alternative TCP port (eg. 26).
  • Set up the new qmail instance as the old one, except for this difference.
    1. Copy the Qmail control files from the old installation to the new one; except:
    2. Remove locals and smtproutes.
    3. Create a new virtualdomains file with just: ".yourdomain.com:yourdomain"
  • You can remove rcpthosts, if you tell tcpserver just to accept connects from your local subnet(s).
  • Additionally, with multilog you have to raise the corresponding log directories.
  • Create a user account "yourdomain" and include a .qmail-default file:
  • |/usr/local/bin/qmvc -uv-
    |/var/qmvc/bin/redirect

    Here, it is important that your standard Qmail is at /var/qmail; otherwise modify the script "redirect"; in particular the path of qmail-inject.

  • In your local DNS, create an A, PTR, and MX Record for the new Qmail instance.
  • You have to tell the E-Mail clients and/or MTAs downstream to use the new FQDN/IP address/Port number.
  • From now on, your MTA acts in a "split horizon". E-Mails from the Internet come through "qmail" while "qmail2" is responsible for outgoing E-Mails, which will be Virus scanned and eventually forwarded through qmail to the Recipient.

     

    6. NullSender-Mails

    6.1. What are "NullSender-Mails"?

    By definition, we call E-Mails with an empty SMTP-Sender "<>" Envelope address a NullSender-Mail.

    6.2. Are "NullSender-Mails" legal?

    Yes they are, and your MTA MUST accept those E-Mails. As required by RFC2821 SMTP notifications are send as NullSender Mails; usually bounces. In general, SMTP notifications are send by the Postmaster of Mailer-Daemon.

    NullSender Bounces recognized by qmvc are labeled as "(=MAILSENDER=)"; others are NullSender E-Mails are shown as "(-MAILSENDER-)", where MAILSENDER comes from the E-Mail Header.

    6.3. Why are NullSender-Mails often virus infected?

    In order to hide it's identity, E-Mails containing a virus are send without SMTP Sender and/or E-Mail Header Sender. qmvc shows those E-Mails in the qmvc.log and the Virulator yields a statistic.

    6.4. How do I stop delivery of unwanted "NullSender-Mail"?

    Though it's impossible to judge in prior whether "NullSender-Mails" are acceptable, experience shows, that most of those E-Mails are fake and potentially dangerous, because of viruses.

    Setting the token "trashnullsendermail=y" in the "nullsender" control file will qmvc advice Qmail not to deliver those E-Mails to the Recipient. However, they will be logged and checked, if the token "earlyquitonnullsender" is set to "n". In addition, you may chose to skip scanning those E-Mails for viruses, in order to save processing time ("skipviruschecknullsender=y").

    NullSender-Trashing should be only used in special cases during limited time periods, i.e. if Virusbombing occurs.

     

    7. Errors in parsing Mails

    7.1. E-Mail Header Errors

    Frequently, corrupted E-Mails with bad Header and Body are seen. Those E-Mails are flagged in the qmvc log, but tried to analyze further.

    These are due to broken Mail User Agents (MUA) and some local postprocessing of E-Mails (Virusscanners adding some information to the header).

    7.2. Reformime Errors

    Starting from QMVC 1.1.0 MAILDROP's reformime replaces metamail. metamail was unreliable in particular reading multipart/alternate body parts.

    In the qmvc log, E-Mails with reformime errors are listed. Occasionally (and dependent on the version), reformime is unable to detect a multipart/alternate attachment. If this happens, in the qmvc.log you see the reformime error "R:104".

    In case a reformime failure is recognized and it's not possible to disentangle the E-Mail body, the entire E-Mail body is fed through the AV scanners. To have a better handle on reformime errors, they are logged in /var/qmvc/log/qmvc.error.

    7.3. Errors in the qmail-send log

    Errors in the qmvc script will show up in the qmail-send log. Typically this happens, if scripting or permission errors occur in qmvc. Please check, whether you have changed the sources.

     

    8. Load and Performance

    8.1. Average Load

    Depending on

  • how many AV Scan Engines are mutually used and
  • how many E-Mails and Attachments have to be scanned per time interval
  • qmvc adds a substantial load to the the system. On a system with a PIII 850 and 512MB memory, scanning 14.000 E-Mails per day with two Anti Virus Scan Engines, the average load running FreeBSD 4.3 was less than 0.3.

    8.2. Performance issues

    Unlike Scan4Virus, qmvc is decoupled from Qmail's send and receive processes and only interfaces with the delivery agent (qmail-local). Apart from a separate Swap partition, performance may enhanced if

  • /var/qmail
  • /var/tmp and
  • /var/log
  • are available on separate disks and filesystems. This will help to process the Qmail queue and qmvc's activity isolated from the rest of the system's tasks yielding better performance. Depending on the amount of traffic, about 4 GB space should be available on every disk.

    8.3. Performance improvements

    The AV Scan Engines are only loaded into memory, if an attachment is found. However, you can stop scanning E-Mails already flagged as "Badmail" setting the parameter "earlyquitonbadmail=y" in the "options" file. Employing several Virus Scanners, scanning my be stopped, if a Virus has already been found. This is accomplished defining "earlyquitonvirus=y" in the "options" file.

    As a general rule, avoid using "empty" filters, i.e. having a control file "badmimetypes" with only commented entries. Setting the option "mimetypes=n" in the "filters" file, avoids reading this "empty" control file.

    8.4. Performance monitoring

    Running qmvclog2html will give you the processing time for E-Mails needed by qmvc. In the produced HTML file a table with the delay times are shown.

    9. Multilingual support

    9.1. Concept

    QMVC allows you to generate "Badmail" and "Virusmail" notification messages to either the Recipient or the Originator (Sender) of the E-Mail. It is possible to customize the contents of those messages to your demand, editing the template files (see below). Further, the Domain part of the Recipient of the incident/notification message is evaluated and a Domain-specific message may be created. Currently, QMVC supports English (default) and German incident messages for the originator.

    9.2. Templates

    In the directory "/var/qmvc/tpl" the incident templates are locate. They are distinguished by name, whether they are - receiver/sender templates (receiver../sender..) or - virus/badmail templates (*_av_*, _*bl_*). The language equals the suffix of the templates, currently supported are German (ge) and English (us).

    9.3. Adding new Templates

    Take the templates, modify them to your needs and save them with a suffix, corresponding to the language (e.g. French = fr). Now you have to change the source code of qmvc. In the functions

    - sender_av_report
    - sender_bl_report
    - receiver_av_report
    - receiver_bl_report

    simply include specific case statement(s), i.e.

    (fr) LANG="fr";;

    From now on, E-Mails from or to the Domain "fr" will receive incident messages in French language.

    10. QMVC's Analysis tools

    10.1. What does "qmFind"?

    qmFind (a perl script) finds in a multilog or splogger formatted qmail-send log mails for a specific Sender/Recipient and shows the last delivery state of the E-Mail.

    qmFind is available as a symlink at /usr/local/bin/. It is very helpful, if you have to answer a question like:

    "Hey, I haven't received mail from ... yet! Where is it?"

    Furthermore, my means of qmFind, you can identify Qmail's internal identifiers for a specific mail, e.g. who was the originator of it (UID).

    10.2. What does "qmvclog2html"?

    qmvclog2html is a perl script to analyze the qmvc.log and to display it's contents in HTML format.

    In addition, a monthly summary is accumulated (flag "-m") and written as /var/qmvc/log/qmvcmonth_YYYYMM.log. Make sure to rotate your qmvc.log and qmvc.rep on a daily base.

    10.3. What is "qmvcmonth" good for?

    The perl script qmvcmonth uses qmvclog2html and Virulator's accumulated output written and transforms this into some nice HTML pseudo-graphics. Furthermore, the month's summary is displayed in a table. In order use visualize this, you have to use a graphics rendering HTML browser (Netscape, Opera, Internet Explorer). The visual results may vary.

    10.4. Strange name "Virulator"!

    Virulator is a powerful tool (perl script) to analyze and display qmvc's activity in HTML according to qmvc.log and correlating it with qmvc.rep. The intention is to show the most "virulent" sources and sinks. It helps you to understand the coming and (intended) going of bad and infected mails and their contents.

    If your Boss asks you, why you have chosen QMVC and what is it good for, show him the output of Virulator!

    10.5. What is "blockvirussender" good for?

    Running the Virulator script periodically (lets say once per hour) will give you an up-to-date virus/QMVC state.

    Virulator will store some results under /var/qmvc/current including a "hit list" of the most virulent SENDERs. This "hit list" is read by the script blockvirussender and - depending on some thresholds - those SENDERs are banned by means of the Qmail control file "badmailfrom".

    10.7. What does the "scanner" script do?

    The script scanner looks at the current installed AV scanners, tries to identify the current release and engine version as well as the currently installed pattern files. The results are printed in an HTML file. For convenience, the information is included into the "Virulator results applying the flag "-s".

    10.8. How to use QMVC's WebCalendar?

    The WebCalendar translates the BSD's ncal information into a HTML table. It includes dummy HTML links to files; thus it can be used as a reference to daily generated log information.WebCalendar should be called once per year. The referenced files have to have the following format:

  • filename1_MMYYYY.html (monthly reports)
  • filename2_DDMMYYYY.html (daily reports).
  • Strategy:

  • Make this /var/qmvc/html directory visible by FTP from your web browser:
    ftp://username:password@yourhost/var/qmvc/html/qmvc_calendar.html/
    and store the bookmark.
  • Running a HTTPD on you MTA, you can easily access all QMVC HTML files for the virtual domains executing the script qmvcportal. You may perhaps need to symlink
    ln -s qmvc_calendar_YEAR.html qmvc_calendar.html
    and move the HTML files for the different years to subdirectories carrying the name of the year, eg. /var/qmvc/html/2001/.
  • For apache, you either have to symlink ~/qmvc/html -> ~/public_html or include a corresponding "html" into your user directive and additionally change the access permissions correspondingly.
  • 10.9. When should I ran QMVC's analyze scripts?

    It is best to run the analyze scripts on a daily and perhaps hourly base.

    Here is an example for root's crontab:

    # QMVC add-ons
    55 * * * * /usr/local/bin/qmvclog2html -c 2>&1 >> /dev/null
    56 * * * * /usr/local/bin/virulator -c 2>&1 >> /dev/null
    59 * * * * /var/qmvc/bin/blockvirussender 2>&1 >> /dev/null
    55 23 * * * /var/qmvc/bin/vupdate 2>&1 >> /dev/null
    57 23 * * * /usr/local/bin/qmvclog2html -m 2>&1 >> /dev/null
    58 23 * * * /usr/local/bin/virulator -s -m 2>&1 >> /dev/null
    59 23 * * * /usr/local/bin/qmvcmonth 2>&1 >> /dev/null
    # End of QMVC add-ons

    10.10. How to anonymize the log information?

    Anonymization means to hide Sender/Recipient information in the log files. You can accomplish this for the HTML-Output generated by qmvclog2html and virulator by means of the flags -r (hide Recipient) and -o (hide Originator).


    Last change: 2003-04-23