Using Clam AntiVirus with MIMEDefang

Installation | Configuration | Updating Definitions | Testing | Logging

How to setup MIMEDefang in conjunction with the Clam AntiVirus (clamav) scanner to help prevent the spread of malware through Unix e-mail servers. Malware is a general term encompassing viruses, trojan horses, worms: software that may damage or render insecure computers and networks. Most malware—some 99% of the 82,000+ known—requires Microsoft software to function.

Microsoft must fix their products to prevent the trivial execution of untrusted code by unskilled computer users. Until then, the hordes of Microsoft systems on the Internet forces responsible e-mail administrators to bear the cost of implementing incoming and outgoing anti-virus scanning. Each successive malware release wastes valuable admin time by forcing us to respond to, document, and otherwise deal with the effects of Microsoft’s negligent product design.

Overview

These notes reference clamav 0.65 and MIMEDefang 2.35 on FreeBSD 4.8 and RedHat Linux 6.2 and 7.3 systems with Sendmail 8.12.9 running as dedicated e-mail servers. A dedicated e-mail server runs no other services that require antivirus scanning. Additional configuration steps may need to be done if file sharing services are offered on the same system, such as allowing access to the clamd daemon from users besides the defang user.

A simpler solution for clamav and Sendmail might be to use clamav-milter.

The following image outlines the relationships between the various programs and their configuration files. The sendmail daemon (configured via sendmail.cf which in turn is built from a sendmail.mc file) communicates with the mimedefang-multiplexor daemon via a socket; mimedefang-multiplexor in turn communicates with mimedefang.pl via a socket. The behavior of mimedefang.pl is configured via the /etc/mail/mimedefang-filter configuration file; custom Perl code may be used to make calls to an in-memory Mail::SpamAssassin object (configured via /etc/mail/spamassassin/sa-mimedefang.cf) or to tell the clamd daemon to scan specified messages via a socket. The clamd daemon reads its configuration from /usr/local/etc/clamav.conf, and malware definitions from /usr/local/share/clamav/viruses*.db. The malware definitions are updated with the freshclam utility, here via the custom up-avdefs script run from cron(8). Errors and other log messages are sent from sendmail, mimedefang-multiplexor, mimedefang.pl, clamd, up-avdefs, and crond to the syslog daemon, which saves the messages to logfiles as specified in /etc/syslog.conf.

As of clamav version 0.80, the clamav.conf has been renamed to clamd.conf.

Installation

Install clamav | Install MIMEDefang

Abbreviated install notes for clamav and MIMEDefang. Other documentation covers installation of these programs in more detail.

I strongly recommend using a test box to first try out MIMEDefang on before moving to a production server.

Install clamav

Consult the package or ports tree of the system for a native version; otherwise, install from source. The most recent versions of clamav require the GNU Multi-Precision (GMP) library for signature verification support.

clamav is under active development: following the most recent release will be needed to ensure the latest malware are blocked and bugs fixed.

  1. Add clamav user and group.
  2. Build and install clamav.

    3. $ ./configure --disable-clamuko && make && sudo make install

  3. Create and enable a clamd startup script.

    4. Running the clamd daemon to scan for malware is recommended over the slower clamscan utility. clamd will need to be launched at system startup time. clamav comes with sample init.d scripts for the RedHat and SuSE Linux distributions.

  4. Setup the clamav.conf configuration file.

    5. Use the following clamav.conf to start with, but look over the default clamav.conf and other documentation to ensure the features, paths, and timeouts are properly configured for the system in question.

    • Ensure LocalSocket matches $ClamdSock in the mimedefang.pl script.

      • This will only be relevant if the installation paths of MIMEDefang have been altered; /var/spool/MIMEDefang/clamd.sock should work fine by default.

    • Run clamd as the MIMEDefang user.

      • $ grep defang clamav.conf
      User defang

    • Configure the ThreadTimeout and other limits as needed.

      • ThreadTimeout should be set lower than the mimedefang-multiplexor(8)-bbusyTime value, as well as the R timeout on the INPUT_MAIL_FILTER for MIMEDefang in the sendmail.mc file.

    Ensure that the defang user has permission to read clamav.conf.

  5. Check permissions on the definition database directory.

    6. $ ls -al /usr/local/share/clamav
    total 2400
    drwxr-xr-x 5 clamav clamav 170 Jun 28 16:37 .
    drwxr-xr-x 9 root wheel 306 Jun 23 19:18 ..
    -rw-r--r-- 1 root wheel 86 Jun 23 19:18 mirrors.txt
    -rw-r--r-- 1 clamav clamav 1109995 Jun 23 19:18 viruses.db
    -rw-r--r-- 1 clamav clamav 113484 Jun 28 16:37 viruses.db2

  6. Run freshclam to update the definitions.

    7. The definitions in the source or port may be outdated due to new malware being released.

    $ sudo freshclam --http-proxy=proxy:8000
    Current working dir is /usr/local/share/clamav
    Checking for a new database - started at Sat Jun 28 16:37:04 2003
    …

Install MIMEDefang

See the MIMEDefang Howto for general MIMEDefang installation instructions, or look for a port or package for the system in question. The following list details install choices that may affect later notes in this documentation.

Configuration

Replace with Warning | Reject all Malware | Discard Malware | Quarantines

Once the sendmail, mimedefang-multiplexor, and clamd daemons are running properly, configuration of MIMEDefang takes place via custom Perl code in the /etc/mail/mimedefang-filter file. This file must be edited to include code that both calls clamd to scan messages and determines what to do based on the result. Responses include rejecting or discarding the message at the Simple Mail Transport Protocol (SMTP) level, replacing the offending attachment with a warning message, and more. See mimedefang-filter(5) for details on the various message handling functions. Several malware handling strategies are outlined below; the example code will need to be added properly to the existing mimedefang-filter. To test that changes will at minimum load properly, use the -test argument to mimedefang.pl.

$ mimedefang.pl -f /etc/mail/mimedefang-filter -test
Filter /etc/mail/mimedefang-filter seems syntactically correct.

To reload the /etc/mail/mimedefang-filter file after changing it, most sample init.d scripts for MIMEDefang have a reread option.

$ sudo /usr/local/etc/rc.d/mimedefang reread

MIMEDefang can also scan messages for spam via Mail::SpamAssassin and other means; malware scanning should be done first, if possible. Spam scanning can be time consuming, and spam is typically tagged for user action, while malware rejected due to the risks posed by their content.

Replace with Warning

Use action_drop_with_warning or action_replace_with_warning to remove the malware content and insert a custom warning message. This method helps where malware attaches itself to an otherwise important document; the user will by able to notify the sender to clean up their system and resend the document. On the other hand, if a large number of malware are received, the user will have to spend time cleaning the messages out; altering the subject with a warning tag should help identify the messages with warnings.

In the case of Sobig.F in late August 2003, several users complained about the sheer volume of defanged messages being forwarded to them from their previous location. I consider discarding the malware (and bounces related to malware) and sending an optional periodic report of e-mail activity for the user (“1,254 dropped today from the following senders…”) the best practice at present.

sub filter_begin () {

my ($code, $category, $action) = message_contains_virus_clamd();

if ($category eq 'virus' and $VirusName ne 'Eicar-Test-Signature') {

$FoundVirus = 1;

action_change_header('Subject', '*** VIRUS *** ' . $Subject);
action_delete_all_headers('X-Virus-Status');
action_add_header('X-Virus-Status', "Yes, name=$VirusName");

} elsif ($category ne 'ok') {

md_syslog('err',
"$QueueID: clamd error: code=$code, category=$category, action=$action");
action_tempfail("error: problem running virus scanner");
return;

}

}

sub filter ($$$$) {
my ($entity, $fname, $ext, $type) = @_;
return if message_rejected();

if ($FoundVirus) {
my ($code, $category, $action);
$VirusScannerMessages = "";

($code, $category, $action) = entity_contains_virus_clamd($entity);

if ($category eq "virus") {
return action_drop_with_warning(
"Dropped $fname ($type) containing virus $VirusName.");

} elsif ($category ne 'ok') {

md_syslog('err',
"$QueueID: clamd error: code=$code, category=$category, action=$action"
);
action_tempfail("error: problem running virus scanner");
return;

}
}

}

MIMEDefang scans all messages passing through the server, whether incoming or outgoing. Outgoing malware represents an internal problem one may not want to advertise to external systems. Additional perl code would be required to determine which messages are outgoing, and to discard or reject them instead of letting them escape to the Internet as warning messages.

Reject all Malware

The action_bounce routine will return a notification to the sender of the message in question. However, the sender may not see this message, depending on how their e-mail system handles the Delivery Status Notification (DSN), or they may see false accusations due to malware that fake the sender information, such as Bugbear or Klez. The following example shows fallback to clamscan should clamd fail, and properly discards messages containing malware known to forge the sender information.

This method requires that the list of malware known to forge the sender address be manually updated as new malware are created. I currently favor the discard all malware method, below.

sub filter_begin () {


my ($code, $category, $action) = message_contains_virus_clamd();

if ($category eq 'virus') {
$FoundVirus = 1;
} elsif ($category ne 'ok') {
md_syslog('err',
"$QueueID: clamd error: code=$code, category=$category, action=$action");

# try clamscan if clamd failed
my ($code, $category, $action) = message_contains_virus_clamav();

if ($category eq 'virus') {
$FoundVirus = 1;
} elsif ($category ne 'ok') {
md_syslog('err',
"$QueueID: clamav error: code=$code, category=$category, action=$action"
);
action_tempfail("error: problem running virus scanners");
return;
}
}


}

sub filter ($$$$) {
my ($entity, $fname, $ext, $type) = @_;
return if message_rejected(); # Avoid unnecessary work


# need to check each part, as a leading Eicar virus would otherwise
# allow a message containing a real virus to pass
if ($FoundVirus) {
$VirusScannerMessages = '';
my ($code, $category, $action) = entity_contains_virus_clamd($entity);

if ($category eq 'virus') {
return if handle_virus();
} elsif ($category ne 'ok') {
md_syslog('err',
"$QueueID: clamd error: code=$code, category=$category, action=$action"
);

# try clamscan if clamd failed
my ($code, $category, $action) = entity_contains_virus_clamav($entity);

if ($category eq 'virus') {
return if handle_virus();
} elsif ($category ne 'ok') {
md_syslog('err',
"$QueueID: clamav error: code=$code, category=$category, action=$action"
);
}
action_tempfail("error: problem running virus scanners");
return;
}
}


}


sub handle_virus {

md_graphdefang_log('virus', $VirusName, $RelayAddr);

if ($VirusName eq 'Eicar-Test-Signature') {
return 0;
}

action_quarantine_entire_message(
"virus queueid=$QueueID, relayaddr=$RelayAddr, name=$VirusName");

# discard known sender forging viruses
# TODO maintain this list as new malware written
if ($VirusName =~ /klez|bugbear|nimda|hybris|yaha|braid|sobig|fizzer|↵
palyh|mimail|Trojan\.Dropper|Gibe|Worm.Bagle|Worm.SCO|Worm.SomeFool/i) {
action_discard();
} else {
action_bounce("error: message appears to contain virus $VirusName");
}
return 1;
}

The list of malware known to forge sending information will need to be updated periodically to keep abreast with malware development.

Discard Malware

This approach runs the risk of nobody being notified should a legitimate message be eliminated, but may be necessary on a high-volume e-mail system, or where other methods are used to monitor and respond to the malware. For instance, action_quarantine_entire_message could be used before action_discard, and details about the message logged for processing by other scripts.

The following code will also discard DSN sent from sites that improperly bounce malware that forge the sender information. The sender for these messages will be <>, available in MIMEDefang as $Sender. However, allowing such messages to pass through would be a bad idea, as malware could easily be written to use the special <> sender. Also, not all systems use the proper <> sender when delivering bounces.

sub filter_begin {


my ($code, $category, $action) = message_contains_virus_clamd();
if ($category eq 'virus') {

$FoundVirus = 1;

} elsif ($category ne 'ok') {
md_syslog('err',
"$QueueID: clamd error: code=$code, category=$category, action=$action");
action_tempfail("error: problem running virus scanner");
return;
}


}

sub filter ($$$$) {
my ($entity, $fname, $ext, $type) = @_;
return if message_rejected(); # Avoid unnecessary work


# need to check each part, as a leading Eicar virus would otherwise
# allow a message containing a real virus to pass
if ($FoundVirus) {
$VirusScannerMessages = '';
my ($code, $category, $action) = entity_contains_virus_clamd($entity);

if ($category eq 'virus') {
md_graphdefang_log('virus', $VirusName, $RelayAddr);

unless ($VirusName eq 'Eicar-Test-Signature') {
action_quarantine_entire_message(
"virus queueid=$QueueID, relayaddr=$RelayAddr, name=$VirusName");
action_discard();
return;
}
} elsif ($category ne 'ok') {
md_syslog('err',
"$QueueID: clamd error: code=$code, category=$category, action=$action"
);
action_tempfail("error: problem running virus scanner");
return;
}
}


}

To send a notification for each message quarantined, use the send_quarantine_notifications function. Sites that discard large numbers of malware will not want to use this, and instead rely on reports generated from the logs, as 10,000 malware result in 10,000 notifications.

sub filter_end ($) {
my($entity) = @_;

send_quarantine_notifications();

}

Quarantines

Quarantine of suspicious messages allows an administrator to review the data. Should a message be improperly quarantined, it can be redelivered to the proper recipients; otherwise, the HEADERS file and system log data will reveal the sending system should an incident need to be reported. Scripts can be run periodically on the logs and quarantine directory to generate optional reports to users indicating how many messages and from whom were blocked on a daily or weekly basis, and also assist in gathering the required message details for use with incident reporting.

Some of the above code examples quarantine malware by default; see also the action_quarantine_entire_message and similar documentation under mimedefang-filter(5) for details on how to enable quarantines. Consider using a format easily parsable by a script for the quarantine subroutine argument to facilitate scripting, and include the reason why the quarantine is being done (spam, malware) along with other metadata.

action_quarantine_entire_message("virus queueid=$QueueID, relayaddr=$RelayAddr, ↵
name=$VirusName");

Quarantined messages will consume disk resources. Ensure automated tasks run to ideally clean up or at least warn about heavy disk usage in the /var/spool/MD-Quarantine quarantine directory. Also consider locating the quarantine directory on a partition that is not shared with any system partitions; excessive quarantines should not be able to stop the operation of the system or e-mail services.

# example cron job to clean up quarantine directory
45 7 * * * find /var/spool/MD-Quarantine -type d -mtime +3 -maxdepth 1 -print0 ↵
| xargs -0 rm -rf

Updating Definitions

Forcing Updates | clamd Notification

The anti-virus definitions will need to be updated as long as current Microsoft systems exist on the Internet. One way is to run the following up-avdefs script via crond periodically. This script calls freshclam(1), and reports on the update status via syslog.

$ sudo crontab -l | grep up-avdefs
53 1,7,13,19 * * * /usr/local/sbin/up-avdefs

up-avdefs will configure freshclam to use a proxy by default, and will sleep randomly to space out simultaneous runs from multiple client systems. With a proxy, systems that are delayed should get proxy cache hits on the updated definitions, which will reduce the load on the definition servers.

Forcing Updates

The up-avdefs script can also be run from the command line. With sudo(8) and OpenSSH public key authentication, a script can be created to force a definitions update on multiple servers from an administrative client system. This would usually be done in response to a known definitions update that otherwise would take some time to be processed by the periodic updates.

#!/bin/sh
# list of hosts to force malware definition updates on
for host in mail mx1 mx2; do
ssh -o PreferredAuthentications=publickey $host \
sudo /usr/bin/up-avdefs -f
done

To avoid sudo prompting for a password over the ssh link, add something like the following to the sudoers file on each e-mail server using visudo(8). For more information, see also sudoers(5).

# allow wheel group to update malware definitions
%wheel ALL=NOPASSWD: /usr/bin/up-avdefs

clamd Notification

As a consequence of running clamd as the defang user, and freshclam as the clamav user, freshclam will not be able to properly notify clamd of definition changes with the --daemon-notify argument. There are several ways to work around this problem.

To debug whether freshclam is working properly, use the --log-verbose option.

$ sudo freshclam --log-verbose --daemon-notify
Current working dir is /usr/local/share/clamav
Checking for a new database - started at Mon Jun 30 14:22:07 2003
Connected to clamav.elektrapro.com.
Reading md5 sum (viruses.md5): OK
viruses.db is up to date.
Reading md5 sum (viruses2.md5): OK
Downloading viruses.db2 .......................................................↵
.................... done
Database updated (containing in total 8736 signatures).
connect(): Permission denied
ERROR: Can't connect to clamd.
Database updated from clamav.elektrapro.com.

Testing

To ensure malware scanning is functioning properly, a set of messages containing various known malware should be saved to test the e-mail system with. One to use is the harmless Eicar test virus, in both the raw and archive formats.

A Worm.Sobig.F in MIMEDefang quarantine format allows testing from systems with sendmail, such as the e-mail server itself. Submit the included ENTIRE_MESSAGE file to a user (postmaster, for instance) via sendmail and see how the system handles it.

$ cd test-virus
$ sendmail postmaster < ENTIRE_MESSAGE

Using the above configurations for MIMEDefang and clamav, log messages should be sent to syslogd from both mimedefang.pl and clamd that will look something like the following. Configuration of syslogd is usually done in the /etc/syslog.conf configuration file.

Aug 27 22:38:04 <mail.info> mx1 mimedefang.pl[25039]: MDLOG,h7S5c3gj027594,virus,↵
Worm.Sobig.F,192.0.2.1,<user@example.org>,<user@example.org>,Re: Details

Aug 27 22:35:03 <local6.info> mx1 clamd[27575]: /var/spool/MIMEDefang/mdefang-h7↵
S5YKgj027556/Work/msg-25039-160.scr: Worm.Sobig.F FOUND

See also TESTVIRUS.org for third-party delivery of the EICAR test virus.

Logging

Some of the daemons and scripts mentioned above log to syslogd(8), which saves the data to logfiles as specified in /etc/syslog.conf. Interesting log data should be parsed for, reported about, and acted on. Ideally there should be continuous searching on the incoming logs for known problems, and periodic sweeps (usually daily) to report about unknown log entries and summarize activity. However, defining what is interesting and how to deal with it will vary by site and evolve over time. The Best Log Analysis Tools page has more information on analyzing logs.

Critical events to report on are usually learned about after problems occur, though rules can be created in advance by looking through the source code of applications for errors that will be logged, or when the problem points to a local system. The following example shows a swatch configuration entry that sends e-mail about MDLOG messages involving malware from hypothetical on-site subnets (192.168.0.0/24 and 10.0.0.0/8); that is, a local machine has sent malware, and should be investigated by a local administrator.

# look for virus logs that original from "local" subnets
watchfor /: MDLOG,[^,]+,virus,[^,]+,(192\.168\.0|10\.)/
mail=root,subject=LOG WARN: mail: on-site malware source
throttle 30:00,use=regex

Throttling of reporting is critical to prevent the monitoring system from generating too many alerts; swatch has the throttle parameter to configure such.

An example log entry that the above regular expression would match looks like the following.

Jul 17 11:24:21 ducky mimedefang.pl[6251]: MDLOG,h6GJwM3c016682,virus,↵
Worm.Sobig.F,10.0.1.124,<forged@example.org>,<user@example.com>,Re: Movie

Another example is flagging unsuccessful malware definition updates for review by an admin.

# custom up-avdefs warning (malware definition update problems)
watchfor /(?i)up-avdefs\[\d+\]: freshclam error: /
mail=root,subject=LOG NOTE: security: malware definition update problem
throttle 30:00,use=regex

On the other hand, common log events should be screened out from periodic reporting to better highlight uncommon logs. The following are various Perl regular expressions used to ignore logs from various applications mentioned above.

# clamd with LogSyslog enabled
clamd\[\d+\]: Daemon started\.
clamd\[\d+\]: Log file size limited to \d+ bytes\.
clamd\[\d+\]: Running as user \w+ \(UID \d+, GID \d+\)
clamd\[\d+\]: Reading databases from [\/\w.-]+
clamd\[\d+\]: Protecting against \d+ viruses\.
clamd\[\d+\]: Unix socket file [\/\w.-]+
clamd\[\d+\]: Setting connection queue length to \d+
clamd\[\d+\]: Maximal number of threads: \d+
clamd\[\d+\]: Archive: Archived file size limit set to \d+ bytes\.
clamd\[\d+\]: Archive: Recursion level limit set to \d+\.
clamd\[\d+\]: Archive: Files limit set to \d+\.
clamd\[\d+\]: Archive: Limited memory usage\.
clamd\[\d+\]: Archive support enabled\.
clamd\[\d+\]: Mail files support enabled\.
clamd\[\d+\]: Self checking every \d+ seconds\.
clamd\[\d+\]: Timeout set to \d+ seconds\.
clamd\[\d+\]: SelfCheck: Database status OK\.
clamd\[\d+\]: Socket file removed\.
clamd\[\d+\]: --- Stopped at \w+
clamd\[\d+\]: SelfCheck: Database modification detected\. Forcing reload\.
clamd\[\d+\]: Reading databases from [\/\w.-]+
clamd\[\d+\]: Database correctly reloaded \(\d+ viruses\)
clamd\[\d+\]: [\/\w.-]+: [\/\w.-]+ FOUND

# messages from mimedefang* processes
mimedefang-multiplexor\[\d+\]: stats
mimedefang-multiplexor\[\d+\]: Killing idle slave \d+ .+?(Idle timeout|↵
Slave has processed \d+ requests)
mimedefang-multiplexor\[\d+\]: Killing slave \d+ .+?: Slave has processed ↵
maxRequests requests
mimedefang-multiplexor\[\d+\]: Killing slave \d+ .+?: Idle timeout
mimedefang-multiplexor\[\d+\]: Reap: Idle slave \d+ .+? exited due to SIGTERM ↵
as expected
mimedefang-multiplexor\[\d+\]: Reap: Killed slave \d+ .*?exited normally
mimedefang-multiplexor\[\d+\]: Starting slave \d+ .+?:
mimedefang-multiplexor\[\d+\]: Slave \d+ resource usage:
mimedefang\.pl\[\d+\]: filter_\w+ said ACCEPT_AND_NO_MORE_FILTERING
mimedefang\.pl\[\d+\]: MDLOG,\w+,
mimedefang.+?: filter:.+?discard=1
mimedefang.+?: Discarding because filter instructed us to

# up-avdefs working fine
up-avdefs\[\d+\]: freshclam ok

NovoSial Icon

This document is released into the public domain. Version 1.3.4

Site provided by Peter N. Steinmetz