Installing sendmail with aliases and procmail support

While the default sendmail that's included in most Linux distributions usually works satisfactorily for an individual workstation user, it's almost always necessary to build and install a better one manually in a production environment. That's because users could be sending mail through Outlook, Thunderbird, PC-Pine, or any number of other email clients.

This page describes how to compile and install sendmail with email aliases and procmail support. Email aliases let you create mailing lists and give alternate names or addresses for users. Procmail is a tool for filtering email to remove spam, sort mail, etc. The procedure for configuring procmail is described here and here.

If you are running a version sendmail earlier than 8.13.6, you need to upgrade because of a serious security problem. As of version 8.12, you must also create a new user (such as smmsp) and change the permissions of various files as described in the file sendmail/SECURITY.

Preparation

1. Back up the following files:

   /etc/mail/sendmail.cf /etc/sendmail.cf /etc/mail/submit.cf /usr/sbin/sendmail

2. Create a user and group for sendmail. This can be anything, but most people choose `mail' or `smmsp'. Many sendmail problems are caused by various files having the wrong ownership or permissions.

   useradd smmsp groupadd smmsp

   This step is essential, as sendmail will not start up unless it has its own userid.
   The entries in /etc/passwd and /etc/group should look something like this:
   

   cat /etc/passwd | grep smmsp smmsp:x:25:25:Sendmail:/home/smmsp:/bin/false cat /etc/group | grep ^smmsp smmsp:x:25:smmsp

Install libdb

If you don't care about email aliases, skip this step. Otherwise, if you don't have libdb on your system ('locate libdb'), get it from www.sleepycat.com and install it. For Unix/Posix systems:

cd db-4.2.52/build_unix ../dist/configure make su make install

Make a note of where the libraries were installed (e.g., /usr/local/BerkeleyDB.4.2/lib).

After installing, it may be necessary to perform these three extra steps (as root), depending on whether an old version of libdb is already present.

1. First, copy db.h to /usr/include. This is essential, as the db.h must match the library. If it doesn't, sendmail won't compile.

   cd db-4.2.52 cp build_unix/db.h /usr/include/

   
   
2. Second, get rid of the old libdb libraries.

   cd /usr/lib mkdir libdb-old mv libdb* libdb-old

   
   
3. Third, add the new libdb path (/usr/local/BerkeleyDB.4.2/lib) to /etc/ld.so.conf and reconfigure the run-time linker.

   vi /etc/ld.so.conf ldconfig

Compile and install sendmail

Compile sendmail. In some linux distributions, you will need to have the source code for a kernel installed before sendmail will compile. However, sometimes this can be avoided by creating an empty version.h file.

touch /usr/include/linux/version.h tar -xzvf sendmail.8.11.6.tar.gz cd send*

1. First, edit the config files if you plan to use procmail.

   cd cf/cf cp generic-linux.mc sendmail.mc

   Edit sendmail.mc and add

   MAILER(procmail)dnl

   at the end. Add any other commands as needed.
2. Create and install new config files using the following command:

   sh Build sendmail.cf sh Build submit.cf su make install-cf

   In earlier versions, the submit.cf file was not copied automatically if an old version of the file already existed. If you try to use an old version of submit.cf, when you start sendmail it will say:

   NOQUEUE: SYSERR(root): can not chdir(/var/spool/clientmqueue/): Permission denied

   This error can also occur if sendmail is running as the wrong user or group.
   
   Alternatively, install the config files manually.

   cd cf/cf cp config.cf /etc/mail/sendmail.cf cp submit.cf /etc/mail/

3. Create a devtools/Site/site.config.m4 file containing the paths for libdb. Change the paths to indicate wherever the libdb libraries and include files are located.

   APPENDDEF(`confMAPDEF', `-DNEWDB')dnl APPENDDEF(`confINCDIRS', `-I/usr/include/db2') APPENDDEF(`confLIBDIRS', `-L/usr/lib') APPENDDEF(`confLIBS', `-ldb')

   The exact syntax will vary depending on your system. On one of our computers, where we installed a new libdb, the file contained the following four lines:

   APPENDDEF(`confLIBS', `-lssl -lcrypto -ldb') APPENDDEF(`confLIBDIRS', `-L/usr/local/ssl/lib -L/usr/local/ssl/lib -L/usr/local/BerkeleyDB.4.2/lib -L/usr/lib') APPENDDEF(`confINCDIRS', `-I/usr/local/ssl/include -I/usr/include') APPENDDEF(`confMAPDEF', `-DNEWDB')dnl

   Sometimes compilation fails with undefined references to `pthread_mutex_trylock' and other pthread functions. If this happens, change the confLIBS line to

   APPENDDEF(`confLIBS', `-ldb -lpthread')

   
   NOTE: If you use the site.config.m4.example file that is included with sendmail, be sure to comment out items you don't want, or sendmail will probably not compile.
   
   See more uses of APPENDDEFs in "Thunderbird and APPENDDEFs" below.
4. Compile sendmail

   sh Build

   If you change any configuration files and need to compile a second time, use the -c option.

   sh Build -c

   The -c option rebuilds all the files to include your changes. It should automatically add the correct options to the commands, for example:

   cc -O2 -I. -I../../sendmail -I../../include \ -I/usr/include/db2 -DNEWDB -DNOT_SENDMAIL \ -c -o vacation.o vacation.c

   
   IMPORTANT: It's necessary to watch the screen continuously while sendmail is building, because the build script does not stop if there's an error.
   
   If your linux system is really old, you will get error messages like

   map.c:2084: `DB_FCNTL_LOCKING' undeclared (first use in this function)

   This means you need to install a newer copy of the Berkeley DB libraries from www.sleepycat.com.
5. Check to make sure it created an executable sendmail in the directory obj.Linux.#.#.##.i686/sendmail. If it bombs out, edit the file include/libsmdb/smdb.h, removing the lines

   # ifndef NDBM # ifndef NEWDB ERROR NDBM or NEWDB must be defined. # endif /* ! NEWDB */ # endif /* ! NDBM */

   and type sh Build -c again.
6. Install the new sendmail

   sh Build install

   Scroll back through the output and make sure there were no errors. If it says:

   install: invalid group `smmsp' make[1]: *** [install-sendmail] Error 1

   you must fix the groups and repeat.
7. Set the correct permissions and ownerships:

   mkdir /var/spool/clientmqueue chown smmsp:smmsp /var/spool/clientmqueue chmod 770 /var/spool/clientmqueue chmod 700 /var/spool/mqueue chown root.root /var/spool/mqueue chmod 777 /var/spool/mail chown root.root /var/spool/mail chmod 0555 /usr/sbin/sendmail chown root.smmsp /usr/sbin/sendmail

   NOTE: /var/spool/mail must be world read/write, otherwise clients can't open their inbox. The permissions for individual mailboxes for each user should be 600:

   -rw------- 1 daboss users 20219537754 2006-03-31 10:15 daboss

   
   Check to make sure of the following:

   -r-xr-sr-x root smmsp /usr/sbin/sendmail (as installed) -r-xr-xr-x root smmsp /usr/sbin/sendmail (sometimes necessary) drwxrwx--- smmsp smmsp /var/spool/clientmqueue (as installed) drwxrwxrwx smmsp smmsp /var/spool/clientmqueue (sometimes necessary) drwx------ root root /var/spool/mqueue -r--r--r-- root bin /etc/mail/sendmail.cf -r--r--r-- root bin /etc/mail/submit.cf

   NOTES:
   (1) The sendmail installation script sets the sendmail binary guid (r-xr-sr-x). With sendmail version 8.14, this now causes email clients such as pine to hang, regardless of the ownership of clientmqueue and mqueue. So we have found it necessary to set sendmail's permissions to 555 (chmod g-s sendmail) until we figure out what's happening.
   
   (2) Sendmail will always complain about the permissions on clientmqueue unless they are set to 0770. However, at least on Suse systems, this prevents ordinary users from sending mail. The characteristic symptom is that only root can send mail, while all others get an error message.
   
   (3) The outgoing queue file /var/spool/mqueue must be owned by root:root or sendmail will complain about a bogus queue file and throw away the message.
   
   
8. Add a relay-domains file in /etc/mail.
   This is a list of IP addresses or domain names that are permitted to relay mail, i.e. send mail to port 25 on your system. It should include the computers or domain names on your site, one per line. Example:

   mydomain.com 192.168.1.45 192.168.1.46

9. Add a local-host-names file in /etc/mail.

   cp /etc/sendmail.cw /etc/mail/local-host-names

   If local-host-names doesn't exist, create one. It should contain a single line:

   myhost.mydomain.org

   If the server has more than one name, add them to this file, one per line.

Thunderbird and APPENDDEFs

As mentioned above, an APPENDDEF is a compilation flag that may be needed to add functionality. For example, if you use the stock sendmail, Thunderbird users may have trouble sending mail. They will see the following error message:

The message could not be sent because connecting to SMTP server {your server name} failed. The server may be unavailable or is refusing SMTP connections. Please verify that your SMTP server setting is correct and try again, or else contact your network administrator.

In Linux, add the following to sendmail-8.14.4/devtools/Site/site.config.m4:

APPENDDEF(`confENVDEF', `-DSASL') APPENDDEF(`conf_sendmail_LIBS', `-lsasl')

The main docs at sendmail.org have instructions for other operating systems. See more uses of APPENDDEFs in Step 3 of "Compile and install sendmail" above.

We didn't have to use this option. Every time we re-install a new version of the OS, which contains the stock sendmail, Thunderbird users complain that they can't send mail. What appears to be happening is that T-bird is connecting to port 587 instead of 25 and sending mail by SMTP instead of using IMAP as specified in the account settings. For whatever reason, the stock sendmail never works, but when we compile a custom sendmail as described here the problem goes away.

Compilation problems

Sendmail usually compiles on Linux without a problem as long as libdb is set up. However, sometimes compilation bombs out with a slew of undefined references to do __dn_skipname and related stuff in domain.o. The solution is to add -lresolv to LIBS. Even the most recent version (sendmail-8.14.7 as of this writing) has this problem.

Solution: Edit sendmail-8.14.4/devtools/Site/site.config.m4 and add -lresolv to the confLIBS Appenddef like so:

APPENDDEF(`confLIBS', `-lssl -lcrypto -ldb -lresolv') APPENDDEF(`confLIBDIRS', `-L/usr/local/ssl/lib -L/usr/local/ssl/lib\ -L/usr/local/BerkeleyDB.4.2/lib -L/usr/lib') APPENDDEF(`confINCDIRS', `-I/usr/local/ssl/include -I/usr/include') APPENDDEF(`confMAPDEF', `-DMAP_REGEX') APPENDDEF(`confMAPDEF', `-DNEWDB') APPENDDEF(`conf_sendmail_ENVDEF', `-DMILTER')

Then rebuild sendmail with sh Build -c.

Testing sendmail

1. First, as root, test it to make sure the 'newaliases' command works. If it says

   Cannot rebuild aliases: no database format defined Cannot create database for alias file /etc/mail/aliases

   It means sendmail couldn't find the libdb files. Check that db.h is in /usr/include, that the new libraries are in their correct location, the old ones have been removed, and that /etc/ld.so.conf has a line indicating the path to the new libraries.
2. Restart inetd to make sure imap and pop3 aren't crashing.
3. Make sure sendmail is running. Unlike earlier versions, it sometimes now has two or more entries:

   ps -aux | grep mail root Mar30 0:01 sendmail: accepting connections smmsp Mar30 0:00 sendmail: Queue control smmsp Mar30 0:01 sendmail: running queue: /var/spool/clientmqueue

4. Test to make sure users can send and receive mail in all of the following:
   1. smtp client (e.g. pine)
   2. http (e.g. squirrelmail via browser)
   3. pop3 client (e.g. windows)
   4. imap4 client (e.g. windows)
   Also, check the system logs to make sure there are no weird error messages.
5. If you like pain and you want to change sendmail.cf, the easiest way is to edit sendmail.mc and then create it using

   m4 /etc/mail/sendmail.mc > /etc/mail/sendmail.cf

6. You may wish to test to ensure that your sendmail installation is not acting as an open mail relay by typing the command: telnet relay-test.mail-abuse.org There is also a web service at www.abuse.net/relay.html. You can download software to test it yourself from http://www.monkeys.com/mrt/. To be valid, the testing must be done using an account on a machine in some other domain (such as a temporary email account on abuse.net).
   
   However, testing this way is risky. If the testing software discovers that you're an open relay, your hostname or even your entire domain may end up in their public blacklist, thereby blocking valid email sent by your users. Last time I checked, the testing site www.ordb.org does this. We have found these blacklists are so inaccurate that we had to disable the blacklist check in spamassassin to avoid losing incoming mail. Thus, remote testing may do more harm than good.

Error messages

/usr/local/BerkeleyDB.4.2/lib/libdb.so: undefined reference to `pthread_condattr_setpshared'
/usr/local/BerkeleyDB.4.2/lib/libdb.so: undefined reference to `pthread_mutexattr_destroy'
/usr/local/BerkeleyDB.4.2/lib/libdb.so: undefined reference to `pthread_mutexattr_setpshared'
/usr/local/BerkeleyDB.4.2/lib/libdb.so: undefined reference to `pthread_mutexattr_init'
/usr/local/BerkeleyDB.4.2/lib/libdb.so: undefined reference to `pthread_mutex_trylock'

APPENDDEF(`confLIBS', `-ldb -lpthread')

 Initializing SMTP port (sendmail)/etc/mail/submit.cf: 
    line 430: readcf: option RunAsUser: unknown user smmsp: 
    No such file or directory 

su
Cannot rebuild aliases: no database format defined
Cannot create datase for alias file /etc/mail/aliases  

SMTPD_LISTEN_REMOTE="yes"

 APPENDDEF(`confMAPDEF', `-DMAP_REGEX')

See also Pine email problems.

Tweaking sendmail configuration

1. Changing hostname: Some computers are on two different domains. To get sendmail to use the second domain in its headers instead of your FQDN, add the following to submit.mc and sendmail.mc:

   FEATURE(always_add_domain)dnl MASQUERADE_AS(got-syphilis.seconddomain.org)dnl FEATURE(masquerade_envelope)dnl

   It may also be necessary to change the settings in your email client. For example, in pine, if you have the line:

   customized-hdrs = Reply-To: nbonaparte@grand-armee.fr

   change it to

   Reply-To: nbonaparte@got-syphilis.seconddomain.org

2. Slowing down dictionary attacks and distributed denial of service (DDoS) attacks: (this information is from http://www.technoids.org/dossed.html).
   Add the following to sendmail.mc:

   # block dos attacks # no of seconds connection rate FEATURE(`access_db')dnl FEATURE(`ratecontrol', ,`terminate')dnl define(`confCONNECTION_RATE_WINDOW_SIZE',`10')dnl # no of connections simultaneously define(`confCONNECTION_RATE_THROTTLE', `8')dnl FEATURE(`conncontrol', ,`terminate')dnl define(`confBAD_RCPT_THROTTLE',`10')dnl FEATURE(`greet_pause', `5000')dnl # stop dictionary attacks define(`confMAX_RCPTS_PER_MESSAGE', `10')dnl

   WARNING: The above options can seriously slow down outgoing email messages if you send a large attachment to multiple recipients. Test before using. We have stopped using these features, but they might be useful if your mail server is frequently under attack.
3. Stopping server from sending receipts: Some people try to check up on you by requesting a receipt when the email is delivered. Spammers may also use this to validate your users' email addresses.

   # no receipts on delivery define(`confPRIVACY_FLAGS', `noreceipts')dnl

   Check your email client; some clients send receipts as well.
4. New anti-spam features in sendmail version 8.14:

   FEATURE(`require_rdns')dnl FEATURE(`block_bad_helo')dnl FEATURE(`badmx')dnl

   The first rule will reject any incoming message from servers which don't have a proper reverse DNS. The second one blocks messages that have an unqualified HELO/EHLO argument, or that uses one of our names instead of the actual client name. The third one blocks senders who have bad MX records. These are all features of spammers.
   
   MAP_REGEX must be in your site.config.m4 file for the third one to work.
   
   NOTE: These rules also slow down outgoing emails that have multiple recipients. Use at your own risk. They only work in Sendmail 8.14 or higher.

After changing the .mc files, run sh Build sendmail.cf, sh Build submit.cf, su, make install-cf as before and restart sendmail.

Sample sendmail.mc file

This is the sendmail.mc file we use on this system. Some of the features are specific for version 8.14.

# no receipts on delivery define(`confPRIVACY_FLAGS', `noreceipts')dnl # use other domain name dnl# FEATURE(always_add_domain)dnl dnl# MASQUERADE_AS(my_hostname.com)dnl dnl# FEATURE(masquerade_envelope)dnl # dkim INPUT_MAIL_FILTER(`dkim-filter', `S=inet:8891@localhost') #anti-spam features for sendmail 8.14 FEATURE(`require_rdns')dnl FEATURE(`block_bad_helo')dnl FEATURE(`badmx')dnl divert(0)dnl VERSIONID(`$Id: generic-linux.mc,v 8.1 1999/09/24 22:48:05 gshapiro Exp $') OSTYPE(linux)dnl DOMAIN(generic)dnl MAILER(local)dnl MAILER(smtp)dnl MAILER(procmail)dnl

This is the site.config.m4 file we use on this system (entropy).

APPENDDEF(`confMAPDEF', `-DMAP_REGEX') APPENDDEF(`conf_sendmail_ENVDEF', `-DMILTER')

This is the site.config.m4 file we use on Engram.

APPENDDEF(`confLIBS', `-lssl -lcrypto -ldb') APPENDDEF(`confLIBDIRS', `-L/usr/local/ssl/lib -L/usr/local/ssl/lib -L/usr/local/BerkeleyDB.4.2/lib -L/usr/lib') APPENDDEF(`confINCDIRS', `-I/usr/local/ssl/include -I/usr/include') APPENDDEF(`confMAPDEF', `-DNEWDB')dnl APPENDDEF(`conf_sendmail_ENVDEF', `-DMILTER')