using tcpwrappers with dovecot on fedora (aka denyhosts for dovecot)

(draft)

I've been planning to move my mail server off a cheap-ole-vps server for quite a while now. Recently I realised that they implemented a 300 thread limit for the vps, and this is killing my glassfish server occasionally, which is seriously heavy weight in its use of threads and dovecot login processes.

However due to laziness, and the fact there there is a big chunk of data in the form of Maildir and /var/lib/mysql is proving a blocker. Maybe a project for the downtime between xmas and new years.

Another limitation of the vps is a lack of access to the underlying iptables kernel module configuration which rules out fail2ban style systems for blocking brute force attacks against ssh.



However the problem with the iptables vps restriction is not sshd brute force attacks, because various strategies such as port-knocking or alternative service ports, the problem is dovecot brute force attacks.

Dovecot

In principle dovecot is a very secure system implementing a number of strategies to both restrict exploit damage, and provide highly secure chroot configurations. Combined with a user strong password policy, this should be little problem.

Combined with the vps limitations there are a few issues;

1) /var/log/secure full of brute force entries
2) logwatch full of the same
3) Dovecot launches a process per login, so given the process limitations above, it amounts to dos attack on the server


A work-around to these problems is to use Denyhosts to monitor the /var/log/secure file for authentication failures, and add the evil hosts to the /etc/hosts.deny file. There are 2 issues to resolve to get this working and they are

1) dovecot does not support tcpwrappers by default in the fedora package
2) denyhost does not check the authentication failure log entries produced by dovecot by deafult.



So getting this working is a 2 stage process.

1) download the source rpm for dovecot, rebuild with the tcpwrappers options and install the new package over the distro vanilla version.

2) add directives to /etc/denyhosts.conf to match the dovecot authentication failures log entries



Configuring Dovecot for tcpwrappers


dovecot added support for tcpwrappers in version 2.x??? however it requires that the binary be configured using the enable tcp wrappers options. Under fedora 14, the packged dovecot does not have that option...


hence a rpmbuild is required with a modified SPEC file, which is actually pretty simple

steps

1) download the src package from the repos
2) install the source package
3) edit the spec file
4) rebuild and install the package


optional

1) test tcpwrappers support by adding a line to /etc/hosts.deny and then try to login to dovecot;
dovecot:



Denyhosts

Denyhosts is a small python script which runs as a daemon under fedora, but can alternatively be configured via cron entries if you want that. It feels a bit like a hack because iptables is the intuitive place to block connections, but it seems to do the job well.

My one criticism was the lack of documentation on processing the custom log entries, but once I had a working example success quickly followed.
 
On fedora the daemon is configured for checking the log every 30 seconds, so in an unlucky run multiple attackers might get 29.9ish seconds to open a whole bunch of connections for dovecot logins, which might reach the 300 mark and shut out genuine users.

To deal with this problem you can tune the /etc/dovecot.conf file to limit the number of processes available to a host, and particularly the number of process for non-authenticated connections. (I think the default is 10 @TODO - what is this set to, and what is a reasonable number?)


Configuring Denyhosts for Dovecot





Denyhosts requires 2 changes to the Regular expressions that control the log entry processing, one to make denyhosts look at dovecot authentication entries, and the second to find the ones that are authentication failures.

(these lines also include directives to parse and block vsftpd attacks)
I added the following lines to /etc/denyhosts.conf

SSHD_FORMAT_REGEX=.* (sshd.*:|\[sshd\]|pam_unix\(dovecot:auth\):|vsftpd.*:) (?P.*)
#USERDEF_FAILED_ENTRY_REGEX = dovecot.*authentication failure.*rhost=(?P.*)
#USERDEF_FAILED_ENTRY_REGEX = .*authentication\ failure;\ logname=\ uid=\d{1,3}\ euid=\d{1,3}\ tty=dovecot\ ruser=(?P.*)\ rhost=(?P.*)
USERDEF_FAILED_ENTRY_REGEX = authentication.*failure user=(?P.*) rhost=(?P\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3})
USERDEF_FAILED_ENTRY_REGEX=authentication failure.* rhost=(?P\S+)
\s+user=(?P\S+).*
USERDEF_FAILED_ENTRY_REGEX=authentication failure.* rhost=(?P\S+).*

Warnings against doing stupid things

Obviously you run the risk of blocking yourself and locking out genuine users who have either forgotten or reset their passwords. So prime the /etc/hosts.allow file with your own ip address and any corporate ranges that apply.

One idea I was thinking of was to add to hosts.allow any hosts that had previous authenticated successfully, so as to allow users these hosts scope to mess up passwords and details.

Another alternative is to purge the entries in a very short time like 30 minutes.