def testUser
	result = `exim4 -bt user@domain.com`
	assert(result.include?('user@domain.com -> /home/user/Maildir/'))
end

Please note I changed the username actually used to a fake value, to not attract even more spam. The test above verifies the correct routing by checkng the final delivery to the users mail directory. I've got similar tests for wildcards and a mailing list, because these required adjustments to the configuration in order to work. I didn't add tests for spam yet, if anyone has some feel free to contact me :-).

These tests are not fully end to end, as they test only the routing on exim itself. Especially for the mailinglist software, these tests only check that exim delivers mail to the mailinglist, not what the mailinglist does after that. Therefore, I added two other tests, that use the smtp module provided by ruby , one for an ordindary user, and one for a mailinglist.

For the mailing list test, I set up a special test list, with only one subscriber on it. Letting the mailing list test fail for the first time was interesting, as it fell into the mailinglists error handling. I try to specify as little information as possible in the test mail, so the first one was rejected because it contained an implicit destination - in e-mail, the To: field is optional, so I had left it out for the plain e-mail test I adapted the mailinglist test from. So now I am sure asserting that the body of the message is present is the right thing to test for - the error message from mailman does not contain the message body, only some of the headers.

The mailinglist test (names have been changed, to protect the innocent) looks like this:

def testMailinglist
	`rm /home/user/Maildir/new/*`
	require 'net/smtp'
	msg = ["Subject: Test\n", "To: some_list@domain.com", "\n", "Now is the time for a mailinglist\n"]
	Net::SMTP.start('domain.com') do | smtp |
		smtp.sendmail(msg, 'user@domain.com', ['some_list@domain.com'])
	end
	
	sleep 5 #wait for delivery
	
	dir = Dir["/home/user/Maildir/new/*"]
	assert_equal(1, dir.size)
	filename = dir.first
	contents = IO.readlines(filename)
	assert(contents.include?("Now is the time for a mailinglist\n"), contents)
 end

Note the sleep in the middle - the test has to wait a few seconds for smtp, exim and mailman to do their job and deliver the mail to the directory of the test user.

I'm quite happy with the automated tests - the testscript also auto-generates the exim configuration file for me, so it functions as a sort of automated build for the mailserver configuration. There is nothing as reassuring as

7 tests, 9 assertions, 0 failures, 0 errors

If you'd like to have the full source code for the testcases, contact me directly, as I don't feel like search-and-replacing the e-mail addresses etc. in the test. Feedback on the way I set this up is also very welcome.