I was reading about set based development again, in the toyota way fieldbook and in the new book Mary Poppendieck is writing. If you want to participate in the review process, join the leandevelopment yahoogroup).

So, what does set based development have to do with looking for new stuff for the blog? Well, I'm still using the old rublog software for this blog. I'm trying several other things in private, with a couple of my friends to evaluate, and I hope to start using textpattern on a new blog (in dutch) that I'm planning to have soon (toyota way principle 13 - make decisions slowly by consensus, implement rapidly) now that I have found that textpattern supports most of my needs.

One of the nice things about not converging on a single solution early, is that you can allow yourself to be pleasantly surprised by what you find (even when you were not really looking :-) ). I compose electronic music every once in a while (think techno, ambient), and would like to have a podcast for that, because my friends keep asking me "and, did you make any music recently?" and I usually forget to mail around urls. While I was looking to resolve an issue I was having with textpattern, I stumbled across loudblog, a podcasting system. It took me only about fifteen minutes to set it up, so now I have a podcast, and my friends can subscripte to it if they want to keep up with what's happening musically :-) .

If I would have been totally focused on just selecting the one perfect blogging tool, I probably wouldn't have tried out loudblog and I wouldn't have had a podcast, but postponed selecting a podcasting tool until later. If I'd see bringing up the podcast in the light of set based development - it wasn't. I tried one thing, installed it, liked it and that's good enough for now (until I get dissatisfied with it and want something new ;-) )

I'm not the only one trying out new things, as I saw Johanna Rothman introducing the AYE blog.

Nico Mommaerts wrote me about his subversion experience, in response to my subversion troubles. He had data corruption problems too, they disappeared after he switched the file system type under the repository from berkeley db to fsfs.

Apparently, the subversion team feels the same – the default file system type for svnadmin create is now (version 1.20) fsfs instead of bdb (in a version 1.14 I had on another box). If you want to find out the file system type of a repository, there is a file called ‘fstype’ in the db directory of your repository that tells you.

I know several of this blog’s readers use subversion (I tend(ed?) to recommend subversion recently). I hope this post may save you a couple of frustrating hours hunting for a vague data corruption problem.

Yesterday I wanted to add some images to my website, from a computer I don’t normaly work on. Therefore I made a fresh checkout. Before completing the checkout, subversion exited with:

svn: Checksum mismatch on rep '1v6': 

and the two checksums that might be off. I finally found the solution for subversion checksum mismatch in the subversion users mailing list.

Several other workarounds suggested removing the offending directory and trying update, but that didn’t work (not even on a working copy). If you follow the instructions , you may be left wondering how to find the checksum in the representations.dump file as I was. I tried searching for the checksum, but that didn’t go so well.

The ‘1v6’ from my error message indicates a node number or something. There’s only one in the file. And, as instructions" say, you really do need 16 repeats of \00, like so:

1v6
 ((fulltext 1 2 (md5 16 \00\00\00\00\00\00\00\00\00\00\00\00\00\00\00\00)) 3 1v6)

I tried replacing it with the same amount of slashes as were there in the first place, but that didn’t work.

For future reference, I had this problem with svn, version 1.2.0 (r14790) on windows and reproduced it on debian linux, and still had the problem with version 1.2.3 (r15833).

I like subversion for all its’ features, but in moments like this, I guess I’d prefer a version control system that uses the filesystem as-is. Probably my repository has been corrupted since revision 2 (out of 130 revisions…) as svn dump stopped when dumping this revision. Apparently, I’m using this repository write-only, and subversion doesn’t see anything wrong when one one only perfomrs commits.

In case you wonder, the file in question was fine, it was only the checksum in the database that was corrupted…

In hindsight, creating a repository is quite straightforward. I decided to create a special svn user and group, so svnserve doesn’t have to run as root. With -r we can restrict the places where repositories can be, which also makes it easier for clients to specify urls.

  su -c "svnserve -d -r /var/repositories/"  svn

To access the reposotory remotely was easy,

  svn checkout svn://willemvandenende.com/svn/project

Only problem: the checkout is read only by default. Checking out read-write I wanted to do with authentication. Two things are needed for that:

• add username and password to /var/repositories/svn/paswd
• supply password and username when doing a checkout of the project
  • subversion will (only?) prompt for a username when anonymous access is set to ‘none’
  • the alternative is to supply —username (seen in ‘svn help checkout’ )

so a working checkout looks like this,

  svn checkout svn://willemvandenende.com/svn/project --username willem

def testUser
	result = `exim4 -bt user@domain.com`
	assert(result.include?('user@domain.com -> /home/user/Maildir/'))
end

Please note I changed the username actually used to a fake value, to not attract even more spam. The test above verifies the correct routing by checkng the final delivery to the users mail directory. I've got similar tests for wildcards and a mailing list, because these required adjustments to the configuration in order to work. I didn't add tests for spam yet, if anyone has some feel free to contact me :-).

These tests are not fully end to end, as they test only the routing on exim itself. Especially for the mailinglist software, these tests only check that exim delivers mail to the mailinglist, not what the mailinglist does after that. Therefore, I added two other tests, that use the smtp module provided by ruby , one for an ordindary user, and one for a mailinglist.

For the mailing list test, I set up a special test list, with only one subscriber on it. Letting the mailing list test fail for the first time was interesting, as it fell into the mailinglists error handling. I try to specify as little information as possible in the test mail, so the first one was rejected because it contained an implicit destination - in e-mail, the To: field is optional, so I had left it out for the plain e-mail test I adapted the mailinglist test from. So now I am sure asserting that the body of the message is present is the right thing to test for - the error message from mailman does not contain the message body, only some of the headers.

The mailinglist test (names have been changed, to protect the innocent) looks like this:

def testMailinglist
	`rm /home/user/Maildir/new/*`
	require 'net/smtp'
	msg = ["Subject: Test\n", "To: some_list@domain.com", "\n", "Now is the time for a mailinglist\n"]
	Net::SMTP.start('domain.com') do | smtp |
		smtp.sendmail(msg, 'user@domain.com', ['some_list@domain.com'])
	end
	
	sleep 5 #wait for delivery
	
	dir = Dir["/home/user/Maildir/new/*"]
	assert_equal(1, dir.size)
	filename = dir.first
	contents = IO.readlines(filename)
	assert(contents.include?("Now is the time for a mailinglist\n"), contents)
 end

Note the sleep in the middle - the test has to wait a few seconds for smtp, exim and mailman to do their job and deliver the mail to the directory of the test user.

I'm quite happy with the automated tests - the testscript also auto-generates the exim configuration file for me, so it functions as a sort of automated build for the mailserver configuration. There is nothing as reassuring as

7 tests, 9 assertions, 0 failures, 0 errors

If you'd like to have the full source code for the testcases, contact me directly, as I don't feel like search-and-replacing the e-mail addresses etc. in the test. Feedback on the way I set this up is also very welcome.

After taking a long break (tip: take a break when stuff like this doesn't work. It's usually more effective than labouring on). My mail cilent kmail is picky in what mail it downloads, depending on which button I press... So, for the next round I'll take the stuff I did on the command line and copy it into a test script, so I can see if the mail is still working (ordinary users, wildcards, mailinglists). Then I can use that as a basis to improve spam- and virusfiltering as well (spam is apparently still sent out over the mailinglists).

Configuring a mailserver is bewilderingly complex. With only an afternoon spent to upgrade about every package on my web- and mailserver a few years into the future, I may consider myself lucky - on other platforms than debian linux this would take quite a bit more work. Much of the old configuration was taken into consideration by upgrade scripts.

Nevertheless, I'm surprised that what I consider to be a fairly standard setup of mailserver, anti-spam and virus protection and mailinglists takes so much manual configuration. Most configuration I do is copy-paste from websites and mailinglists.

For my own memory and possibly your configuration pleasure, I include some of the details below:

Stepping through why the mailinglists didn't work yet, I found out that the *: willem alias wasn't working anymore. Exim configuration is much different from version 3 to version 4... Surprisingly, handling e-mail with a 'catch all' address does'nt work out of the box (anymore). See this message in the debian mailinglist how to create a catch all address in exim4.

Mailserver configuration seems so different, it is best to start over with the mailinglists and antispam configuration.

This installation log shows some useful defaults (including some nifty spam-filtering and anti-virus addons I haven't used yet) and commands to check the configuration.

exim4 -bV