Earlier on, I wrote about the changes we're making for the XP Day Benelux 2005 session selection process. For XP2005, Vera Peeters has adapted some of these ideas already, and it seems to be working fine so far. The main change from the process of last year, is that all of those who hand in a session are also invited to participate in the review.

Vera worded the values,goals and condition for this review much more succinctly than I can, so I'll reproduce her invitation to session organisers here (with her permission):

We value communication, feedback, simplicity and courage. As a workshop review committee, we want:

• an open and transparant review process
• useful and valuable feedback for the submitters
• a lot of input to create a great conference
• an atmosphere that encourages information exchange
• participation and involvement of a large community

That is why we decided to invite all workshop submitters to participate in the WorkshopReviewCommittee. Everyone who accepts this invitation, agrees with the following rules:

• You review at least 1 proposal, according to the WorkshopReviewProcess
• You can choose which proposals you wish to review, from those proposals that don't have 3 reviewers yet
• The review forms are filled in on the wiki, which is password protected and accessible by the WorkshopReviewCommittee only
• The reviews are not anonymous

As a session organiser, I'm very happy with this, and it seems to work out very well in practice, as most reviews have now been completed. The sessions I sent in have not been fully reviewed yet, but so far I've received some excellent and detailed feedback. Wether a session is well received or not, does not matter so much - detailed feedback helps to improve the session itself as wel as its' description.

Also, the lack of anonimity (which is the norm for other conferences, and even tracks within the same conference) balances out the fact that (as Marc Evers pointed out) a session organiser can skew the results in his favour by being extra harsh on other sessions. If you do that, the other reviewers / session organisers will be on to you quickly, and you'll only damage your own reputation. With anonymous review, this is somewhat easier to pull of (or so some people think. If there are comments, style is quite recognizable...).

As far as support software goes, non anonymous reviews make it possible to use a wiki for the review, instead of a more elaborate review support system.