Thinking Tools for Conference Reviewers

Recently, I’ve been reviewing session proposals for Agile 2011. It’s a rewarding job, even though I’m not getting paid for it. I was asked to help out, and said yes because I knew I would learn a lot from it. Since this is the first time I’m reviewing sessions like this, I started out by doing some research into what to look for.

As a reviewer, my job is not to accept or reject proposals. So what is it? Well, my assumptions while reviewing sessions have been that the purpose of my review is to:

• increase the chances of the proposal becoming accepted
• help the session be useful for its participants
• make the session easy to understand and select for participants

I already have way too much information inside my head. Reviewing conference proposals from a bunch of very smart people seemed like something that would fill my brain right up and jam it. Therefore, I realized I needed to use some kind of tool to manage the review process. It’s really not different from the monkey that decides to use a stick to get to the banana. I just needed to find the proper stick.

Right now, my favorite thinking tool is the framework. For example, I just love Esther Derby’s and Diana Larsen’s retrospective framework. It’s versatile and powerful, and I can plug all sorts of fun work right into it. I realized what I needed was a framework for my work on reviewing sessions. However, I also needed to get started on the reviews. For that reason, I pulled out what might be the most basic framework of all. After all, thinking up frameworks can be the ultimate yak-shaving activity, I didn’t want to go there. I’m busy enough as it is.

I don’t know if it already has a name, but I’m naming this basic framework anyway. I’m going to call it the Common Questions framework. You already know the Common Questions, so this won’t be news to you. Here they are:

• Why?
• Who?
• What?
• How?
• When?
• Where?

For the review work, I settled for the first four questions: why, who, what, and how. As I researched and reflected on what to look for (Mark Levison sent me some tips, and a link to a great post my Mitch Lacey) I sorted the things I found into the above four categories. I also added one final thing: my bottom line, a super brief summary of my full review. If I can express something very briefly, that probably means my thoughts are clear to me. If I can’t, that’s my signal to think some more.

Here’s how I use these questions. Having read a proposal, I scan through the questions and write things down as they pop up in my head. Having done a first pass, I work through the questions one by one again, this time going back to the proposal and looking for answers to each question, which – of course – I write down. The set of text snippets I get from this isn’t necessarily very easy to read, so I end the process by working them all into a text that flows reasonably well from start to end. Finally, I try to summarize my thoughts in a single sentence.

I posted these questions to my fellow stage reviewers in case they could be useful to someone else but me, and to get feedback on them. A couple of the other reviewers suggested that I publish this as a blog post as soon as possible, as a resource to would-be submitters to Agile 2011. I would have done it sooner, but this week has been crazy.

In case someone still finds use for it, here’s my set of questions.

– – –

Why

The title and introduction will be used to “sell” the session to participants, so they need to not only clearly explain the session, but also read smoothly. The title and the intro will be used in the public conference schedule.

• Does the title help me see if this session is for me?
• What about the intro – can I make a first judgement if this fits for me, based on it?
• Are there intended learning outcomes? … and does it seem like the session design will lead to those?
• Does anything more need to be said about the session’s clarity of purpose?

Who

Quality is value to some person, therefore, we need to know who the session is intended for to be able to judge quality.

• Do I understand what the intended audience is, and how they would benefit from this session?
• Are specific roles mentioned? If not, how is the audience explained?
• What is said about the number of participants?
• Does the stated level (beginner, expert, etc) seem right? In other words: is this session really for beginners/experts/etc?

What

What are the contents of this session?

• Are the contents specific? … or is it hard for me to understand what will really happen in the session?
• Is the topic about agile?
• Is the topic timely, fresh, and unique? … or is it old, stale and has been done too many times before?

How

What can be said about the process to be used in this session? (Also referred to as “mechanics”)

• Does the description match the stated format? Formats can be, for example: workshop, tutorial, talk, etc.
• Is the text well written and well organized?
• Does the size of the proposal match the size of the session? In other words: if it’s a longer/larger session, is the amount of detail greater?
• How will the host be able to scale the session up and down as needed? The number of participants can turn out to be smaller or larger than expected.
• If the session is big and long: is there more than one host to help make it work?
• Does the session format model what is taught? For example: Talking about coaching is less helpful than practicing it.
• How is interactivity handled?
• Previous experiences with this? Has the session been run before by this host? What is the speaker’s experience level? Any references available?
• Is there a brief but clear speaker bio, that helps me judge if this session host fits me?
• In general, does the design seem feasible?

My Bottom Line?

– – –

Do you have experience of reviewing conference sessions? What are your best tips? What do you think about the questions above?