Dana Chisnell workshop: Making sense of the data: Collaborative analysis techniques for user research

25 August, 2010 at UX Australia

Starting with a collaborative exercise, Dana took us through a KJ Analysis – an 8-step group prioritisation exercise invented by Jiro Kawakita. This got us out of our seats, scribbling on post-it notes, moving them about, and trying (very hard) to get them to stick to the pretty wallpaper that seemed to be making its own judgement calls on what was important and what was not.

The KJ is one part of a collaborative process Dana outlined for us over the afternoon whose steps are:

Tell stories – These are documented stories (in email, docs, wikis, blogs) about the people in the testing sessions. These can be written in a few hours over a drink and encourages the team to have a shared view of the users that will also engage and focus the stakeholders.

Rolling issues – A list of observed issues captured by the team on a white-board between research sessions to track recurring observations (both good and bad). Debriefing between sessions generates agreement on issues. The observations can then be transferred to a spreadsheet to consider the weighting of issues.

KJ – A team post-it note writing, grouping and prioritisation exercise that can take up to an hour. Dana suggested that the minimum number of participants for a KJ would be 5 or 6 but she’d heard of someone doing it on their own (though I can’t see how the technique would lend itself to that scenario!). You can run it with a hundred people if you want to – you just need a lot of wallspace and a lot of post-it notes.

Observation to direction – A spreadsheet is then used to list out each observation and its related inferences, opinions and directions. The time required to create the document depends on the number of issues. The first time it is run with a group of people it may take longer but it will speed up as they become familiar with the process.

Some key points I took from the workshop:

• Make a rule that you can’t participate in a KJ without attending at least a couple of the user research sessions.
• Creating a rolling issues list along with a KJ provides validation.
• A goal of these techniques is to create a shared experience among team members and stakeholders.
• It’s important to have process because it stops wasting time going around in circles. It provides focus.
• These techniques work well at any part of the design cycle.
• A “guess the reason” game (where you look at other people’s designs which fail and consider the reasons why) can help to warm up a team and avoid jumping to design solutions without taking all the steps.
• This process can remove the need for lengthy reporting because stakeholders are involved and aware of issues from the user perspective.
• “What comes up in a usability test is not always a usability problem. It can be a business-model problem.”

The slides don’t appear to be up yet however Dana has posted a similar presentation from UX Lx in May 2010 titled “Making smart design decisions – Collaborative analysis techniques”, and you can find out more about Dana on her Usability Testing blog.