I was checking out the weather on Sydney Morning Herald, a web site for a newspaper here in Sydney, Australia, when a little window appeared asking me to take part in a survey so that they can understand their customers better. I’m always happy to complete surveys, in large part because I’m curious to see how other people structure them, and I always try to answer honestly. I’ve built and analysed many surveys, and people often underestimate the level of effort required in creating a survey structure that can generate useful data.
While I don’t know the objectives of the SMH survey, several aspects of it made me realise that they’re likely to end up with mis-representative and unusable data.
Early in the survey, it asked me for my main country of residence, so I picked “Australia”:
That was followed by a question about the state of Australia that I live in, so I picked “New South Wales (NSW)”:
The next question left me confused:
The question asked me for my monthly household income in US dollars. Considering the SMH is an Australian newspaper, and that I had just specified that I was living in Australia, asking me to try and specify my monthly income in a foreign currency felt just a little absurd (however, it’s not the first time that I’ve seen an Australian company do this in a survey). Some people may not realise it’s asking about another currency (or is that an unintentional mistake?), some might try to be helpful and convert their income to US dollars, some may pick the easy option of “Prefer not to say”, some may pick a random figure, and others may drop-out of the survey because it suddenly feels a little difficult or not applicable to them.
A few alternatives for the survey could be to ask for a monthly income in the currency of the country of residence, or, with the SMH’s presumably majority-Australian audience, it could ask for monthly income in Australian dollars while skipping the income question if the country of residence was not Australia. The decision to personalise or skip the question would depend on the objectives of the survey.
Another flaw in the survey was when they asked “Where do you access the Internet from?” but they provided mixed options like “Home” (a location) and “Smart Phone” (a device), and asked to “Please select all that apply” but provided the options as radio buttons so that only a single selection could be made:
Those weren’t the only issues I had with the survey, but they’re enough to point out that if you’re using surveys for research then their content and structure are vital and should be tested thoroughly, and that if you receive data from a survey it’s important to know how it was captured so that it can be analysed appropriately.
