Clueless Comments About Surveys
Steven Den Beste has a post about some surveys he’s received recently. I think it’s a sarcastic post, but I can’t tell for sure as he seems overly serious in the majority of his writing.
Anyway, he has some comments about surveys and survey methods. I’ve read a bit about surveys after experiencing a disaster at work a little while ago. He seems to have two main points:
1) The surveyors could have answered most of their questions by examining the site themselves.
He writes:
It strike me that if an academic wants to learn about blogs, then the best approach is field work: read a bunch of ‘em. You can observe them in their native habitat without leaving the comfort of your desk, as long as you have a net connection.
Of course they could, but how would that be efficient? I suspect they are trying to collect data from more than one blogger. (Or is Steven’s ego that big?)
They are also asking questions that can’t be answered by reading someone’s blog. They probably think that they will get more truthful answers from a survey than reading possibly fictitious blogs. I can think of other reasons for a survey that probably complements “field work”.
2) The surveyors have preconceptions and a hypothesis they are trying to prove.
Again — of course. He says the surveyors are academics, so it’s not surprising they are trying to prove a hypothesis. What else do academics do and why is it wrong to gather research data via a survey?
The fact that Steven doesn’t seem to fit in the box the surveys are attempting to draw around him either means he is an outlier or, more likely, the survey is poorly constructed. It’s simply challenging to create a good survey. You have to decide on closed vs open questions, get the wording of the questions right, the order correct, keep it a reasonable length, and more. Then, you have to test it and pick an appropriate sample set.
One of the tips to good survey building is to be up-front about the purpose of the survey. As Steven writes, “What in the heck is this person trying to learn?” Since the surveyor wasn’t very clear about that, everyone who answers the questions will do so from a different context. The collection of responses will be impossible to compare meaningfully.
But, the real lesson from his post is that the recipient does not have to respond to, let alone read, the survey. And even if he does, he probably shouldn’t write about it. Not that I’d tell Steven what to write about!