Why are research surveys always so bad?
Why are research surveys so bad?
I just completed a research survey, called SNAAP, targeted at art school alumni. It was bad. Like uncalibrated 5-point Likert scales, multiple-choice questions that don’t cover all possibilities, no “maybe” for yes/no questions where people have legit reasons to answer neither yes nor no, no Back buttons, optional questions where you can’t reset your answer, etc. — like, the usual mistakes.
Actually this one was worse than usual: It had a few free-form questions that had too little space for the kind of information they asked for. I took the time to shorten my answer the first time; the second time I was so pissed I just wrote “your field is too short”.
And do people actually time their surveys? Surely not? This one claimed it shouldn’t take more than 20 minutes[Note 1]; it took me 45 (yes, I timed it — using Toggl — I suppose 2x isn’t really any worse than the usual mis-estimate).
I mean, I actually had to dig out my records to check what I did two years ago. How did they come up with that 20-minute estimate when they’re asking people to dig out records not once, but like three times?
With a survey designed like this they’re going to get invalid data. Aren’t researchers interested in valid data? Why can’t they test their surveys before they send it out?
Oh wait. I made a survey myself for my own thesis so I know how little time we have.
Maybe a grad student wrote it. Sigh. Notes