Ambrose Li

Why are research surveys always so bad?

Why are research surveys so bad?

I just completed a research survey, called SNAAP, targeted at art school alumni. It was bad. Like uncalibrated 5-point Likert scales, multiple-choice questions that don’t cover all possibilities, no “maybe” for yes/no questions where people have legit reasons to answer neither yes nor no, no Back buttons, optional questions where you can’t reset your answer, etc. like, the usual mistakes.

Actually this one was worse than usual: It had a few free-form questions that had too little space for the kind of information they asked for. I took the time to shorten my answer the first time; the second time I was so pissed I just wrote “your field is too short”.

And do people actually time their surveys? Surely not? This one claimed it shouldn’t take more than 20 minutes‍[Note 1]; it took me 45 (yes, I timed it using Toggl I suppose 2x isn’t really any worse than the usual mis-estimate).

I mean, I actually had to dig out my records to check what I did two years ago. How did they come up with that 20-minute estimate when they’re asking people to dig out records not once, but like three times?

With a survey designed like this they’re going to get invalid data. Aren’t researchers interested in valid data? Why can’t they test their surveys before they send it out?

Oh wait. I made a survey myself for my own thesis so I know how little time we have. Maybe a grad student wrote it. Sigh.


  1. Yes, I double-checked; I took a screen shot.