Critical Foundations of Survey Design
Surveys are a much used tool for gathering quantitative and qualitative evaluation data.
On the surface, a survey is simply a list of questions. Easy! Just write down what you want to know about someone’s perceptions, knowledge, attitudes or behaviours, and press send.
Yes, and no.
Surveys are only valid, reliable and useful when underpinned by thoughtful application of survey design principles.
So, what are these principles, and how do you apply them?
Andrew Hawkins recently shared his five golden rules for survey design, drawing on his background in psychometrics and deep knowledge of evaluation and research methodology.
1. Put yourself in the shoes of the respondent
Think about how you can make the survey clear, attractive (and as short as possible!) to make people want to fill it out. Remember; they are doing you a favour.
2. Quantify what you must ‘confirm or deny’ with closed items.
If you’re testing a hypothesis, it’s best to do so with closed questions (multiple choice, rating scales, rank orders), which generate quantitative data.
3. Seek out the unknown with open-ended text responses that you code and analyse afterwards.
If you’re exploring alternative hypotheses, use open ended (short answer or text box) responses. You can thematically analyses these responses, code them and do other analysis to strengthen your insights.
4. Go for more responses to fewer items.
Shorter surveys are easier to complete and tend to get more responses. A higher response rate means your findings are more likely to be statistically representative or valid.
5. Expect the magic to come from the analysis, not the number and variety of survey items.
Thoughtful analysis of key questions will generate stronger insights than adding more questions. This is because peoples’ opinions tend to relatively similar. For example, there tend to be a narrow range of reasons why people are or aren’t satisfied with a program or service, but the people who use the program or service will have a broad range of characteristics. By collecting simple demographics, you can explore patterns in responses.
Want access to the full hour-long webinar? It’s free, just sign up to our newsletter.
Need Help?
We can provide applied workshops that combine our extensive knowledge of survey design with your team members’ expertise. In these workshops, we work together to analyse your existing or future surveys and refine them to ensure meaningful data for use.
Give us a call to discuss how our applied workshops can help you with your next survey, on (02) 9373 9900
