Survey Says! how to create a meaningful survey
We’ve all been there. The interminable program planning meeting. Suggestions tossed out and shot down. The unanswerable over-riding question: what do people really want? How can we ever know? If we offer it, will they come?
And then someone utters the magic words, “I know! Let’s send out a survey and ask people what they want!”
Great! Sign up for a free Survey Monkey account and pose your questions to the parish! Right?
Maybe not so fast.
While digital surveys can be excellent tools to elicit information and engage parishioners in a decision-making process, they are also difficult to design and execute well. There is a reason a whole school of academic discipline exists focusing on effective survey techniques. A poorly designed survey can, at best, be a waste of your time and generate little usable data; at worst, it can alienate or offend people who have trusted you with their opinions.
What do you want to know?
Kate Huston is the Lay Minister for Formation at Saint Paul’s Cathedral in Oklahoma City and teaches courses in public policy, public administration, organizational theory, and American Politics at Oklahoma City University. When designing a survey, she recommends that rather than thinking first of what questions you will ask, you first consider what data and what type of data you want to generate.
The type of data you want will dictate the type of question structure you use – and even what digital platform you use to ask them.
Take a simple example – perhaps you want to determine what time a commission can hold a single meeting. A simple Doodle poll link with a selection of time slots might be sufficient.
But there are situations in which a similar question about meeting times might be structured to yield additional and more helpful information. Rather than a Doodle poll asking what time to meet for a Bible study based on a set number of options (essentially a multiple-choice question), you might restructure the question to ask people to rank their choices (a ranking question). Now you learn not only what times people are available, but their preference. Given the flexibility of your planning, you could even offer an open-ended question directed to people interested in Bible study and asking them to indicate when they would prefer to meet, leaving them to indicate their own time slots. Similar questions might better define exactly what it is people mean when they consider the term "bible study!"
Quantitative or Qualitative?
Another way to think about your data is whether you want quantitative or qualitative data. In helping congregations seek to make the best decisions possible for their mission and ministry, Kate utilizes tools from social science research including a variety of quantitative and qualitative survey techniques.
Quantitative data is information that can be measured and represented with numbers.
Qualitative data is derived more from peoples’ experiences and impressions. It is often more narrative in nature and doesn’t lend itself to simple answers.
The additional effort to read, review, and code stories can be well worth it, Kate explains. These kinds of questions and the data that they yield may offer unexpected but illuminating results, as well as reveal context, patterns, or trends.
Tanya Eustace Campen, who is the Director of Intergenerational Discipleship of the Rio Texas Conference of the United Methodist Church, also appreciates the results generated by qualitative approaches, which she uses extensively in her work on children’s spirituality. She says if you are curious to know and reflect peoples’ experiences, you want to elicit qualitative data.
Other Ways to Yield Data: Focus Groups, Interviews, and Observation
Of course surveys aren’t the only method to elicit data. Both Kate and Tanya talk about the value of interviews and focus groups. Tanya stresses that when designing a process to elicit qualitative data – whether one-on-one interviews, focus groups, or open-ended survey questions – think about who you are inviting to the table. You need to be intentional in making sure people are not only invited but feel safe to speak. Sometimes there is safety in numbers – but sometimes people are hesitant to speak freely. If you sense a reticence on the part of participants or notice you aren’t getting the feedback that you want, you may need to change your process.
Kate also reminds us that as we design both qualitative and quantitative processes, that we need to consider our own implicit biases and make sure we aren’t asking leading questions. Also, seeking feedback from a demographically broad population is often important to counter our biases.
Furthermore, it’s important to remember that the people who elect to respond to surveys or to participate in focus groups or interviews are often those people who have the strongest opinions – both positive and negative. If you sense you are getting polarized results, you may want to revisit your process. A multi-step process is sometimes effective. You might use a more open-ended qualitative process to establish your context; then use that information to construct a more quantitative survey.
Finally, it’s important to remember that there is a lot of useful information to be gleaned by observation. This might be quantitative – how many people attend church and how many remain for coffee hour? Make this observation regularly over a period of time – what are the averages? What are the trends? What external factors might influence participation on any particular date? Or you might consider qualitative observations – how are people engaging with one another? Are they animated? Are they talking across demographics? Does this change when you modify the room set up? When you provide food?
What to do with the Data
Once you’ve crafted your survey or process, publicized it, gotten a solid response from a cross-section of your population, now you just have to review the results and plan accordingly, right?
Yes and no.
An important part of the survey or other data-gathering process is follow-up. You need to not only use the data to formulate your plans, but you have an obligation to the people who participated. If you ask their opinions and they took the time to respond, you need to acknowledge their efforts and share your results. If people feel that they’ve given their opinion and been ignored, it would have been better if you never asked for their opinion in the first place!
Also, it’s important when you report back to people that you not gloss over any negatives you’ve uncovered. Presenting an overly rosy picture is inauthentic and counterproductive. Honestly acknowledging the challenges creates an environment where people may be more likely to be part of a solution.
Simple Survey Design
So if you do decide that a survey is the best process to gather information, here are a few basic tips from Kate:
- 10-15 questions is a good target size. Fewer than 5 and it’s not worth it; more than 15 and it can take too long.
- Include a mix of question types – multiple choice, ranking, and open-ended.
- Make sure your multiple-choice answers don’t overlap; also allow for “all of the above” or “none of the above” or “don’t know.”
- With multiple-choice questions, determine whether you want a single answer or if the question should allow users to select all that apply.
- Allow for anonymous responses. People tend to be more honest anonymously and will be less likely to provide the answer they think you want.
Kate reminds us that there may be times when a survey is overkill, for example when you truly only need information from a small subset of people. In that instance it may be more effective to simply call them! And if you consider nothing else, remember to always put the ANSWER before the question!
Tanya’s final words of wisdom remind us of the importance and necessity of doing the work to gather data. She suggests that if you are planning, thinking, and talking about certain group, the first thing you should do is to go and LISTEN to them and then ask what they think.
- Lisa Brown
Our goal at Membership Vision is to help churches and other faith communities to tell their stories in the digital space. Each church, irrespective of size, has a living and active story to tell, and technology provides an opportunity to share that story in a way that is welcoming and engaging. We ease the burden of keeping communications current, by leveraging content, and harnessing the many ways that members of our communities connect with each other, both inside and outside of the church walls. We aim to remove technological hurdles and allow churches to communicate online in an effective and sustainable way. Contact us firstname.lastname@example.org or call (805) 626-0143 to talk about the ways we can help your church build a digital presence.