We all recognize the value of alumni surveys. We also know they’re resource intensive and that we only get to do them every few years, so it’s important to get them right. Even the smallest design flaw might mean we lose responses from hundreds of alumni.
Sara Hunt Munoz, Senior Director for Strategic Initiatives in Fordham’s Office of Alumni Relations, recently shared her team’s experience with improving their alumni survey at the CASE District 2 conference. I recently spoke to her about her presentation and asked her to share her insight.
Let’s start with the basic premise that the alumni survey can be improved. How can it be improved?
We only had a 3.1% participation rate the first time we did it. It’s recommended that you do a survey every three years to collect longitudinal data. 2019 was our third year out from our first survey, so we decided to do it again. We learned a lot from our first survey and used that insight to make minor tweaks to our second survey to see if we could improve that participation rate.
We segmented the survey into pages so that people had to click “next” to advance. Any time they clicked next, the previous page’s data was saved. Our first survey was one long scroll, essentially one page. If somebody abandoned the survey, we were not able to collect that data because it wasn’t saved. We actually had about 800 people abandon our 2019 survey without completing it, but because of the way we set it up, we were still able to collect the data that they had given us—it wasn’t all for naught. They may have filled out the first few pages and been bored or busy and not able to finish the entire thing.
We got feedback after the last survey in 2016 that the survey was long and cumbersome, so we tried to simplify it by giving alumni a progress bar to let them know how much more time it would take to complete it. Last time, we didn’t have that at all, and people felt that there wasn’t an end in sight. Having a progress bar made them stick with it longer.
We also personalized the survey. Obviously, some of it feels canned because it’s coming from a company that does this for other universities, so we tried to make it feel like Fordham in its look, feel, and language. We had our logo on it, we used Fordham coloring as often as we could, we used an introductory letter from the head of our alumni association and the head of alumni relations to remind them that we had done this three years ago and how valuable the information was, and we highlighted a couple of things that we had put into place based on the feedback we had received just to show everybody that we’re listening and taking their feedback seriously. Then we thanked them for their time and introduced them to the survey.
Some of the questions we were given from the company we worked with—“How well does your university do at X?”—we changed to “How well does Fordham do at X?” to model the survey question after the vernacular. We made minor tweaks like that to be as Fordham-specific as possible and make it feel more personal to the university and to the person taking the survey.
Back to a point you made earlier about completion rates. In your 2016 survey, your participation rate and your completion rate were the same because someone had to complete the survey to participate at all. How does your new participation rate compare to your old one?
We were 3.1% participation the first year, and now we’re up to 5.38% participation in 2019. Still not great—we have 185,000 alumni, and only five percent of them took the survey—but the data is still statistically valid. And it shows that almost 4000 people did something for Fordham, that they were engaged in some way by taking the survey.
Did you improve the survey in others ways beyond copy tweaks and pagination?
Yes. The default questions are compound questions, like “Which of these 12 things was most important to you as a student? Pick 3.” It’s a long survey because they’re multiple choice questions, so we tried to trim the fat. We looked at who answered which questions the first time, why, and how. We trimmed some questions about student experience. While important, there’s not a lot that alumni relations can do to improve student experience, so we tried to make those questions more general so they could inform alumni programming. “What clubs were you in when you were a student, what were your passions?” And then we had some open-ended areas where people could put in their own comments. We asked them about one faculty or staff person that was really influential for them as a student and let them talk about that person and what they did for them. Trimming down the multiple choice questions and giving them open text boxes to add their own comments in was helpful.
When you were changing questions, were you concerned at all about keeping data consistent across surveys?
We knew it could affect the longitudinal data, so we were careful to make sure that the questions we kept were nearly identical to the ones we asked before. We tried to remove questions in their entirety that we thought weren’t as helpful as they could be. Sometimes, we tweaked some of the answer choices. We added diversity and inclusion because that’s become a real topic of interest that we didn’t touch on at all last time. When talking about student experience and alumni experience, we asked people to weigh in on how diversity and inclusion affected their experience.
Something else that is important is marketing the survey, getting the participation rate up in the first place. Did you change your marketing strategy for the survey from 2016 to 2019?
Yes, we did. One thing we did was acknowledge survey fatigue. In one of our email communications we said, “We get it: You eat at a restaurant, you get asked to take a survey. You stay at a hotel, you get asked to take a survey. We know we’re asking a lot, but please do this for Fordham, your input is valuable and we will do something with your feedback.” We acknowledged that people probably get 3-4 surveys a week and that we’re just one in their inbox. We promoted the survey everywhere, on social media, in emails, with flyers at events. We tried to get the word out there as often as possible and in every medium that we could.
I wrote a note to alumni staff, a peer-to-peer plea, telling them, “We know you have a unique relationship with the university, we want to hear more about it.” We think that was really effective.
We also sent one last push to people who had participated in our first survey, thanking them again for doing it, acknowledging that their feedback was important, and telling them “We did these three things based on what we learned last time,” and encouraged them to take it again. We also kept the survey open longer. We weren’t getting the returns that we wanted, so we kept it open a few more weeks and did a few more email blasts to see if it helped, and it did. It’s sort of arbitrary, you don’t need to have a firm end-date. We also started pushing incentives. We had a sweepstakes going for everyone who took the survey, they were entered into a drawing for five fabulous prizes!
It’s probably early to talk about what you’re doing with the results for this second survey, but can you speak to how you intend to use those data?
Sure. It’s going to take us 4-6 weeks to analyze all the data, and we just closed the survey a few weeks ago. We plan to share all the information with our fundraisers. My boss, who is the head of alumni relations, will meet with each dean and go over key takeaways from the survey. We'll also share it with our board members because there may be ideas or suggestions that they can use or that inspire them to make certain changes.
We learned a lot about our alumni last time, and we’re going to see if those things hold true. Last time, they said they weren’t getting the career support they wanted from the office of alumni relations, and they weren’t sure if they were still eligible for career services that are typically reserved for students. We were able to ramp up our career services offerings—we do more webinars now, we do workshops, and we also promote our career counsellors. We want to see if we’ve improved or if other things come to the surface that we need to spend more time and resources on. We learned last time that people want to hear about Fordham events in their area even if they can’t go. They want to know that Fordham’s in California and in Chicago and in Texas because it makes them feel warm and fuzzy about the university and they want to be able to brag that their alma mater is, say, going to see a Broadway show or opening a campus in South Africa. Even if we think, “Oh, they’re in Chicago, they’ll never want to hear about it,” they do want to hear about it. So we tweaked our monthly e-newsletter to include things that at one time we didn’t think people wanted to hear about.
Is the data you’re collecting anonymous?
The initial email we send out has a link to click on that is attached to their Fordham ID. We don't need to know who each individual is and don’t use that information, but that does allow us to tell whether they’re a donor, when they graduated, and different demographic data that is helpful for us when integrated into the results.
Is there anything else about improving the alumni survey process that is important to you that you want to share?
Yes. It’s important to continue communicating with your alumni even after the survey closes. We did a thank you on social media. In the newsletter that goes out next month, we’re going to pull some nuggets from the data and say, “5.38% of you took the survey this year, here’s what we’ve already learned, stay tuned for the full results later this spring.” We want them to know that we’ll share data with them as soon as we have them. In 2016, we made infographics with maps and bar graphs that were visually stimulating so alumni understood that we were taking the survey seriously and that it was going to inform our programming and marketing down the road.
And then that part I mentioned where people can pay tribute to staff or faculty: I went back and wrote a special message to all those people last time and pulled the quotes verbatim and put them in an email to them to build better relationships with our faculty and staff. It’s nice to have those touch points with faculty and staff members. That really tickled people. They felt good hearing that they had an impact on people from five years ago to people from fifty-five years ago. It’s a nice anecdote you don’t always get when you’re seeing someone at an event or speaking with them online about getting their ID card.