Quality data is the lifeblood of our industry and the foundation for insights. Currently, the industry is wrestling with a number of threats to quality data, including AI, bots, bad actors, and fraud. Unfortunately, the industry is also dealing with self-inflicted challenges, including insufficient panel management, poorly designed and lengthy surveys, an ineffective application of sampling technology, and ultimately a poor participant experience.
While we recognize the need for improvement, we understand the variables at work and can solve for these concerns.
Quirks Magazine recently published a couple of good articles on this topic.
“An open letter to the insights industry on data fraud”, written by Keith Rinzler of 1Q and co-signed by a group of industry professionals, client-side and agency-side.
“Client-side researcher strategies for protecting panel data integrity”, written by Efrain Ribeiro, Karine Pepin, Mary Beth Weber, Tia Maurer, Carrie Campbell
The arguments made in these articles are on point. Those of us focused on ensuring data integrity in our operations are already familiar with these issues and most have a process in place to manage the threats.
I was part of the team at AC Nielsen BASES building one of the earlier online panels in the late 90s, and have managed other panels since. The variables now are different, and the macro and micro dynamics are much different, but they can be solved.
Referenced in the articles noted above, as well as through various industry initiatives, a few trends and critical issues need more attention:
I went through this process years ago to better understand the concerns, and I often refer to this as ‘how to ruin an experience’.
Take a well-intended consumer and subject them to a survey router with unlimited survey opportunities. Once they’re in the machine and begin terming out at multiple surveys due to poor targeting or excessively lengthy and complex screeners, they’ll soon start to wonder why they did this to themselves. They just wanted to complete a survey and maybe earn a couple dollars.
One of two things will happen at this point - either the consumer decides to stop participating in surveys entirely or they decide to slightly change responses to ‘try to qualify’ for a survey. It’s human nature and we’re sometimes forcing a bad process and experience on well-intended consumers.
It’s clear to most of us that a 1-to-1 communication strategy with research participants, through email, text, or apps, is the gold standard. It allows researchers to engage directly with panelists and align opportunities with purpose, and to focus on the long-term health of the relationship. Many larger panels have abandoned this connection in favor of technology intended to increase supply. This allows for greater blending and reach, both well-intended, but it hasn’t worked as well as anyone would like. In some circumstances, larger panels working with a variety of partners struggle to identify from which panel or source a consumer joined your survey. That’s a problem. It impacts recontact rates, validation, and transparency.
Perhaps the best way to sum it up is to look at how we talk about sample now. We used to ‘send more invitations’ to get additional responses to a survey. Now we ‘push more traffic’ to the survey – a sign we’ve distanced ourselves from the human on the other side.
Field management by experienced researchers with solid instincts has always been critical to a project’s success. There are so many ways a project can go sideways – poor survey design, poor sampling/recruitment strategy, poor validation, poor quota and program management, poor planning – but a good Field Manager is the glue keeping it all together.
Technology helps our industry in many ways but also introduces a lot of new variables to the mix, including bots, AI, fraud, click farms, sample routers, and a lack of research acumen and alignment from partners. Field management requires more time and strategy than ever before. You’re probably thinking about the field managers who routinely save your projects by managing both the rigor/fundamentals of research and at the same time the variables that technology and new partnerships produce.
New partners are entering the scene with a renewed focus on quality, tapping into professional networks and social media for B2B validation and using phone-to-screenshare solutions for reaching consumers and professionals. We expect the usage of these and other creative recruitment services to grow. We’re also seeing more collaboration between tech and sampling partners, agencies, and clients to instill confidence in the processes that lead to quality data.
There are four key contributors to the quality equation and each plays a critical role.
With respect to sampling and quality, tech plays an important role in highlighting behind-the-scenes sampling concerns, from bot detection, excessive survey taking, AI, demographic validation, and other items not easily detected by others in our space. If you’re not utilizing a 3rd party solution for monitoring these things, you’re missing an important piece of the puzzle.
Tech companies are also responsible for introducing new solutions and platforms for the insights industry. Many of these are quite good and improve the experience for qualitative and quantitative research participation.
Sample and recruiting partners are on the front lines and able to detect and stop bad behavior or poor quality before it starts. They vet the sources for recruitment and manage the profiles for those participating. They also have unique visibility into the usage and response patterns from their sources and can take direct action when needed. Automation and sample routers are challenges that haven’t been sufficiently managed.
Gaining access to new and more diverse audiences is essential. Having a good sample partner working with you to understand the impact of sourcing and blending decisions, while also working with you on quality, is critical.
The research agency plays a unique and critical role in tackling data quality. We make decisions on the technology we use in research, the sampling and recruiting partners to consider, and the steps to manage quality from start to finish. We manage the partners and the process from and are ultimately accountable for the overall execution.
Agencies who aren’t laser-focused on the contributors of quality and the impact of partner and process decisions, are not doing enough to support clients. A good quality program requires a deep understanding of the variables impacting data quality and a disciplined process for ensuring they’re managed properly.
More clients are getting involved in the process now than ever before, potentially as a result of their own poor experiences or other questions relating to data. More client involvement is welcome; they have a unique opportunity to drive change from the top. By working with quality partners who are aligned with them on the steps to quality, they help determine the path for winning partnerships in our industry. Partners who aren’t working to solve quality concerns will be left behind.
Recently a major manufacturer of consumer products decided to require partners to be ISO 20252 certified. While this step doesn’t guarantee to solve all of the quality concerns, it does put the industry on notice and require greater accountability.
The quality concerns are real and rightfully getting a lot of attention at the moment. But these problems can be solved through effective collaboration between partners and clients, by maintaining focus on the discipline and rigor of marketing research principles, and working to mitigate the threats through effective human and tech applications.
