The survey is all wrapped up now, and we are ready to present you with the results. In this article, you\u2019ll find out what recruiting mistakes UXers encounter most often, which ones they consider to be the most detrimental, and how you can prevent those mistakes from happening in your next study.<\/p>\n\n\n\n
7 Biggest Recruiting Mistakes + Tips to Overcome Them<\/h2>\n\n\n\n
When it comes to research recruiting mistakes, we asked 2 main questions:<\/p>\n\n\n\n
Question #1 – <\/p>\n\n\n\n
\u201cWhich of the following mistakes have you observed or experienced in research recruiting? (select all you experienced)\u201d <\/em><\/strong><\/p>\n\n\n\n This question is designed to uncover frequent missteps in the research recruitment process from the perspective of the respondents. By presenting a series of potential mistakes, it aims to shed light on which aspects of recruitment are most error-prone, gauge the prevalence of these issues, and identify key areas for improvement based on their frequency. <\/p>\n\n\n\n Question #2 – <\/em><\/strong><\/p>\n\n\n\n \u201cFrom the list, which mistake do you consider the most critical or detrimental in research recruiting?\u201d<\/em><\/strong> <\/p>\n\n\n\n This question aimed to pinpoint which mistake is perceived by participants as the most damaging or impactful in hindering the research recruitment process. The respondents were asked to highlight the most severe mistake from their previous selections, aiming to rank the problems in order of their potential to be detrimental to the research process.<\/p>\n\n\n\n The second question was followed by the open-ended question: <\/p>\n\n\n\n \u201cHow would you handle or overcome this mistake<\/strong>?\u201d, <\/p>\n\n\n\n offering the respondents a chance to propose their own solutions and share ways in which they recommend combating the mistake they identified as the most critical. <\/p>\n\n\n Based on the Research Recruities survey results<\/a>, the most commonly experienced research recruiting mistake is failing to properly screen the participants<\/strong>, which was the option chosen by 17.6 % of our respondents. <\/p>\n\n\n\n Following closely is the mistake of rushing the recruitment process<\/strong> (13.4 %) and not defining clear targeting criteria<\/strong> (10.9%).<\/strong> <\/p>\n\n\n\n The answers to the second question, however, painted a slightly different picture, clearly pointing out the 8 most detrimental UX research recruiting mistakes, according to our participants. Below, we\u2019ll take a look at each of them and their nuances and include tips from our respondents and UX experts for handling those mistakes.<\/p>\n\n\n\n This mistake was selected as the most challenging by most of the respondents \u2013 37,46%. In essence, it refers to the failure to establish precise and specific criteria for selecting study participants. <\/p>\n\n\n\n In many cases, this means including individuals who may not fully represent the actual characteristics, behaviors, or needs of the future use group. This mistake often leads to incorrect assumptions about user behavior and waste of resources on irrelevant research. <\/p>\n\n\n\n This problem often arises when the project scope is broad, or when there’s a rush to start the research without adequate planning. Additionally, it may stem from an insufficient knowledge of the intended user group or poorly defined research study goals.<\/p>\n\n\n\n One of the most common responses was defining the goals & hypothesis of the research clearly, talking to stakeholders, and, finally, \u201cspending a lot of time creating a good screening questionnaire\u201d. <\/p>\n\n\n\n Here is an example of how NOT to approach the process of recruiting a particular user group and what to do instead, from Parker Sorensen<\/a>, Associate Director, Conversion Optimization.<\/p>\n\n\n\n \u201cI believe this comes down to good test planning. I have seen (and in my early days or research did this myself) where researchers recruit a general audience and try to have participants imagine a scenario (ie. “imagine you are looking to purchase a home in the next 1-3 months”). In my experience, it is difficult to impossible for participants to put themselves fully in the scenario – because they are not actually there. A better way is to recruit the exact audience (ie. screen for “which of the following are you planning to do in the next 1-3 months”, give 5-10 options, and make “purchase a home” required.\u201d<\/em><\/p>\n\n\n\n For those at the early stages of their research, here\u2019s a great tip for creating specific targeting criteria from Blanche Letakis<\/a>, Associate Product Manager & UX Researcher:<\/p>\n\n\n\n \u201cI start research studies with a slightly broader target and as sessions go on, I learn more about our target users, and tweak the criteria, including screener surveys.\u201d<\/em><\/p>\n\n\n\n Screening is undeniably one of the most problematic aspects of UX research recruiting. It is also, according to our data, the most prevalent recruiting mistake. <\/p>\n\n\n\n Unfortunately, ineffective screening is a common problem because it is linked to a lack of a thorough screening procedure and poor screening criteria compared to the study\u2019s goals. <\/p>\n\n\n\n Poor screening harms your research study in many ways, including biasing study results, wasting resources, and allowing unqualified or \u201cfake\u201d participants to enter your study.<\/p>\n\n\n\n Here\u2019s what our respondents suggest for fine-tunning your screening strategy:<\/p>\n\n\n\n \u201cIf possible, test before launching to a wider audience, peer-review and opt for manual reviewing of the participants before getting the confidence that the screening process works as it should.\u201d<\/em> – Andreea Dalia Lazar, PhD<\/a>, Senior Researcher<\/p>\n\n\n\n \u201cUsing a research panel that allows UXers to filter for specific criteria, is the first step I try to take. Additionally, we use a variety of screener tactics to help ensure the people signing up or taking our unmoderated tests are the right people.\u201d<\/em> – Kelly Meredith<\/a>, Senior Design Researcher<\/p>\n\n\n\n \u201cAsk questions regarding the product but don\u2019t be too straightforward to minimise the amount of “professional respondents”\u201d<\/em> – Alyona Poluiko<\/a>, UX Researcher<\/p>\n\n\n\n Nikki Anderson<\/a>, Founder @ User Research Academy, also recommends crafting as detailed a screening questionnaire as possible:<\/p>\n\n\n\n \u201cThink through every piece of information you need from the participant and craft that into a screener question – it might mean the screener is a bit longer (try not to go more than seven questions) but it will enable you to get the right people for your study!\u201d<\/em><\/p>\n\n\n\n Convenience sampling is the selection of participants based on their ease of access or availability and not an accurate sampling of the target population. <\/p>\n\n\n\n The high dependency on this type of recruiting takes place when decision-makers select only convenient ways of getting access to participants. Such an approach results in no need to recruit people to take part in the study and instead use the services of colleagues, friends, and family members. <\/p>\n\n\n\n However, even though this method seems acceptable in some cases (for example, when you are recruiting from the general population), most of the time it still results in biased insights. <\/p>\n\n\n\n To counter this issue our respondents recommend:<\/p>\n\n\n\n Keeping a database of your past testers allows you to recruit them again and again for another study, and with that, you don\u2019t have to invest much time and effort in recruitment. It\u2019s a great alternative to convenience sampling that can actually provide useful insights.<\/p>\n\n\n\n With that being said, be sure to provide fair compensation, because it can directly affect the participants\u2019 willingness to participate in your next study.<\/p>\n\n\n\n \u201cCompensate in time! This is something my company is currently struggling with, but what I’ve found (from my past company) is that when we pay the participants on time, they are more willing to participate again and even refer their network to us for future studies.\u201d<\/em> – Karen Mirmanas<\/a>, UX, Inclusive Design and A11y ResearchOps Coordinator.<\/p>\n\n\n\n Here\u2019s another great tip for growing your database of respondents from Caitlin Sullivan<\/a>, Product Discovery and User Research Expert:<\/p>\n\n\n\n \u201cAsk every participant you find who is a really good fit if they know someone who might be up for joining your research. It can make recruiting the right kind of people much faster!\u201d<\/em><\/p>\n\n\n\n The term diversity in this context not only means demographic data such as the users\u2019 age, gender, and origin but also a variety of user habits, backgrounds, and access requirements.<\/p>\n\n\n\n When ignoring diversity you\u2019re risking that your product will be only catered to a narrow user base, neglecting the needs of others who didn’t fit the initial participant profile. <\/p>\n\n\n\n Of course, it’s crucial to remain focused on the specific user group your product or service is intended for. However, it\u2019s also critical to balance the importance of not overlooking diversity, while adhering to your targeting criteria. <\/p>\n\n\n\n To ensure you\u2019re looking at a variety of customer types, Kelly Meredith<\/a>, Senior Design Researcher, recommends:<\/p>\n\n\n\n \u201cWhen looking to talk to a variety of customer types, rather than sending one invite link and hoping for the best, we use a few methods to ensure we are talking to the right mix of customers. We use Calendly routing forms, which have been a huge help! We create a different event type per type of participant and use the routing form (screener) to filter prospective participants to the right event. <\/em><\/p>\n\n\n\n For example, we can chat with 5 large customers, 5 small customers, and 5 non-customers in the most efficient and effective way. Before this feature or for those without Calendly Pro, we use Alchemer to create a screener and use the logic to direct potential participants to different sign up links.\u201d<\/em><\/p>\n\n\n\n If you don’t carefully verify who’s joining the study, you risk having scammers complete the study and skew the results. These individuals, often looking to game the system for a monetary reward, can slip through if we’re not filtered out during the screening stage of the study. <\/p>\n\n\n\n Their presence isn’t just a numbers issue; it distorts the data, leading to insights that don’t truly reflect our target audience’s perspectives and behaviors. Filtering them out is crucial to ensure the integrity of the research data.<\/p>\n\n\n\n To safeguard your research from being compromised, it’s essential to verify participant information upfront. This step ensures that only qualified participants contribute to your study, maintaining the reliability of your findings.<\/p>\n\n\n\n \u201cDo not allow participants to schedule study sessions directly once they\u2019ve qualified through the screener. Teams of scammers will sometimes re-take screeners until they\u2019ve \u201cpassed,\u201d then use those answers to qualify under other additional identities or share the answers with other scammers. <\/em><\/p>\n\n\n\n Inviting participants to schedule their sessions takes longer, yet it offers additional benefits. It enables a secondary screening process, provides a chance to confirm the accuracy of the contact information they’ve provided, and allows you to cross-reference responses to identify potential duplicates among the screeners.\u201d<\/em> – Michele Ronsen<\/a>, Author, Researcher, Educator, Founder, UX Coach.<\/p>\n\n\n\n Here\u2019s how another respondent, a UX researcher with 5-10 years of experience recommends approaching that problem:<\/p>\n\n\n\n \u201cWrite better screener questions (no yes\/nos, etc.), and when things are tricky – call to verify.\u201d<\/em> – UX Researcher, 5-10 years of experience<\/p>\n\n\n\n Not checking whether participants have been part of similar studies before can also impact the integrity of your research. <\/p>\n\n\n\n People who’ve done this kind of testing\/interview before might not respond like new users would. They’ve got their own expectations and are too used to the process, which can color their feedback. <\/p>\n\n\n\n So, without the previous participation check, you’re missing out on fresh, unbiased insights, which are gold for getting a real understanding of what your target audience thinks or feels.<\/p>\n\n\n\n To dodge this issue, you can set up a system to track and screen out the participants who’ve been in similar studies before. This could mean keeping a record of past participants or using specific questions during the screening process to spot anyone who’s already taken part. This way, you ensure the feedback you’re getting is fresh and representative of new users.<\/p>\n\n\n\n If you recruit from a user panel of a UX research tool such as UXtweak<\/a>, they conduct this check themselves, so you don\u2019t have to worry about it.<\/p>\n\n\n\n When project deadlines are looming and budgets are tight, there’s a temptation to speed up the participant recruitment process to stay on track. <\/p>\n\n\n\n This pressure often comes from stakeholders as the main decision-makers. However, rushing the recruiting and just grabbing whoever’s available without thorough screening can backfire. This approach often results in a participant group that’s not diverse enough or truly reflective of the intended user base.<\/p>\n\n\n\n The irony is, while rushing might seem like a shortcut, it can actually lead to more work later. If the initial recruitment isn’t done right, you might end up having to redo parts of the study or need to conduct more research to compensate for the skewed results, eating up more time and resources than if you’d taken the time to recruit properly from the start.<\/p>\n\n\n\n Jasmine (Lo) Winata<\/a>, Research Operations Specialist, emphasizes the importance of informing stakeholders and communicating the need for a longer and more structured recruiting process:<\/p>\n\n\n\n \u201cPushback on stakeholders who are rushing recruitment and lay down the realities and expectations regarding recruitment, especially with a new pool (looking for brand new users vs. using a pool we already have access to). Let them know that recruitment will still happen, but the desired number and quality of users may not always meet expectations.\u201d<\/em> – Jasmine (Lo) Winata<\/a>, Research Operations Specialist<\/p>\n\n\n\n Even though the advice above already seems like a load of insights, we\u2019ve got a lot more of professional tips during the time of the Research Recruities project. Recruiting is a wide and complex topic, and there is so much more to discuss and learn! <\/p>\n\n\n\n Apart from the insights on the biggest research recruiting mistakes, we also collected hilarious and strange recruiting stories from researchers around the world, gathered general recruiting tips and strategies from UX pros like Stephanie Walter, Debbie Levitt, Nikki Anderson, Caitlin Sullivan, Michele Ronsen, Larry Marine and others!<\/p>\n\n\n\n \u27a1\ufe0f Read the stories: Hilarious & Strange: UXR Participant Recruiting Stories that You Need to Hear<\/a><\/p>\n\n\n\n![\"\"](\"https:\/\/uxmag.com\/wp-content\/uploads\/2024\/05\/image-10.png\")
<\/figure><\/div>\n\n\n1. Not defining clear targeting criteria <\/h2>\n\n\n\n
How to combat this mistake?<\/h3>\n\n\n\n
2. Not Screening Participants Properly<\/h2>\n\n\n\n
How to combat this mistake?<\/h3>\n\n\n\n
3. Over-reliance on Convenience Sampling<\/h2>\n\n\n\n
How to combat this mistake?<\/h3>\n\n\n\n
\n
4. Ignoring diversity<\/h2>\n\n\n\n
How to combat this mistake?<\/h3>\n\n\n\n
5. Not Validating Participant Information<\/h2>\n\n\n\n
How to combat this mistake?<\/h3>\n\n\n\n
6. Not Checking for Previous Participation<\/h2>\n\n\n\n
How to combat this mistake?<\/h3>\n\n\n\n
7. Rushing the recruitment process<\/h2>\n\n\n\n
How to combat this mistake?<\/h3>\n\n\n\n
What\u2019s next?<\/h2>\n\n\n\n