What happened? After all, Meta is known as a prestigious employer, having both a large research staff and a highly competitive hiring process. The internal research that Zuckerberg references has never been publicly shared, but it\u2019s fair to assume it was done to a high standard of quality. Yet repeated findings in one context \u2014 internal research \u2014 failed to replicate in another, and this had huge consequences for the company.<\/p>\n\n\n\n
As UX Researchers, our goal is to inform decisions with quality research. But even with the best of intentions, research sometimes fails. Depending on the severity of the failure, and how integral those findings were to company strategy, the consequences can be grave. <\/p>\n\n\n\n
There are three major ways a UX research study can fail: the study wasn\u2019t sound, the findings were wrong, or the findings don\u2019t apply as widely as hoped. In this article, we\u2019ll look closely at common mistakes under each of these categories and explore ways to address them.<\/p>\n\n\n\n
The study wasn\u2019t sound<\/strong><\/h3>\n\n\n\nIn 2008, Walmart ran a customer survey. There was one clear takeaway: customers wanted less clutter in the stores.<\/p>\n\n\n\n
The company went right to work, simplifying store layouts and presenting streamlined options. They cleared and slimmed the aisles, reduced displays, and minimized selection. Having spent millions to refurbish stores, customer satisfaction soared \u2014 but there was a swift and dramatic loss in sales. By some estimates, it ended up costing Walmart almost $2 billion in potential revenue.<\/p>\n\n\n\n
One analyst suspected that Walmart leadership was fishing for data to support a hypothesis they had<\/a>: to compete with Target, with its sleek and sophisticated stores, Walmart needed to follow suit. And if a researcher is motivated to do so, it\u2019s not hard to design a study that will guarantee a predetermined result.<\/p>\n\n\n\nPlacing one\u2019s thumb on the scale is one example of this first category of error: conducting a study that has fundamental flaws. This can manifest in other ways. You may have unclear objectives. You could have too few participants, or you may have recruited a biased sample. We can also introduce measurement error by asking flawed, inappropriate, or leading questions.<\/p>\n\n\n\n
In a technical sense, we would describe all of these as problems related to\u00a0internal validity<\/a>. They compromise the integrity of the research.<\/p>\n\n\n\nOf the three major failures, this sin is the deadliest. It doesn\u2019t matter how good your analysis or report is if the study is flawed from the beginning. And though unforeseen issues can later compromise the later stages of a project, the researcher has the most control at this point \u2014 and therefore, the most responsibility.<\/p>\n\n\n\n
Researchers need to develop both an intuitive feel as well as a clear process for nipping these mistakes in the bud. Formal instruction on experimental design and threats to internal validity can help. <\/p>\n\n\n\n
There was a problem with the findings<\/strong><\/h3>\n\n\n\nDuring the 2008 and 2012 US presidential elections, Nate Silver accurately forecasted not just Obama\u2019s victory, but how many individual districts and states would vote. Historically, pollsters would look at one or a few recent polls. Silver\u2019s probabilistic approach involved including as many as possible and weighting each by past accuracy and poll quality.<\/p>\n\n\n\n
In 2016, however, Silver\u2019s publication\u00a0538<\/a>\u00a0gave Hillary Clinton a 71% chance of winning the presidency.\u00a0They summarized their forecast<\/a>\u00a0saying, \u201cClinton is probably going to win, and she could win by a big margin.\u201d When Trump won, Silver pointed out that events that are only 30% likely to happen will still happen 30% of the time. Unfortunately, many of 538\u2019s readers misinterpreted the forecast, and Silver\u2019s credibility suffered.<\/p>\n\n\n\nCommunicating the bounds of one\u2019s certainty is difficult \u2014 and the time to get it right is before the forecasted event. <\/p>\n\n\n\n
Our work can also suffer from\u00a0issues related to probability and uncertainty<\/a>. And there are many ways to draw or convey an incorrect conclusion from an otherwise solid study. We might think we\u2019ve found something that was never really there (a false positive<\/a>), or we might fail to find something that we should have (a miss<\/a>). And when we describe a real finding, we might think it\u2019s\u00a0more or less important<\/a>\u00a0than it actually is.<\/p>\n\n\n\nWhen our takeaways don\u2019t reflect the underlying reality, it leads stakeholders into bad decisions. Researchers must be vigilant in their analyses, challenging their findings and considering alternative explanations. And we should be careful to explain how certain or uncertain a finding is.<\/p>\n\n\n\n
The study and findings were sound, but only in its context<\/strong><\/h3>\n\n\n\nIn the early 1980s, Coca-Cola was steadily losing market share to its sweeter competitor, Pepsi. To try and boost sales, they decided to test a refreshed recipe for their flagship beverage. The results from extensive market research were conclusive: consumers had an overwhelming preference for the new, sweeter Coke. It seemed so obvious that the company should move forward with the refresh that some distributors even threatened to protest if they didn\u2019t.<\/p>\n\n\n
\n![\"80's](\"https:\/\/substackcdn.com\/image\/fetch\/w_1456,c_limit,f_auto,q_auto:good,fl_progressive:steep\/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F1a1f4392-ada8-43e7-bdaf-38ab7f6df1ae_640x867.png\" "\\"80's")
<\/a>An advertisement for New Coke<\/em><\/figcaption><\/figure><\/div>\n\n\nUpon release, the market reacted strongly to\u00a0New Coke<\/a>\u00a0\u2014 but not in the way the company had anticipated. The outrage and boycotts were so intense that Coca-Cola quickly back-pedaled, retracting the product and putting the original back on shelves.<\/p>\n\n\n\nMany have since speculated on how Coke\u2019s research could have misled them so badly. Some point to the nature of the taste tests, where participants got what amounted to a single sip, rather than the full serving of a can or bottle. Others point to customer loyalty. New Coke was released as a wholesale replacement of the original recipe rather than another option, when customers had built up nearly a century of brand associations.<\/p>\n\n\n\n
In either case, what was true in the laboratory didn\u2019t remain true in the real world. <\/p>\n\n\n\n
UX research studies and their findings can be sound in themselves, but if they don\u2019t carry over into the context we\u2019re interested in \u2014 that of our actual users \u2014 our work has little value. Just about any element of our study could differ in a meaningful way from real life: its timing, the way we framed our questions and tasks, our participants\u2019 goals and motivations, the presence of a researcher-observer, or the experience itself.<\/p>\n\n\n\n
When findings fail to generalize, the problem is rooted in the study\u2019s\u00a0external validity<\/a>. Researchers must carefully consider the context of their studies and replicate the conditions they are interested in as much as possible. And when we\u2019re reporting our findings, we must consider the possible contexts in which they may not remain true.<\/p>\n\n\n\nConclusion and summary<\/strong><\/h3>\n\n\n\nThere are many ways a UX research study can go wrong, but the root cause of the failure rolls up into one of three major categories:<\/p>\n\n\n\n
Placing one\u2019s thumb on the scale is one example of this first category of error: conducting a study that has fundamental flaws. This can manifest in other ways. You may have unclear objectives. You could have too few participants, or you may have recruited a biased sample. We can also introduce measurement error by asking flawed, inappropriate, or leading questions.<\/p>\n\n\n\n
In a technical sense, we would describe all of these as problems related to\u00a0internal validity<\/a>. They compromise the integrity of the research.<\/p>\n\n\n\n Of the three major failures, this sin is the deadliest. It doesn\u2019t matter how good your analysis or report is if the study is flawed from the beginning. And though unforeseen issues can later compromise the later stages of a project, the researcher has the most control at this point \u2014 and therefore, the most responsibility.<\/p>\n\n\n\n Researchers need to develop both an intuitive feel as well as a clear process for nipping these mistakes in the bud. Formal instruction on experimental design and threats to internal validity can help. <\/p>\n\n\n\n During the 2008 and 2012 US presidential elections, Nate Silver accurately forecasted not just Obama\u2019s victory, but how many individual districts and states would vote. Historically, pollsters would look at one or a few recent polls. Silver\u2019s probabilistic approach involved including as many as possible and weighting each by past accuracy and poll quality.<\/p>\n\n\n\n In 2016, however, Silver\u2019s publication\u00a0538<\/a>\u00a0gave Hillary Clinton a 71% chance of winning the presidency.\u00a0They summarized their forecast<\/a>\u00a0saying, \u201cClinton is probably going to win, and she could win by a big margin.\u201d When Trump won, Silver pointed out that events that are only 30% likely to happen will still happen 30% of the time. Unfortunately, many of 538\u2019s readers misinterpreted the forecast, and Silver\u2019s credibility suffered.<\/p>\n\n\n\n Communicating the bounds of one\u2019s certainty is difficult \u2014 and the time to get it right is before the forecasted event. <\/p>\n\n\n\n Our work can also suffer from\u00a0issues related to probability and uncertainty<\/a>. And there are many ways to draw or convey an incorrect conclusion from an otherwise solid study. We might think we\u2019ve found something that was never really there (a false positive<\/a>), or we might fail to find something that we should have (a miss<\/a>). And when we describe a real finding, we might think it\u2019s\u00a0more or less important<\/a>\u00a0than it actually is.<\/p>\n\n\n\n When our takeaways don\u2019t reflect the underlying reality, it leads stakeholders into bad decisions. Researchers must be vigilant in their analyses, challenging their findings and considering alternative explanations. And we should be careful to explain how certain or uncertain a finding is.<\/p>\n\n\n\n In the early 1980s, Coca-Cola was steadily losing market share to its sweeter competitor, Pepsi. To try and boost sales, they decided to test a refreshed recipe for their flagship beverage. The results from extensive market research were conclusive: consumers had an overwhelming preference for the new, sweeter Coke. It seemed so obvious that the company should move forward with the refresh that some distributors even threatened to protest if they didn\u2019t.<\/p>\n\n\n Upon release, the market reacted strongly to\u00a0New Coke<\/a>\u00a0\u2014 but not in the way the company had anticipated. The outrage and boycotts were so intense that Coca-Cola quickly back-pedaled, retracting the product and putting the original back on shelves.<\/p>\n\n\n\n Many have since speculated on how Coke\u2019s research could have misled them so badly. Some point to the nature of the taste tests, where participants got what amounted to a single sip, rather than the full serving of a can or bottle. Others point to customer loyalty. New Coke was released as a wholesale replacement of the original recipe rather than another option, when customers had built up nearly a century of brand associations.<\/p>\n\n\n\n In either case, what was true in the laboratory didn\u2019t remain true in the real world. <\/p>\n\n\n\n UX research studies and their findings can be sound in themselves, but if they don\u2019t carry over into the context we\u2019re interested in \u2014 that of our actual users \u2014 our work has little value. Just about any element of our study could differ in a meaningful way from real life: its timing, the way we framed our questions and tasks, our participants\u2019 goals and motivations, the presence of a researcher-observer, or the experience itself.<\/p>\n\n\n\n When findings fail to generalize, the problem is rooted in the study\u2019s\u00a0external validity<\/a>. Researchers must carefully consider the context of their studies and replicate the conditions they are interested in as much as possible. And when we\u2019re reporting our findings, we must consider the possible contexts in which they may not remain true.<\/p>\n\n\n\n There are many ways a UX research study can go wrong, but the root cause of the failure rolls up into one of three major categories:<\/p>\n\n\n\nThere was a problem with the findings<\/strong><\/h3>\n\n\n\n
The study and findings were sound, but only in its context<\/strong><\/h3>\n\n\n\n
<\/a>Conclusion and summary<\/strong><\/h3>\n\n\n\n