Advanced Quant Techniques with Small Samples: The Right Way
Ben Cortese leads KS&R’s Decision Sciences & Innovation team, where he focuses on advanced quantitative methodologies, synthetic data evaluation, AI governance, and model validation. With more than a decade of experience applying statistical and analytical techniques to complex research challenges, Ben helps organizations assess emerging methods with rigor, transparency, and confidence.
Ben is leading KS&R work on the ECHO Index, a framework designed to evaluate the quality, reliability, and trustworthiness of synthetic data by benchmarking it against human response. He holds a PhD in statistics from Syracuse University and brings a practical, evidence-based perspective to how synthetic data can be tested, understood, and responsibly applied in research.
Allison is an experienced director with a penchant for unraveling complex datasets and distilling them into actionable insights. With a background in Cognitive Neuroscience and Statistics, she's KS&R’s go-to quantitative expert, especially within the Healthcare Industry Team. Allie's proficiency in machine learning and diverse statistical techniques makes her an asset across all industry verticals. She excels in methodologies such as segmentation, conjoint analysis, and advanced modeling. Outside of work, Allie enjoys outdoor pursuits with her husband and two English bulldogs. Exploring nature and discovering dog-friendly city spots are her passions. Allie's blend of analytical expertise and genuine enthusiasm enriches both our team and the solutions we deliver to clients.
In the market research industry, there is a lack of consensus when it comes to sample size guidelines and guardrails, especially when conducting advanced quantitative research (think anything beyond stat testing). A common question arises: “How small is too small?” to achieve a statistically significant or worthwhile result. To tackle this challenge, we set out on a journey to define clear sample size recommendations for traditional online research and explore the potential benefits of mixed mode research – collecting data for advanced quant techniques through qualitative interviews.
The Micro Modeling Approach
This initiative led to the development of “Micro Modeling”, a framework for executing advanced quantitative research with small samples. Micro Modeling provides guidance by addressing several key questions:
- What sample size is necessary to have confidence in MaxDiff and Conjoint results?
- Do respondents provide higher quality quantitative data when collected via an In Depth Interview (IDI)?
- Can we produce valid results through mixed mode research with smaller samples than traditional online research?
- Are there limitations or guardrails to keep in mind when executing this type of research?
Healthcare Case Study
Efforts were focused within the healthcare space, as HCP data is difficult and costly to obtain, especially when conducting research in rare disease states. There are broad applications beyond healthcare in tech, B2B, and any other industry where the population is limited or sample is scarce. Our rigorous, two-phased approach consisted of a large-scale quantitative study using an online survey, alongside a qualitative phase, with over 500 total cardiologists interviewed. Data collection was facilitated by our partners at EMI, ensuring robust and diversified inputs.
Results and Recommendations
Results from the quantitative phase were clear. Recommendations for MaxDiff fall on a spectrum where smaller base sizes (even smaller than we expected ) do indeed provide valuable insights, even if somewhat limited. As expected, larger base sizes drive more confidence. A straightforward summary of results demonstrates some key sample size thresholds.
Unfortunately, we had limited success with conjoint and were unable to validate results with smaller samples than historic benchmarks. Therefore, we suggest maintaining the status quo when conducting conjoint research through online surveys.
The second phase focused on further reducing recommended thresholds when data is collected through qualitative interviews. Similar to the quantitative phase, there is evidence of an opportunity to leverage small samples, as we observed reliable outcomes for MaxDiff with small base sizes using a mixed mode approach. The recommended thresholds achieved through Micro Modeling are in fact smaller, a strong indicator that conducting a tradeoff activity via an IDI improves the quality of data.
The similarities between the quant and qual phase extend to conjoint as well. We were unable to validate a viable small sample solution, even with data collected via an IDI.
Additional Benefits
Advanced quantitative analysis with small samples is commonplace, and with Micro Modeling, we have established a framework and guidelines, validated via rigorous statistical analysis, to execute MaxDiff. Added confidence in this methodology with small samples was our key objective, but there are several additional benefits extending the utility of Micro Modeling.
- Quant Insights from Small Samples: Qualitative studies with as few as 15 IDIs can yield at least some level of viable quantitative insights.
- Extending Qualitative Research: Micro Modeling bolsters qualitative insights through a quantitative lens which is especially useful for message testing and feature prioritization.
- Enhancing the “Why”: Through Micro Modeling, participants explain their thinking, providing an in depth understanding of the decisions made in a prioritization activity.
Micro Modeling’s 5 Big Things
Micro Modeling was successful at defining a framework for executing a MaxDiff activity with small samples. When considering Micro Modeling, keep these five big things in mind.
- Validated. Our results indicate that Micro Modeling via MaxDiff is viable.
- Better Quality Data with Qual. Using data collected through IDIs, MaxDiff models performed better than those run with data collected through online surveys.
- Confidence is Tied to Base Size. We suggest five distinct thresholds balancing precision and sample size for MaxDiff studies.
- Big Quant Studies Still Have a Place. Large quantitative studies provide a level of accuracy and precision that is not supported by Micro Modeling.
- Complex Models Need Bigger Base Sizes. Analysis of conjoint models requires larger base sizes than Micro Modeling, despite higher quality data.
