Every hiring assessment vendor says they’re “backed by science.” I’ve said it myself, probably in a prior newsletter. But after 20+ years in this field, I’ve come to think that phrase has stopped meaning much of anything, because almost nobody who says it will show you the work behind it.
So this post is me showing the work. Because I believe the best way to build trust in our industry is to be clear and specific about what we actually do, where we’ve been tested, what we’ve learned, and how we’ve continued to improve. Harver has a strong story to tell, and I think being transparent about both the results and the journey behind us makes that story even stronger.
Two teams, checking each other
At Harver, the science behind every assessment spans two disciplines that don’t report up through each other: People Science and Data Science. These two teams work in collaboration and where one leads the other supports and provides additional perspective.
People Science, our I/O psychologists, owns job analysis and the content and construct validity of what we measure. They’re the ones asking: does this test actually cover the skill it claims to? Does it measure what it says it measures, and not something else entirely?
Just as importantly, they help ensure that the specific assessment strategy and hiring process we put in place for each client is scientifically and legally defensible and aligned with what that organization is actually trying to accomplish. Validity isn’t simply a property of a test; it also depends on whether you’re using the right measures, for the right job, for the right purpose.
Data Science focuses on reliability and criterion validity. They test whether our assessment items consistently measure what they’re intended to measure and whether assessment scores actually relate to outcomes that matter, such as early attrition and job performance.
And both sides of that work have to account for fairness. We don’t assume an assessment is fair simply because it was designed to be fair. We test for adverse impact to understand whether selection outcomes differ meaningfully across protected groups. When the data indicate a potential disparity, that’s a signal to investigate what’s driving it, including the assessment, the model, and how it’s being applied.
Two kinds of proof, not one
We run many type of validation studies and I’d like to focus on two that we predominantly use with our clients. The difference between them matters more than most vendors let on.
A concurrent (benchmark) validation study looks at your current employees, their assessment scores and their current performance, and ensures whether the two line up. It’s fast (about two weeks, once we have the data) and it’s a reasonable first read.
A predictive validation study is the harder, better test: we score candidates before they’re hired, then wait, six months, sometimes twelve, and check whether those scores predicted who actually became a top performer. This is the gold standard, because it’s the only study design that mirrors what the assessment is actually being asked to do in production: predict the future, not describe the past.
Validity coefficients can vary, so make sure you understand what you’re looking at when you see them. It’s much easier to get high coefficients when there is a direct relationship between the assessment (e.g., detail orientation) and the criterion (e.g., consistency). For those, you can find a fairly high relationship of .45-.55. When it’s more general, you will find lower relationships because there is more noise and more things that can affect the result. So, if you’re looking at the relationship between a test battery (i.e., multiple assessments) and overall job performance, which we know is multi-faceted, you’re more likely to find a relationship of .15-.25. Our goal is always to understand the direct relationships between assessment construct and the KPI/criteria it should impact and the overall battery and overall performance. This way we can determine if we should improve a part of our measurement or should add to our measurement.
You shouldn’t have to choose between accurate and explainable
For years, the tradeoff in this field was implicit: classical statistical models were interpretable but limited, and machine learning was more powerful but opaque. We didn’t think that tradeoff should exist, so we built around it.
Clients can choose a classical psychometric model or a machine learning model, the choice is theirs, based on their risk tolerance and their need for innovation. But when a client uses an ML model, we don’t ask them to trust a black box. We apply SHAP values, a technique borrowed from game theory that was originally designed to fairly divide credit among players on a team. Applied to a hiring model, it tells you exactly how much each trait (e.g., problem-solving, follow-through, communication) contributed to a specific candidate’s score. It’s the same transparency you’d get from a regression equation, applied to a model far more complex than a regression could ever be.
For machine learning models, we also test performance on holdout data the model has never seen. That helps us determine whether the model can generalize beyond the data used to build it rather than simply fitting patterns in the training data. For classical approaches, we use established psychometric and validation methods appropriate to those models. The goal is the same in either case: evidence that the assessment performs reliably and validly for the purpose for which it will actually be used.
Fairness has to be tested, not assumed
This is the part of the industry conversation I want to address directly, because it’s been badly muddled this year.
Earlier this year, Stanford researchers published a study analyzing four years of historical data from pymetrics, which Harver acquired in 2022. It was positioned as an ‘AI in the Hiring Process’ study, which most people equate to generative AI. To be clear, Harver does not have generative AI in our assessments, and the study was faulty at best.
The study’s methodology has real limitations that matter here. It collapsed every candidate outcome into a single binary, recommend or don’t recommend, when in practice our assessments produce a range of scores that clients configure into tiered bands, role-specific cutoffs, and multiple levels of recommendation. It also drew from a dataset spanning 2018 to 2022 that mixed test data with production data and included models that had already been retired from our platform. It’s not a picture of the system as it exists today. And in the study’s own simulation, every single candidate was recommended for at least 52 different roles. Nobody was categorically shut out, which is the opposite of what “systemic rejection” would look like.
Here’s what holds up independently of any of that: across our full dataset, we test every model against the same statistics regulators use, the four-fifths rule, a two-standard-deviation Z-test, and Fisher’s Exact Test where sample sizes are small, and our tools meet the U.S. EEOC’s fairness standard. An independent audit in 2021 confirmed our approach works as designed, and Harver’s platform passed another independent third-party audit for New York City’s Local Law 144 in 2025 and 2026. We also maintain a voluntary, ongoing bias-audit relationship with BABL AI.
Why any of this should matter to you
Rigor for its own sake is a nice thing to believe in, but it’s not why you’re reading this. Here’s the part that’s actually about your business: clients who’ve gone through this process have seen up to a 45% reduction in early attrition, an 88% reduction in time-to-hire, and a 94% candidate satisfaction rate. That’s not despite the rigor, it’s downstream of it. A model that’s actually valid finds people who actually succeed, and a hiring process people trust is one they finish.
An invitation, not a pitch
If you take one thing from this, don’t just take our word for any of it. Ask any assessment vendor, us included: How do you establish that an assessment is valid for the specific job and population where it will be used? What does your validation process look like at the individual customer level, and when do you conduct predictive validation studies? How do you test for adverse impact, and at what level do you conduct that analysis? Can you explain, in plain language, why your model recommended a specific candidate? And can you show examples where the science translated into measurable improvements in hiring outcomes?
The point isn’t that every assessment should produce the same validity coefficient. It should be that a vendor can show you the evidence, explain the methodology behind it, and demonstrate that its approach works in the context where you’re actually going to use it. If the answer is a shrug or a slide full of claims without the methodology or evidence behind them, that’s worth knowing before you sign a contract, not after.
Want the full technical detail behind our validation and fairness testing? Read how Harver approaches responsible AI in hiring.
