Criterion Validity: How and Why To Measure It

Hiring is a guessing game. Almost 1 in 2 hires fail in the first 18 months and, if we’re honest with ourselves, we don’t usually validate that what we’re looking for in new hires is what we should actually be looking for. Some basic KSAs (knowledge, skills, and attributes) are more important than others depending on the job, but part of the challenge is that jobs are constantly changing making our target a moving one. 

Before recruiting, it’s important to determine which KSAs lead to the highest levels of performance and retention in that specific role. Criterion validity can help you do just that.

What is Criterion Validity, and Why Does it Matter?

Criterion validity is a measure of effectiveness. In recruitment, it refers to the correlation between a candidate’s assessment or interview scores and a given business metric. You want there to be a positive (linear) correlation between a candidate’s test scores and their job performance.

For example, let’s say you’re recruiting telemarketers. You’re looking for candidates who can produce a high number of sales per day. In order to find out whether your pre-employment test is an accurate prediction of your candidate’s productivity, you will need to find out if there is a positive correlation between a candidate’s test scores and their average sales per day. In other words, do the employees that score highly on your pre-employment test also produce high sales numbers?

As a recruitment professional, you’ll want to find employees who deliver maximum value, but in order to find them, you need to know what attributes to look for. That’s where criterion validity comes in; measuring criterion validity is a robust way of ensuring your assessment methods are identifying the most competent candidates to perform the role.

How to Measure Criterion Validity

Criterion validity is one of the most time-consuming and hardest-to-measure validity metrics. The process entails collecting large amounts of data and performing detailed statistical analysis. We know this sounds pretty daunting, so here are a few steps that you can follow to help your process run a bit more smoothly:

Step 1: Identify the critical KSAs for the job

Before you start your analysis, you need to determine what KSAs (Knowledge, Skills, and Abilities) your employees need to be successful in their positions. Consider all important aspects of the job and what someone needs to demonstrate to be successful in the role. If you’ve conducted a step-by-step job analysis then you may want to consult it at this point. Use our free job analysis tool to help with this phase.

Step 2: Establish construct validity

Construct validity determines how well your pre-employment test measures the attributes that you think are necessary for the job. In many ways, measuring construct validity is a stepping-stone to establishing the more reliable criterion validity. Measuring the test’s construct validity makes sure that the test truly measures what you want it to measure.

Step 3: Establish face validity

Imagine you’re a candidate applying for a job, and the company asks you to complete a pre-employment test. As you take the test, you find that the questions don’t seem relevant to the job or the job description. Even before being hired, you already have a negative feeling towards the company, and you may even choose to withdraw your application.

Face validity is when an assessment appears to measure what someone is actually doing in a job. The best assessments give applicants a glimpse of your company’s work culture and how it operates. Realistic Job Assessments have especially high face validity, hence why they are used by some of the world’s biggest employers..

This is why establishing face validity is so important. It’s not enough to make sure your test is valid; you have to make sure your test looks valid too. 

Step 4: Give the pre-employment test to your applicant pool

Some assessment providers will advise you to have incumbent employees take the assessment in order to set thresholds or performance benchmarks. This practice is wrong. Your existing employees will have an unfair advantage against job applicants in that they already do the job they’re being assessed for. The only way to reliably validate criterion validity is to review the performance of an assessment after it is live. The more candidates that take the test and subsequently get hired (i.e. the bigger your sample size), the more accurate your data will be. 

Step 5: Examine the statistical relationships in your data

Using software such as Microsoft Excel, SPSS, or Minitab, correlate the test scores with different business statistics. In statistics, correlation is measured on a scale of -1 to 1, -1 indicating a perfect negative correlation and 1 indicating a perfect positive correlation. A correlation coefficient of 0.20 or greater (or -0.20 or smaller) with a 95% confidence interval (0.05 p-value) is considered statistically significant. 

Ok, we know that was a bit confusing, so let’s try and simplify things with an example. Let’s go back to our telemarketing recruiter. She’s tasked with developing an assessment that helps identify top sales performers.

She creates an assessment and asks candidates to complete it. Candidates complete the assessment and people get hired. Some perform well, some don’t. After a period of time she plugs in both the assessment data and the sales performance data into Excel and sees that the correlation coefficient between her applicant test scores and the volume of sales is 0.95. Because 0.95 is very close to 1, this tells us that there is a very strong positive correlation between the two variables, meaning that if you score well on the test, you are also very likely to produce high sales numbers. Similarly, this means that if you score badly on the test, you are also very likely to produce low sales numbers.

As your correlation coefficient gets closer to zero, the relationship between test scores and sales performance becomes more random: some high-performing employees may score badly on the test, and some under-performing employees may score very well. This shows you that the test is either A) not an accurate measure of a candidate’s KSAs, e.g. poor construct validity, or B) measuring the wrong KSAs, e.g. poor criterion validity.

Summary

Each job is unique and therefore each company should design its own, role-specific assessments. In doing so, it’s important to measure the criterion validity of those assessments over time and adjust them accordingly. measuring criterion validity is not as complex as it seems but if you need help then you’re assessment provider should be able to support you.

Using accurate pre-employment tests is a great first step in creating an effective and efficient recruitment process. 

If you liked reading this post you may also like reading the following:

• Construct Validity: how to measure it and why it matters
• What is a Realistic Job Assessment and how does it work?
• Pre-Employment Test Validity vs Test Reliability
• Situational Judgement Test: How to Create Your Own
• Job analysis: The ultimate guide to job analysis

Want help finding an efficient recruitment plan for your business? Request a demo and learn more about how ThriveMap can improve your company’s recruitment strategy!