Inclusive Recruitment
Reliability & Validity
How reliable and valid are your measurements?
Reliability
To put it simply, your measurement is reliable if you get consistency when you measure the same thing again.
However, consistency on it's own, does not mean that it is automatically valid. For instance, we may be getting the same answer, but is our test accurate?
It's clear that the first picture is not reliable since the arrows are hitting a different place every time.
But in the next picture, where the arrows are hitting the same spot, many people assume this is valid.
However, while your test is consistent (you keep getting the same answer if you test again), it is not valid.
You were aiming to hit the bull's eye every time!
Therefore you are measuring something else consistently and you need to work out what that something is!
Validity
To put it simply, your measurement is valid if you can accurately (and consistently) measure what you intended to measure.
Many people only consider the most common types of validity:
FACE - does the person taking the assessment think it's a relevant measure?
CONTENT - is the content of the assessment applicable and comprehensive enough? Ask yourself, what else needs to be measured?
METHODOLOGICAL - are the methods used appropriate e.g. if I wish to assess someone's coaching skills, observing them coaching is much more likely to be accurate than getting them to answer questions about coaching.
There are a large number of different types of validity. I think my last count was 12.
Although, arguably a couple are a subset of another, so let's say 10-12.
Most people however, do not think about all of these, believing them not to be relevant to their specific situation.
Personally, I think they are all relevant because the question each validity asks should still be asked and answered, even if you're not going to be doing any stats to prove it.
Having a discussion about the question that each validity specifically asks is very important. This ensures that you've taken it into consideration as part of your design.