Question Difficulty and P-Value: How They Connect
In educational assessment, there are actually two different meanings of “p-value” that you might encounter. Let me explain both and how they relate to question difficulty:
P-Value as Item Difficulty (Most Common in Education):
When analyzing test questions, p-value often means the proportion of students who got the question right:
- P-value = 0.90: 90% of students got it right = EASY question
- P-value = 0.50: 50% of students got it right = MODERATE difficulty
- P-value = 0.10: 10% of students got it right = HARD question
The Relationship:
- Higher p-value = Lower difficulty (more students succeed)
- Lower p-value = Higher difficulty (fewer students succeed)
Classroom Examples:
Easy Question (p-value = 0.95): “What is 2 + 3?” – Almost everyone gets this right
Moderate Question (p-value = 0.60): “Solve: 3x + 7 = 22” – Most students get it, but some struggle
Hard Question (p-value = 0.20): “Explain why the character’s motivation changed in Chapter 5” – Only strong students succeed
What This Means for Your Teaching:
- Questions with p-values 0.80-1.00: May be too easy, not helping you identify learning gaps
- Questions with p-values 0.40-0.70: Just right for discriminating between students
- Questions with p-values 0.00-0.30: May be too hard, poorly written, or covering material not taught
Quick Rule: If you want your test to help identify which students understand the material, aim for questions with p-values around 0.50-0.70. Questions that everyone gets right or everyone gets wrong don’t give you useful information about student learning differences.