McNemar's Test

Definition

Core Statement

McNemar's Test is a statistical test used on paired nominal data (typically 2x2 contingency tables). It tests if the proportions of "discordant" (changing) pairs are equal. It is essentially a paired version of the Chi-Square test.

Purpose

Before/After Studies: Did the treatment change the Pass/Fail status?
Algorithm Comparison: Do two classifiers disagree on the same test set significantly?
Matched Pairs: Testing differences in discordant pairs in case-control studies.

When to Use

Use McNemar's When...

Outcome is Binary (Yes/No).
measure is Paired/Repeated (Same subject Before & After).
You want to know if the change in one direction (No $\to$ Yes) is more frequent than the other (Yes $\to$ No).

Do Not Use When...

Samples are independent (Use Chi-Square Test of Independence or Fisher's Exact Test).
Outcome is continuous (Use Paired T-Test).

Theoretical Background

The Contingency Table (Paired)

	After: Positive (+)	After: Negative (-)
Before: Positive (+)	$a$ (Yes/Yes)	$b$ (Yes $\to$ No)
Before: Negative (-)	$c$ (No $\to$ Yes)	$d$ (No/No)

Concordant pairs ( $a, d$ ): Didn't change. The test ignores these.
Discordant pairs ( $b, c$ ): Changed status.

The Statistic

χ^{2} = \frac{(b - c)^{2}}{b + c}

Under $H_{0}$ (No change), we expect $b = c$ . The statistic follows a Chi-Square distribution with 1 df.

Worked Example: Effect of Ad Campaign

Problem

You survey 100 people Before and After seeing an ad.
Question: "Will you buy the product?" (Yes/No).

Data:

Yes/Yes ( $a$ ): 20 (Loyal)
Yes $\to$ No ( $b$ ): 5 (Lost customers)
No $\to$ Yes ( $c$ ): 25 (Gained customers)
No/No ( $d$ ): 50 (Never interested)

Analysis:
We only care about the changers: 25 people switched TO the product, 5 switched AWAY. Is this net gain significant?

Calculation:

Hypothesis: $H_{0} : P (Yes \to No) = P (No \to Yes)$ .
Statistic: $χ^{2} = \frac{(5 - 25)^{2}}{5 + 25} = \frac{(- 20)^{2}}{30} = \frac{400}{30} \approx 13.33$
Critical Value: $χ^{2} (1, 0.05) = 3.84$ .
Decision: $13.33 > 3.84$ . Reject $H_{0}$ .
Conclusion: The ad campaign significantly increased purchase intent (Net gain is real).

Assumptions

Paired Data: The two observations must be linked (same person, matched pair).
Nominal Variable: Dichotomous (Binary).
Sufficient Discordance: Ideally $b + c > 25$ . If small, use Exact Binomial Test.

Limitations

Pitfalls

Ignoring Concordance: It can feel weird that 90% of your data ( $a$ and $d$ ) is ignored, but that's correct—if they didn't change, they provide no evidence of an effect.
Confusion with Chi-Square: If you format this as a standard 2x2 and run Chi-Square test of Independence, it answers "Is Before state associated with After state?" (Yes, obviously). It does NOT answer "Did the rate change?".

Python Implementation

from statsmodels.stats.contingency_tables import mcnemar

# Table layout: [[Yes/Yes, Yes/No], [No/Yes, No/No]]
# Note: Layout conventions vary! 
# Standard Statsmodels: [[a, b], [c, d]]
table = [[20, 5],
         [25, 50]]

# McNemar Test
result = mcnemar(table, exact=False, correction=True)

print(f"statistic={result.statistic}, p-value={result.pvalue}")

if result.pvalue < 0.05:
    print("Significant change (Marginal Homogeneity rejected)")

Paired T-Test - Continuous equivalent.
Chi-Square Test of Independence - Unpaired equivalent.
Wilcoxon Signed-Rank Test - Paired ordinal equivalent.