Karl Pearson’s Correlation

Karl Pearson’s Correlation — The Compass of Co-Movement

Imagine you’re responsible for reducing delays across an airline’s network. You ramp up crew experience through targeted rostering and mentoring and ask: “Are we actually bending the delay curve?” Or in a classroom, a teacher tries a simple intervention—better sleep hygiene—and wonders: “Are sleep hours and memory performance moving together?”

That instinct—checking if two variables move together—is the core of correlation. In the late 19th century, Francis Galton noticed that very tall parents tend to have tall-but-closer-to-average children. Karl Pearson then forged a precise, general tool to measure this co-movement for any pair of numeric variables: the Product–Moment Correlation Coefficient, commonly denoted as r.

Why Pearson matters

• Fast signal check: Before committing to heavy analytics, r is a crisp early indicator.
• Scale-free: r is unaffected by units (minutes, kilograms, marks…)
• Direction & strength: Sign tells direction (↑↑ or ↑↓), magnitude tells tightness (weak ↔ strong).

What Pearson actually answers

• Sign: Do X and Y move in the same direction (+r) or opposite (–r)?
• Strength: How tightly do their points cling to a straight line (|r| close to 1)?
• Linearity: Pearson “sees” linear patterns best (it can miss curves).

---

✈ Aviation Example — Crew Experience vs Delay Minutes

Hypothesis: more experienced crews tend to reduce departure delay minutes per flight (procedural fluency, coordination with ground, fewer last-minute errors).

Observation (illustrative dataset): we collect paired observations for crew experience (years) and average delay minutes.

• If r is strongly negative (e.g., r ≈ –0.95), rising experience aligns with falling delays.
• If r is near zero, experience may not be the driver—or other variables dominate (weather, ATC slots, gate conflicts).
• If r is positive, you may have a confound (e.g., experienced crews are assigned to complex, delay-prone routes).

🧠 Psychology/Education Example — Sleep vs Memory Performance

Hypothesis: more sleep (within a healthy range) tends to improve memory retention and test performance.

• If r is positive and strong, longer sleep aligns with higher scores.
• But beware of curvilinear effects: too little or too much sleep can both harm memory (an inverted-U). Pearson might miss that if the data spans extremes.

---

Reading r correctly

• +0.70 to +0.90: Strong positive alignment (useful signal).
• –0.70 to –0.90: Strong negative alignment (useful lever).
• ~0.0: No linear trend (either unrelated or non-linear/hiding confounds).

Where Pearson shines

• When the relationship is approximately linear.
• When the data are roughly continuous and free of extreme outliers.
• As a pre-regression triage—a quick scan before modeling.

And where it can mislead

• Non-linear realities: Sleep vs memory may be inverted-U; Pearson may show r ≈ 0 across the full range, hiding a strong middle-range effect.
• Outliers dominate: A single irregular day (mass diversion) can distort airline correlations.
• Omitted variables: Experience may correlate with delays because experienced crews get assigned to stormy airports (confounding by weather or route difficulty).
• Correlation ≠ Causation: Strong r is an invitation to investigate—not a verdict.

---

Practical insights & actions

• Scatter first: Always plot X vs Y. If the cloud bends, consider transformations or non-linear methods.
• Slice wisely: Check r by route class, season, aircraft type, or student cohort. Averages can lie.
• Control confounds: Move from correlation to regression (add weather, load, time-of-day) or to matching/causal designs.
• Report uncertainty: With small n, r can wobble—add confidence intervals or hypothesis tests for r.

Bottom line: Pearson’s r is your fast, scale-free compass for linear co-movement. It doesn’t drive the aircraft or grade the paper, but it tells you which direction to look, and how strongly.

From Covariance to Pearson’s r — Formula, Steps, and Worked Examples

1) Intuition → Covariance → Correlation

Covariance measures whether deviations from means co-rise or co-fall. Pearson scales it by the spread of X and Y, producing a neat –1 to +1 number.

r = Cov(X, Y) / (σX · σY)

                    Σ (xᵢ − x̄)(yᵢ − ȳ)
      r  =   -----------------------------------
              √[ Σ (xᵢ − x̄)² · Σ (yᵢ − ȳ)² ]
        

              nΣxy − (Σx)(Σy)
      r =  ------------------------------
          √{ [nΣx² − (Σx)²] [nΣy² − (Σy)²] }
        

Experience (yrs): [1, 2, 3, 4, 5, 7, 9, 12]
      Delay (minutes):  [22,20,19,17,16,14,12,10]

Experience X: [1, 3, 5, 7, 12]   →  x̄ = 5.6
      Delay Y:      [22,19,16,14,10]    →  ȳ = 16.2

      Compute deviations and sums:
      Σ(x−x̄)(y−ȳ) = −76.6
      Σ(x−x̄)²     = 71.2
      Σ(y−ȳ)²     = 84.8

      r = −76.6 / √(71.2 × 84.8) ≈ −0.986
        

Sleep (hrs):  [4.5, 5.0, 6.0, 6.5, 7.0, 7.5, 8.0, 8.5]
      Memory score: [50, 60, 64, 68, 72, 74, 78, 77]