Lack of group-to-individual generalizability is a threat to human subjects research — with the 2019 exchange of letters

PNAS 115(27): E6106–E6115 (2018), DOI 10.1073/pnas.1711978115, PMC6142277. Also read, from the same PNAS issue 116(14), 2019:

• Adolf & Fried, "Ergodicity is sufficient but not necessary for group-to-individual generalizability" (PMC6452692);
• Hamaker & Ryan, "A squared standard error is not a measure of individual differences" (PMC6452686);
• the authors' replies to Adolf & Fried (PMC6452707) and to Hamaker & Ryan (PMC6452651).

None is in the PMC open-access subset, so the E-utilities XML held only the front matter. All five were read from their public PMC article pages (HTML saved, <article> element converted with pandoc). Provenance: papers/carry_on/fisher2018_group_to_individual.provenance.json.

What was read

• The article: all 406 lines of the converted text, including Table 1, the five-sample descriptions, methods, results, discussion and references.
• All four letters in full.
• Not read: Figures 1–4 of the article and the letters' figures (images); the article's SI appendix (temporal dependence tests, Table S1).

Question

Do statistics computed across people (group level) describe the people in the group (individual level)? That holds only for ergodic processes, those homogeneous across people and stable over time.

Method

• Six intensive repeated-measures datasets:
  • an experience-sampling study of anxiety and depression, 78 people, 4 times a day for 30 days;
  • heart rate and respiratory sinus arrhythmia in 69 people;
  • daily affect in 101 people with personality disorders, 100 days;
  • two pre-treatment experience-sampling samples of about 64 people each (10 a day for 6 days);
  • a Dutch crowd-sourced diary study (975 people, 3 a day for 30 days; 535 used).
• Within-person: each person's own mean, SD and correlations over time, also after removing autocorrelation ("AR residuals").
• Between-person: the same data cut into cross-sections, occasion by occasion. Each cross-section gives one group mean and correlation. The SD of those across occasions is the group-level "variance".

Results

• Means agree: 14 of 15 variables are within 7% between levels.
• Mean correlations sometimes differ.

  • Anxiety and depression sample (within against between):

    Pair	Within	Between
    Depressed mood–anhedonia	.47	.75
    Depressed mood–worry	.40	.71
    Fear–avoidance	.23	.46

  • Positive and negative affect agree fairly well: within against between −.33/−.26, −.58/−.50 and .05/.01.

  • Heart rate and RSA agree in raw data (−.35 against −.39), not after removing autocorrelation (−.18).

  • Low-arousal affect: −.46 against −.66.

  • Overall, 4 of 9 comparisons agree in central tendency and 5 do not.

• Individuals differ widely. Within-person correlations have SDs of about .21–.28. Depressed mood–worry ranges from r = −.11 to .89 across people.
• The headline claim. "The variance around the expected value was two to four times larger within individuals than within groups." That is the ratio of the SD of within-person estimates to the SD of the cross-sectional estimates (2.09–4 for correlations; 3.79–13.20 for univariate SDs).

The exchange

• Hamaker & Ryan: the headline comparison is flawed.
  • The SD of within-person correlations across people estimates a random effect: true individual differences plus sampling error.
  • The SD of cross-sectional correlations across occasions is "a proxy of the squared SE" of the group estimate. It shrinks toward zero as the sample grows and measures sample-to-sample variability, not variability between individuals.
  • Comparing the two says nothing about over-estimated group accuracy.
• Fisher et al.'s reply. They argue their cross-sections are the same population resampled over time, so the group SD is "a third representation … within-sample temporal instability in cross-sectional estimates". They concede it "is fundamentally not a representation of within-subject processes" and that the two variabilities should not stand in for each other.
• Adolf & Fried. Ergodicity is sufficient but not necessary. After conditioning on sources of heterogeneity and temporal instability, "conditional equivalence" can license inferences across levels. Randomisation, structural equation and state-space models do this conditioning. Temporal instability (within- against between-day effects) matters as much as heterogeneity between people.
• Fisher et al.'s reply to Adolf & Fried. They agree on the continuum. They hold that full ergodicity is needed to claim a group model fully explains individuals, and that the burden of proof lies on the group model.

Limits

• The variance-ratio claim does not stand. The critique is sound and the authors largely concede it. The valid findings are the gaps between within-person and between-person mean correlations in some constructs, and the wide spread of within-person correlations.
• Mostly clinical or selected samples with short series (≥ 44–83 occasions per person). Within-person correlations from 60–100 occasions carry sampling error, which inflates their spread. The random-effect SD cleaned of that error is not reported (Hamaker & Ryan's point).
• Inconsistencies found: none in the arithmetic checked. The claim "only 68% of all individual correlational values fall within a range that would be predicted by group data to cover 99.7%" depends on the flawed SD comparison, so it inherits that flaw.
• Checked:
  • "Four of the nine comparisons" agree: positive–negative affect in samples 3–5 plus high-arousal affect in sample 6.
  • "44% lower … (r = 0.40)" and "55% lower … (r = 0.32)" match (0.71 − 0.40) / 0.71 and (0.71 − 0.32) / 0.71.

What it means for Kurisutina

• Question 1 in formal terms.
  • "React like Alice, not like the average person with her views" asks whether the population model, conditioned on Alice's state, describes Alice. That is Adolf & Fried's conditional equivalence.
  • The LISS T2 contrast own-full against own-state asks exactly whether anything remains after conditioning on her current state that her past reactions can predict.
• A concrete risk for replicas. A base model carries between-person associations: people who are sad also report less energy; people who worry are also low. Within a person, the same pair can be much weaker or even reversed (depressed mood–worry: .71 between people, .40 within on average, −.11 to .89 across people).
  • A replica that moves its answers together according to population associations will impose population structure on one person's dynamics.
  • Test (proposal): from a participant's diary bursts, compute their within-person item correlations. Check whether the replica's simulated day-to-day answers reproduce them, or the population's.
• Data implication. Within-person structure can only be estimated from many occasions per person. This supports the measurement-burst proposal (Sliwinski) over one-off interviews for learning how a person's states move together.
• Citation hygiene. Do not cite "individuals vary two to four times more than groups". Cite the gap between within-person and between-person correlations and the spread of within-person correlations.

Cross-references

• summaries/carry_on/sliwinski2009_stress_bursts.md: within-person reactivity and bursts.
• summaries/carry_on/eckstein2022_context.md: person parameters bound to context.
• summaries/carry_on/mancini2011_hedonic_treadmill.md, summaries/carry_on/infurna2016_resilience.md: heterogeneity in trajectories.
• docs/research/liss_q1_design.md: T2 own-full against own-state.