Measuring stability and change in personal culture using panel data

Published in American Sociological Review 85(3), DOI 10.1177/0003122420921538; the published version was not read. Read as the manuscript posted on SocArXiv (8za35, v1, 25 March 2020, CC BY 4.0). OSF marks it as published, with the ASR DOI. Duke University. The code is at github.com/krkiley/panel_change (not read). Provenance: papers/carry_on/kiley2020_personal_culture.provenance.json.

What was read

• All 2,332 lines of pdftotext -layout output (61 pages): abstract, theory, model, results, discussion, Appendices A–B (text), footnotes 1–3, references and the captions of Figures 1–13.
• The figures are images. Figures 6–9 (pages 50–53) were rendered and inspected, because they hold the per-item estimates. Figures 1–5 and 10–13 were not inspected.

Question

When an adult gives a different answer at the next wave, does the change persist or revert?

• The active updating model: $y_{it} = y_{i,t-1} + \nu_{it}$, a Markov process in which the last answer is the new baseline.
• The settled dispositions model: $y_{it} = U_i + \nu_{it}$, a fixed baseline with deviations that bounce back.

Method

• Data. GSS three-wave panels 2006–10, 2008–12 and 2010–14: the same panels as the pilot. The sample is people who answered an item in all three waves, weighted for design and non-response.
• Items. 183 attitudes, beliefs, self-assessments and behaviours, including composite scales, grouped into 15 topics following Hout & Hastings (2016). Demographics, work and objective SES are excluded.
• Model per item: $E(y_{i3}) = \alpha + \phi\beta y_{i2} + (1-\phi)\beta y_{i1}$.
  • $\phi$ = .5 means waves 1 and 2 predict wave 3 equally well, i.e. settled dispositions. The best forecast is then the mean of the past answers.
  • $\phi$ → 1 means only the latest answer matters, i.e. updating.
  • $\beta$ is "consistency", i.e. how predictable wave 3 is at all.
  • Free $\phi$ is compared with $\phi$ = .5 by BIC.
  • Age differences use a split $\phi$ at every cutoff age from 20 to 45.
  • Appendix B repeats the estimates with 2–3-point versions of the scales.
• Known biases, as the authors state them.
  • Measurement error pulls $\phi$ toward .5.
  • A few people making large lasting changes can inflate $\phi$.
  • Population-wide shifts go into the intercept, and changes in spread go into $\beta$.

Results

• Persistent change is rare.
  • 75 of 183 items (about 40%) prefer $\phi$ = .5, i.e. no evidence of updating. They include most abortion, civil liberties, institutional confidence, race and gender items.
  • Most free estimates are .5–.6.
  • Worked example (abortion after rape, $\phi$ = .62, above the 75th percentile):
    • 257 of 2,259 people changed between waves 1 and 2;
    • under the settled model about 129 would keep the new answer at wave 3; 147 did;
    • so about 18 people (< 1%) show persistent change.
  • For most items, persistent change reaches under 1% of the sample in two years, and it reaches 5% for no item. The authors: "knowing what a person said two years ago provides almost no better prediction of their current views than knowing what they said four years ago".
• Age. Of the 108 items with some updating:
  • 22 differ by age at most cutoffs, and mostly the young update more. For trust, political views, letting terminal patients die, and affirmative action for women, updating disappears above 30.
  • 8 items show more updating in older people: religious activity, suicide after bankruptcy, aging parents living with children.
  • 86 show no consistent age pattern.
• What changes persistently.
  • Items with external anchors: party identification ($\phi$ about .67, $\beta$ about .89, the most consistent political item), church attendance (about .66), meeting friends at a bar (about .72) and gun ownership (about .72).
  • Items whose referent changes, e.g. confidence in the executive branch, the highest at about .82. In the panel that spans the change of president (2008–12) it reaches $\phi$ = .95, $\beta$ = .51. In the 2010–14 panel it is .57/.65.
  • Items about personal finances in the recession: finances better or worse (about .71), finding a good job (about .71), satisfied with finances (about .64).
  • Gay rights: all 6 items show updating, which the authors link to the shift in elite opinion.
• Ideology is settled. Political views: $\phi$ about .56, $\beta$ about .74, with no updating above 30. With coarsened scales, changes around the midpoint look like measurement error.
  • Party moves toward ideology, not the reverse ("partisans without constraint").
• Race-attitude items are so inconsistent that they are "difficult to call … either settled or updating". Aggregating items into scales raises $\beta$ but not $\phi$.

Values read off Figures 8–9 for the pilot's targets (approximate, to about ±.01):

Limits

• Only three waves over four years, and adults only. Measurement error cannot be separated from real reverting change: the correlation of $\phi$ with Hout–Hastings reliability is .165.

• The model finds whether any persistent change exists, not how many people change.

• The estimates are $\phi < .5$ for a few items, but the text never interprets these:

  • "Police can hit citizens" is .43 (Figure 8 note);
  • in Figure 6, sex education and gun permits are just under .5.

  $\phi < .5$ would mean the older answer predicts better than the newer one.

• Inconsistencies found:

  • The text names confidence in "banks and the financial system" among the three items with the most persistent change. Figure 9 plots "Banks & finance" at the $\phi$ = .5 constraint, i.e. no evidence of updating.
  • The discussion says "roughly 70 items" show no persistent change; the results say 75.

• Checked:

  • 257/2 ≈ 129 and 147 − 129 = 18 (0.8% of 2,259).
  • 183 − 75 = 108 and 108 − 22 = 86.

What it means for Kurisutina

• Carrying on mostly means not changing.
  • Over two years, most adult attitudes change persistently in under 1% of people. Most of the change seen between waves reverts, whether as noise or as a temporary deviation.
  • A faithful replica should keep a stable baseline, let temporary states move its answers, and let them decay. Lasting updates should be rare and concentrated where the person's life changed something outside their head: job, finances, party, church, a partner.
• A test for drift (proposal). Run the replica through three simulated waves on the same items and estimate $\phi$ and $\beta$ per item. Compare with the human values here.
  • A replica whose context accumulates its own answers is structurally Markov. It would show $\phi$ near 1 on settled items (ideology, abortion, institutional confidence), where people show .5.
  • That is drift from the memory mechanism, not from the person. It is measurable without an LLM judge.
• For the GSS pilot (inferred).
  • The pilot predicts a later wave from one earlier wave. On settled items the best human-level forecast is the mean of all past answers.
  • For wave 2→3 pairs, showing the replica both earlier waves should help on items with $\phi$ near .5, and matters little where $\phi$ is high (finalter, partyid). That is a cheap candidate change for a later run; the frozen pilot is not touched.
  • The primary items for job events (satfin, finalter) are among the most "updating" items, consistent with the recession. That is where an event can plausibly carry lasting information.
  • Two items should be read as mostly noise around a baseline, which bounds what an event can add:
    • happy ($\phi$ .61, reliability .60 per Hout);
    • confinan (no updating in the figure).
• Person-level and age-level calibration. Updating is concentrated in the young for political views and trust, and in the old for some family and religion items. A replica's rate of change should depend on the item and the person's age, not be a global setting. This agrees with Beck & Jackson and Hout.
• Changing referents. An item like "confidence in the executive branch" changes meaning when the president changes. A replica told the year will answer about a different object, so apparent change there is not drift.

Cross-references

• summaries/carry_on/hout2016_gss_reliability.md: reliability and stability of the same panels and items.
• summaries/carry_on/kim2024_ai_augmented_surveys.md: cross-sectional prediction of GSS answers.
• summaries/carry_on/beck2018_idiographic_networks.md: consistency as a person-level trait.
• docs/research/gss_pilot_design.md: target items, pairs and conditions.