Optimizing Schools? Public Perceptions of Algorithmic versus Status Quo Prioritization in K-12 Schooling

Download data and study materials from OSF

Principal investigators:

Rebecca Johnson

Georgetown University

Email: rj545@georgetown.edu

Homepage: https://www.rebeccajohnson.io/

Simone Zhang

New York University

Email: simone.zhang@nyu.edu

Homepage: https://simonezhang.com/


Sample size: 5643

Field period: 06/04/2021-01/24/2022

Abstract
Who counts as an expert in deciding who needs help most urgently? In K-12 schools and other institutions that assess child need, parents position themselves as experts in their own child, potentially clashing with both professional expertise and algorithmic expertise. We study the case of U.S. school districts that have more students in need of high-dosage tutoring in the wake of COVID-19 than help available, fielding a U.S. nationally-representative survey with the NORC AmeriSpeaks panel (N = 5,606 overall; N = 4,368 oversample of K-12 parents; fielded fall 2021). The experiment randomizes respondents to one of four status quo methods that schools could use to ascertain need for tutoring and asks respondents: which is fairer, the status quo method or an algorithm? First, parents are significantly more likely to view parent requests for help as a fair measure of need than nonparents. Second, focusing on current and ever parents, and subsetting to the condition of parents randomized to read the contrast between algorithms and parent requests, we find that more politically conservative, lower educational attainment parents are significantly more supportive of parent requests than more politically liberal, higher educational attainment parents. Overall, the results show cleavages among parents about the fairness of schools making decisions based on algorithmic expertise about need versus parent expertise about need.
Hypotheses

The main hypotheses were as follows, with the full set available in the pre-analysis plan: https://osf.io/vx7de

H1: Across school contexts, respondents will rate algorithms as significantly more fair than each of the status quo methods

H2: Across school contexts, presenting parent requests as the status quo method will lead respondents to rank algorithms as less fair than presenting any of the other three status quo methods.

H3: Across school contexts, minoritized respondents will have differ- ent views about the fairness of algorithms than non-minoritized respondents.

H4: Across school contexts, current and former parents of school-age children will have a smaller gap in favorability between parent requests and algorithms.

Experimental Manipulations
We randomized two components in a 3 x 4 design. First was the ethnoracial context of the district in which the algorithm would be deployed (majority white; majority Black and Hispanic; integrated). Second was the status quo method that algorithmic assessments of need are contrasted to (parent requests; weighted lottery; simple income threshold; counselor judgment). After the initial status ratings, respondents read an update about bias in the algorithm and were asked whether their rating changed. The exact wording is available in the pre-analysis plan.
Outcomes

The main outcome variables were:

1. Binary DV: "Which method for deciding which students get tutors is fairer?"

2. Continuous DV (secondary): When comparing [inserts other method] to the predictive model, how would you rate how certain you are about which is fairer?” Answer choices: 1 = [inserts other method] is definitely more fair; 2 = [inserts other method] is probably more fair; 3 = I’m not sure which is more fair; 4 = The predictive model is probably more fair; 5 = The predictive model is definitely more fair

3. Open-ended response: "“Explain why you think [inserts method they chose as more fair] is fairer than [inserts method they said was less fair]. Please provide a thoughtful response"

4. Change in views after update about algorithmic bias: "With that update in mind, which method should the school district use to select which students get tutors?"

Summary of Results

H1: evidence supported this hypothesis. In the full sample, averaging over school contexts, the algorithm was rated as significantly more fair than each of the other methods via a two-sided chi-square test of differences in proportions: (1) parent requests (test statistic = 44.24, p < 0.0001); (2) simple rule (test statistic = 134.59, p < 0.0001); (3) counselor discretion (test statistic = 37.66, p < 0.0001); and (4) weighted lottery (test statistic = 1086.4, p < 0.0001)

H2: we failed to find evidence supporting this hypothesis. Compared to randomization to parent requests, randomization to the simple rule or weighted lottery caused the algorithm to be rated as MORE fair (p = 0.007, p < 0.0001 respectively). But there was no difference in ratings of algorithmic fairness between randomization to parent requests and counselor discretion.

H3: evidence supported this hypothesis. Compared to non-hispanic White respondents, Black and Hispanic respondents rated algorithms as less fair; when we subset to parents and the contrast between parent requests and algorithms, Asian parents are somewhat more likely to rate the algorithm as fair compared to white, non-hispanic parents (p = 0.05); Black parents rate the algorithm as significantly less fair (p = 0.004), Hispanic as somewhat less fair (p = 0.08).

H4: evidence supported this hypothesis. Subsetting to those randomized to parent requests as the status quo condition, we see that current parents are significantly more likely to rate parent requests as fairer than non-parents (beta = 0.465, p < 0.01) and that ever parents are also significantly more likely to rate parent requests as fairer than non-parents (beta = 0.334, p < 0.05).

References
Rebecca A. Johnson, Simone Zhang, and Katherine Christie. "Who Should Count as an Expert in Student Needs? Parents, Predictive Algorithms, and Polarization over Expertise." Conference presentation. Sociology of Education Association Annual Meeting. February 2022.