This is an area where hacker news shows its weakness. We have:

1. A chart showing a very low increase (1-2 percent)

2. Nothing to control scores rising in every school in America in the last school year (due to reduction of COVID effects).

3. Scores not moving immediately after the ban, but only after the start of a new school year, which means a new cohort of students muddying the data.

Yet the data fits people's biases here (regardless whether it's right or wrong), so the celebrate it and add anecdotes and explanations why it's true.

There's no study that's good enough for HN.

I don't think I've ever seen a science or research article posted here that didn't immediately get picked apart for this or that in the comment section. The methodology is flawed. The data is flawed. The conclusions cannot be drawn. There are confounding variables not accounted for. The sources are questionable. It's become a trope at this point. Either our commenters' standards are way too high, or all of science reporting is deeply flawed.

Ok here is the crucial part of the paper:

It's a difference in differences design, using individual-level test scores and de-seasonalized data (p. 13). Their wording is:

> Y_igst is the outcome of interest for student i in grade g in school s in time period t, HighAct_s is an indicator for high pre-ban smartphone activity schools, D_t is a series of time period dummies (t = 0 indicates the first period after the ban took effect), δ_s is school fixed effects, and θ_g is grade fixed effects. In this setting, β_t are the parameters of interest, reflecting the difference in the outcome of interest between treatment and comparison schools for each period, with the period before the ban serving as the omitted category, holding grade level constant.

To me some modeling choices seem a bit heavy-handed, but I'm not an economist and could not do better.

To be fair, all those details are in the paper. And a 1-2 percent increase does not seem low to me for such a measure.

The lack of control and cohort following are legitimate criticisms. The effect size is not. Even a single-digit percentage increase over a single year from policy treatment is incredibly impressive when we open the door to cumulative effects.

> Yet the data fits people's biases

It does. But it also fits priors, particularly those we've seen documented when it comes to teens and social media.

There is another major factor that could cause scores to rise globally: ChatGPT. It is now good enough that it can explain topics to students at home, like a private tutor.

I'd go further and say its a global weakness and unbelievably destructive. The bulk of current discourse today is:

1. Read a headline/tweet/instagram.

2. Decide whether or not it fits in your worldview.

3. Move forward with the confidence that you are better informed than everyone else who agrees/disagrees with it.

You see it everywhere on all sides of all beliefs.

It didn't use to be like this. We used to read articles, we used to read common news sources, we use to not have media overrun with bad actors who know exactly what to say to get the most engagement and solidify people in their own world views.

It's all over HN and I could have hoped there'd be more willingness to say "let me consider the contents and the source before deciding if I accept it". That attitude is just lost and I don't think it will be regained and I think it's the reason we are all in a death spiral.

I don't think you are being totally fair to the paper, but you do point out something that drives me crazy: local freakouts about year-to-year changes in math tests scores. It's like people don't realize that these 8th grade students are not the same as those.

Yea, it's strange that the line didn't move quickly. I would give grace for a couple weeks to a few months, but next year? The timing feels really disconnected.