Remix.run Logo
astro1234 13 hours ago

As a data scientist it is very frustrating to see a consistent lack of error bars on these numbers or any discussion about the magnitude of uncertainty around them by all news reports every time these numbers get released

mrloopex 12 hours ago | parent | next [-]

This should have nothing to do with data science but basic math literacy. Quantities without context are meaningless. Reporters should know better but if when headlines have to catchy day in day out … shrugs. Stock market reporters are the worst: a 1000 point drop in the Dow meant a lot more when it was 7000 than it does at 40,000.

astro1234 11 hours ago | parent [-]

Oh my god do not trigger me about the news reporting “points” for the DOW or the S&P, I have seriously considered calling news stations about this.

Or not providing inflation adjusted values when referring to historical financial reference points.

I have to go find a happy place now

mjburgess 13 hours ago | parent | prev | next [-]

If honest, the error bars would be so large as to make drawing conclusions from these numbers seem pointless. Certainly, both jobs growth and jobs loss are within the CI.

What's actually being tracked by people using this data is the delta from previous years -- which, conditioning on the same collection methodology etc -- is more reliable.

The actionable piece of info is sign(delta)

rented_mule 7 hours ago | parent | next [-]

Something the best data scientist I ever worked with was adamant about... there is almost never an inherent "speed bump" at 0. If error bounds expand on both sides of 0, we have no idea if the true sign is positive or negative. There is no "directional" read in that case. There are some areas where 0 has a special meaning, but that's rare and the cause of that needs to be understood before assuming that's the case. Everywhere else, 0 is just another point on the number line.

> the error bars would be so large as to make drawing conclusions from these numbers seem pointless

That is often the case, and being truly data based requires acknowledging that. We can then act based on other data, or admit that we're falling back to intuition and abandoning data based decision making for now.

hrhdhdh 12 hours ago | parent | prev [-]

[flagged]

mjburgess 12 hours ago | parent [-]

Estimating the number of jobs lost is very different than estimating the sign of a difference of a fixed "error-prone method for measuring jobs lost"

The former is asking "are my weighing scales accurate?" the latter is, "do I look like i'm loosing weight?"

srghrhsr 8 hours ago | parent [-]

[flagged]

europoorean 13 hours ago | parent | prev | next [-]

As a physicist, I always shed a tear when looking at data without error bars.

As a data scientist/generic programmer now, I don't really see them anywhere. I remember a colleague during my postdoc saying 'don't worry, no error bars in industry' when I asked about regression with ML lol

natsucks 13 hours ago | parent [-]

hah. it's true. no interest in uncertainty except when forced.

Noaidi 13 hours ago | parent [-]

Our fake economy needs even more fake certainty now. They are all hiding from the obvious issue of stagflation because it is the most difficult economic problem to solve. The Fed know this which is why they did nothing with rates.

biophysboy 13 hours ago | parent | prev | next [-]

The bureau of labor statistics publishes confidence intervals in their reports

jmalicki 13 hours ago | parent | next [-]

Not in their primary news release. A footnote says it is "generally conducted at the 90% interval", but the individual numbers do not have their actual computed error bars.

Yes, they do publish them, but it is disappointing their primary releases do not have them.

https://www.bls.gov/news.release/pdf/empsit.pdf

biophysboy 13 hours ago | parent [-]

It’s common in scientific/stats publishing to have the headline state the avg/median, the main text plot the errors, and the supplementary give the explicit numbers. They have a big page about std errors.

jmalicki 12 hours ago | parent [-]

The main text doesn't plot the error bars, and the full data tables in their release don't have error bars either.

You have to really dig into their website to find their std errors.

biophysboy 12 hours ago | parent [-]

bls.gov/ces is the main page for this survey and has a plot.

The error updates annually. Their handbook has a Reliability section where they detail sampling/nonsampling error.

throwaway27448 13 hours ago | parent | prev | next [-]

NBC wouldn't know how to navigate bls.gov if you put a gun to their head. And it's all downhill from there

biophysboy 12 hours ago | parent [-]

Based on the discussions in this thread, they aren’t alone

throwaway27448 12 hours ago | parent [-]

I don't think most of the people in this thread are journalists

biophysboy 12 hours ago | parent [-]

God help us if they ever become one

astro1234 12 hours ago | parent | prev [-]

Yes! I am aware of this and they are really great. Problem is nobody will ever read that, but they WILL watch Fox News, CNBC, etc. and those outlets never even point out that there are errorbars/uncertainty. Then I hear things like outrage when BLS revises numbers up or down (facepalm).

renegade-otter 13 hours ago | parent | prev | next [-]

The numbers are as good as they can get with hard manual work that goes into compiling them.

The problem is that small businesses lag by 1-2 months reporting them, so the initial number being reported is almost always irrelevant, as it mostly applies to numbers coming out of larger corporations.

jstanley 13 hours ago | parent | next [-]

> The numbers are as good as they can get

That doesn't mean they don't warrant error bars.

Error bars aren't there to show that you didn't do a good job collecting the data, they're there to show your level of uncertainty.

vlovich123 13 hours ago | parent [-]

I would say the initial numbers should be embargoed until the number is reliable. People don’t understand error bars and reporting can more easily manipulate the current situation without outright lying.

theptip 13 hours ago | parent | next [-]

No, just report them with error bars. We know the 95th percentile revision size of the distribution.

Moot point as the people who actually act on the data understand how to roll their own anyway.

jfengel 12 hours ago | parent [-]

Error bars would be misleading, because this isn't a sampling error. These start with actual counts. The more important errors are systematic: the definitions of terms, the reporting mechanisms, etc.

Except that those things are constant from month to month, so deltas largely cancel them out.

The problem isn't in the size of the possible errors, but in understanding what those numbers mean and how they are gathered. That requires far more background reading.

twosdai 13 hours ago | parent | prev | next [-]

Both? I don't think error bars and waiting to release data until more accurate are mutually exclusive.

vlovich123 13 hours ago | parent [-]

That’s what I meant. There should be error bars to reflect uncertainty of the inherent data, but they should wait until all the data that could change things comes in rather than using projections which always drastically change

jmalicki 13 hours ago | parent [-]

No, they shouldn't. Timely estimates as to the best knowledge of our current situation is extremely important for decision making under uncertainty.

If the error bars are large, that is important information to know.

vlovich123 13 hours ago | parent | next [-]

What decisions are so important and time sensitive they depend on these numbers right after a month is over? And also we at one point were making decisions without these numbers yet the quality of the decision making isn’t clearly improved.

jmalicki 12 hours ago | parent [-]

"What kind of stocks should my mutual fund invest in?"

"What sort of startups should I fund?"

"Should my business aggressively hire this quarter anticipating growth, or hold out for a possible recession?"

vlovich123 11 hours ago | parent [-]

A few months of extra no data won’t impact the stock investments if no one has that data. Startup funding is similarly not so time sensitive.

As for businesses hiring, they should hire based on their own growth. Using an external signal that itself gets adjusted drastically is similarly problematic; I don’t hire because I think no one else is hiring and the numbers get majorly adjusted up. Similarly I hire a bunch of people and it turns out no one else hired.

No one benefits from faster numbers that are majorly shifted in a few months anyway. You wouldn’t make any different decision if no one had the numbers. The only thing you’re doing differently is when the numbers come out your gambling whether or not you think the true number will be revised in some major way in a few months. That’s not helpful for the economy writ large.

win311fwg 13 hours ago | parent | prev [-]

> If the error bars are large, that is important information to know.

The data is available. If it were important to the causal reader, however, then those casual readers would have already demanded its presence. But they don't care. What difference does it make to Hilary Homemaker? She just wants to get to say "Did you hear about the bad economic news?" when gossiping with her friends.

Those who need to understand the uncertainty will already be looking at more than a news headline anyway.

parineum 12 hours ago | parent | prev [-]

Ah yes, lie to them and withold information because the public is too stupid to understand. Then, when they don't trust you anymore, throw up your hands and say it's not your fault.

jmalicki 13 hours ago | parent | prev | next [-]

That's true for the establishment survey, but the population survey, where they contact individuals, is what is actually used to calculate these numbers.

It is literally calling/writing 60,000 people and asking "are you employed? did you look for a job? Etc."

"How the government calculates unemployment" [0]

| Does anyone in this household have a business or a farm? | Last week, did you do any work for (either) pay (or profit)? | If the answer to question 1 is "yes" and the answer to question 2 is "no," the next question is: | Last week, did you do any unpaid work in the family business or farm? | For those who reply "no" to both questions 2 and 3, the next key questions used to determine employment status are: | Last week, (in addition to the business) did you have a job, either full or part time? Include any job from which you were temporarily absent. | Last week, were you on layoff from a job? | What was the main reason you were absent from work last week? | For those who respond "yes" to question 5 about being on layoff, the following questions are asked:

[0] https://www.bls.gov/cps/cps_htgm.htm

amanaplanacanal 13 hours ago | parent [-]

You have to wonder, since they are doing two different surveys trying to measure the same reality, which one more accurately measures the facts on the ground? Asking businesses how many people they are employing, or asking households whether or not their folks are working? The numbers don't really match up between the two.

jmalicki 12 hours ago | parent [-]

This is very common for economic statistics.

GDP too. That's how you can get more accurate information, and you have actual numbers on the discrepencies.

They rely on the household data more because it doesn't have as much systemic error, even though it does have more sampling error. Asking businesses gives you a more complete picture, but it has much more bias.

Hell, this is why even in accounting, you have double-entry bookkeeping, and even there, the numbers don't always match up.

Or "my bank account doesn't match up with my cashflow statement".

It's not that one is better and you should use that, calculating multiple ways is central to accounting so you can detect errors.

jm4 13 hours ago | parent | prev | next [-]

What makes you think it's small businesses? I would assume most businesses doing any reporting at all are using a payroll system. Mine handles the reporting automatically. The kinds of payroll systems used by small businesses can be set up in a day. If anything, small businesses that make even a small effort to stay on top of things have an easier time than big ones with complicated systems and processes.

pgwhalen 13 hours ago | parent | prev [-]

I think you say “as good as they can get with [the] hard manual work that goes into compiling them” as a way to respect the challenge of the BLS and say they’re doing the best with their resources, but it’s worth pointing out they could be better with more resources.

jmalicki 13 hours ago | parent [-]

DOGE hurt them pretty hard: "The Bureau of Labor Statistics posted a notice on Wednesday stating that it stopped collecting data in three not-so-small cities (Lincoln, Nebraska; Buffalo, New York; and Provo, Utah) and increased “imputations” for certain items"

https://www.cnn.com/2025/06/05/economy/cpi-data-bls-reductio...

cousinbryce 13 hours ago | parent | prev | next [-]

https://www.bls.gov/jlt/jltreliability.htm Read the docs

astro1234 12 hours ago | parent [-]

Oh I’m well aware the BLS knows what they’re doing and publish error bars, they are great. The problem is news articles, no one ever reads BLS reports but they do watch Fox News, CNBC, etc. and those outlets never ever get into uncertainty.

seanhunter 12 hours ago | parent | prev | next [-]

The actual source this comes from has a 90% confidence interval in the published number. When people talk about the economy gaining or losing jobs they are talking about the bureau of labor statistics "nonfarm payroll" number. The methodology and a lot more are on the bls website

https://www.bls.gov/ces/

anigbrowl 11 hours ago | parent | prev | next [-]

Wait until you see the weekly reports for the Strategic Petroleum reserve. Go ahead, try to highlight that text.

https://www.spr.doe.gov/dir/dir.html

amelius 13 hours ago | parent | prev | next [-]

Yes they should have said 23.0 * 10^3.

inigyou 13 hours ago | parent [-]

2.30 × 10⁴ or bust

jewslop 12 hours ago | parent | prev | next [-]

[dead]

selicos 12 hours ago | parent | prev [-]

The BLS has been gutted. It's surprising anything negative is escaping in these reports.

splatter9859 12 hours ago | parent [-]

Exactly. I'm somewhat surprised someone's give-a-damn wasn't broken enough to do a cursory filter. Not like it isn't plainly obvious in other statistics, however.