It seems like most of these "an LLM solved this in only X hours! " could have been "I found an open source solution that did what I needed with X minutes of web search."

Which doesn't mean that the LLM definitely couldn't have accomplished it without the prior art (in either the training set or explicitly in a a web search). But it does seem to be a trend.