The problem is where it's measuring joules of energy. To use cars as an example:

It measures joules of energy as in "how much heat the gasoline we burn produces", some of which we convert to mechanical energy to drive the car, but the majority is just waste heat going out the tailpipe.

By comparison an electric car powered by solar has no tailpipe. There's still a bit of waste heat from electrical resistance, but nowhere near as much.

If we measure like this, by converting a gasoline car to electric (powered by solar for the sake of ignoring some complexity), and driving the same distance, we somehow managed to cut our "energy demand" in half. Despite the fact that we're demanding the exact same thing from the system.

If we measured "joules delivered to the tires of the car" we wouldn't have the same issue. At least until someone starts arguing about how their car is more aerodynamic so joules delivered to the tires should count for more in it.

Edit: We could also go in the other direction. Instead of reporting it as 1kw of solar energy (electricity) it could be 4kw of solar energy (the amount of sunlight shining on the solar panels)... No one does this for obvious reasons, but it's more similar to that primary energy number for fuel in many ways.