what would be the point of the government fining itself though?

Now that I'm thinking of it, it would create the need for an extra gaggle of bureaucrats to oversee the process,so I suppose someone might see a point to it ...

You may think you're funny or something, but boy do I have news for you.

There absolutely are fines for French administrations. And, knowing the French tax system, they've probably found a way to levy VAT and some other taxes on top of those fines.