I feel like code for public systems, government systems should be open source.

A lot of it is! It has been this way for a long time in some parts of the public sector, eg:

https://github.com/ministryofjustice

I don't know of a department that does it as well as MoJ, though. Caveats exist around old private sector implemented systems like the prisons and probation databases etc, which even MoJ itself doesn't own the IP for. But everything made by civil servants or contractors at MoJ ends up published in that org unless there's a good reason not to.

Edit: FOI in principle allows you to request a cut of a git repo etc for a service, so you can impose annoyance upon departments that are less open.