You can do this with libkrun. I use it as a podman backend which works exactly as you describe here.
podman run --rm -it --runtime=krun ubuntu sh
Error: default OCI runtime "krun" not found: invalid argument
Oh, you do need to install it. e.g. on arch: https://archlinux.org/packages/extra/x86_64/krun/
I do have both podman and crun installed, but I don't see any krun or libkrun packages for ubuntu unfortunately :/