Debian Bookworm was released with Podman 4.3 and Linux kernel 6.1 so it should have decent rootless OverlayFS support. But somehow VFS was used by default running Podman rootless containers and this will have a huge impact on performance.
To switch such default away from VFS to OverlayFS, the easiest way is to install containers-storage (which is just a suggested package for podman, thus usually not considered as a dependency).