Debian Bookworm was released with Podman 4.3 and Linux kernel 6.1 so it should have decent rootless OverlayFS support. But somehow VFS was used by default running Podman rootless containers and this will have a huge impact on performance.
To switch such default away from VFS to OverlayFS, the easiest way is to install containers-storage (which is just a suggested package for podman, thus usually not considered as a dependency).
Steps
First, backup your containers if necessary.
Reset storage back to initial state
podman system reset
then
sudo apt update sudo apt install containers-storage
Under the Hood
/usr/share/containers/storage.conf in the package contains a line that sets the default storage driver to OverlayFS.
Validation
Run
podman info | grep graphDriverName
and the value should change from vfs to overlay.
Thanks. Disk usage was nonsense with vfs.