I have lingering setup and I can still access the container but for what ever reason Podman seems to be unable to access the GPU for no apparent reason.
I think this may be an issue with systemd but I’m not entirely sure.
Solution: you need to be logged in for it to work. I accomplished this on a separate VM with autologin to icewm.
No idea then :( AFAIK the logind mechanism I mentioned originally is based only on permissions, but I had never really needed to look into it further.