One last thing ... I went down a long rabbit hole trying to understand why APT (but not apt-get) kept trying to install modules:
linux-objects-nvidia-580-6.14.0-37-genericlinux-signatures-nvidia-6.14.0-37-generic
None of these came up as dependencies of packages that I had installed. Even when I removed the ppa:graphics-drivers server and purged everything APT could know, it still wanted to install these modules even though I had the 550 DKMS installed.
No matter what I tried, nothing seemed to work other than blocking the offending package via pin, e.g. I added a file /var/lib/apt/preferences.d/99-block-nvidia-580 with:
Package: linux-objects-nvidia-*
Pin: release *
Pin-Priority: -1
I don't know what the NVIDIA and Ubuntu maintainers have done here to create this mess (mostly NVIDIA btw) but I guess this is where we are at.
Cannot. Wait. To. Switch. To. AMD.