For now, I'm settling on the 550.144.03 line of drivers. I am still looking at how to get the 550.163.01 drivers from Debian Trixie installed.
Anyway, to remove the existing 550 and 580 drivers and install 550.144.03 and hold:
# remove the master packages
sudo apt remove nvidia-driver-550 nvidia-driver-580 --purge
# clean up the dependencies of the master (or meta) package
sudo apt autoremove --purge
# for some reason these were still installed; so removed them manually
sudo apt remove libnvidia-compute-580 libnvidia-compute-580:i386
# now install the 550.144.03 drivers
sudo apt install nvidia-driver-550=550.144.03-0ubuntu0~gpu24.04.1 \
libnvidia-common-550=550.144.03-0ubuntu0~gpu24.04.1 \
libnvidia-gl-550=550.144.03-0ubuntu0~gpu24.04.1 \
nvidia-dkms-550=550.144.03-0ubuntu0~gpu24.04.1 \
nvidia-kernel-common-550=550.144.03-0ubuntu0~gpu24.04.1 \
nvidia-kernel-source-550=550.144.03-0ubuntu0~gpu24.04.1 \
libnvidia-extra-550=550.144.03-0ubuntu0~gpu24.04.1 \
libnvidia-decode-550=550.144.03-0ubuntu0~gpu24.04.1 \
libnvidia-encode-550=550.144.03-0ubuntu0~gpu24.04.1 \
nvidia-utils-550=550.144.03-0ubuntu0~gpu24.04.1 \
xserver-xorg-video-nvidia-550=550.144.03-0ubuntu0~gpu24.04.1 \
libnvidia-cfg1-550=550.144.03-0ubuntu0~gpu24.04.1 \
libnvidia-fbc1-550=550.144.03-0ubuntu0~gpu24.04.1 \
libnvidia-decode-550:i386=550.144.03-0ubuntu0~gpu24.04.1 \
libnvidia-encode-550:i386=550.144.03-0ubuntu0~gpu24.04.1 \
libnvidia-fbc1-550:i386=550.144.03-0ubuntu0~gpu24.04.1 \
libnvidia-gl-550:i386=550.144.03-0ubuntu0~gpu24.04.1 \
libnvidia-compute-550=550.144.03-0ubuntu0~gpu24.04.1 \
nvidia-compute-utils-550=550.144.03-0ubuntu0~gpu24.04.1 \
libnvidia-compute-550:i386=550.144.03-0ubuntu0~gpu24.04.1
# hold to prevent updates; for some reason also need to hold libnvidia-common-550
sudo apt-mark hold nvidia-driver-550 libnvidia-common-550