AArch64 on Raspberry Pi 2 (rev 1.2), 3B, 3B+

Tips, Tricks and Tutorials Raspberry Pi
4

You should be able to remove the desktop with:

sudo apt install dctrl-tools
sudo apt purge $(apt-cache dumpavail | grep-dctrl -nsPackage \( -XFArchitecture arm64 -o -XFArchitecture all \) -a -wFTask xubuntu-desktop | tr '\n' ' ')
sudo apt autoremove

Then you can:

sudo tasksel

and install what you want!