Happy New Year!
Well, in theory the 64-bit aarch64 should be faster due to new computational CPU features supported in 64-bit mode like NEON.
Also in Linux kernel 5.2 and above there is support for some of the RPI4 GPU features like HDR as per LibreELEC 9.2.0 release notes.
With 64-bit there is an overhead to the memory though - the size of the applications is bigger and they occupy more RAM than the same 32-bit applications. Neither of those is an issue with big enough SD card and with the 4 GB version of RPI4.
In practice currently Raspbian which is 32-bit armhf is the fastest distro for RPI4 because it has Linux kernel specifically built for the Raspberry and also has GPU drivers with support for hardware graphical acceleration without which the CPU is loaded with graphics tasks.
What I like about RPI on the software side is that unlike other SOC boards it has huge community and devoted volunteers that work on improving the software and sharing their work like the Unofficial Ubuntu Server 18. 04.3 image by James A. Chambers.
Also with the announced official support by Canonical for Raspberry Pi starting with Ubuntu Server 19.10 and above and with the declared collaboration with the Rspberry Pi Foundation at some point we'll have fully supported aarch64 image with Linux kernel 5.3.x, with hardware GPU acceleration and latest CPU features, on top of which we'll be able to install Mate (or another desktop environment of choice).