Ok, after like 15hours of playing with it, I got out of ideas, so I'm gonna ask here.
I've installed the latest ubuntu mate 20.10 64bit (+ did all app upgrades), fixed audio problem with pulseaudio, set 4kp60, set gpu_mem 256, followed this https://www.dedoimedo.com/computers/rpi4-ubuntu-mate-hw-video-acceleration.html to get some hw video acceleration but still when I try to play 4K mkv (HEVC 10bit) in vlc, it doesnt work, just black screen... 1080p seems to be quite ok...
Also in chromium/FF youtube is choppy, sometimes even on 360p (even tho I enabled the hw acceleration there too)
Does it work for others ok, or is there no way to fix it?
Because I also tried libreelec and it works fine there (4K mkv and the youtube app there too), so the HW should be able to handle it just fine
also loading/rendering pages in chromium and FF takes a while
I run the OS from Kingston 64GB micSDXC Canvas Select Plus 100R A1 C10
and it's on wifi, but the speed is getting to 50down/40up MiB/s, so loading should not be a problem, probably rendering