I am witnessing a delay of around 5 seconds when opening any folder for the first time (and then about half as long subsequently) in caja file manager.
In Nautilus and Thunar, every folder opens instantly.
I think if opening a file manager take 5 seconds it’s should not be a problem considering it might be do to the fact on what files you have in those folders
It might be worth taking a look to see what you can do to help that file manager out