Caja is essential MATE desktop component. So I think that it is expected, that it is hard to replace it with something different. Moreover Caja is more functional than Nautilus or Thunar. See this screenshot to understand what I mean.
This reminds me topic about Nautilus replacement with Nemo on ordinary Ubuntu. It was almost impossible and non-userfriendly too. IMHO Nautilus and Ubuntu desktop (say GNOME Shell) are degrading very fast, so I'll ignore them.
My idea is to ignore GNOME and Nautilus and continue to use Caja inside MATE. I do not have free time to dig into Nautilus-MATE problem and will not recommend others to dig into it. Our universe is MATE and we should make it spinning and running best.