By the way I accidentally solved this.
When I connect to a remote machine, caja would show file icons. Then I realized that it should've been the opposite because I selected "Local Files Only" in "Show Text Icons" menu.
Then I selected "Always" and it wouldn't show icons on remote machine as well. Then finally i selected "Never" and now i can see file icons on both remote and local machines.
In short I misunderstood this setting. I think the original issue is this: https://github.com/mate-desktop/caja/issues/1047
But my question was not that, I didn't care about seeing file content in icons, I just wanted to discriminate different file types in my folder. So it's achieved by this setting.
Then i can see file icons.