if I set font to Microsoft Yahei in ubuntu mate 17.10, the font rendering effect is shown up.
but the font rendering effect is obviously worse than the effect shown in ubuntu mate 16.04
Check the options in mate-appearance-properties to see if anything from there can enhance how rendering is performed on your display.
If none of that works, there are a series of matches collectively known as Infinality which should help to resolve the problem with more fine- grain options, but you’ll have to add the repositories for it, and the repos for latest Ubuntu doesn’t have the fontconfig-infinality package.