In my case the problem was that mplayer used wrong xvideo adaptor.
$ mplayer -v file.avi
using Xvideo port 97 for hw scaling
So it's using port 97.
Running xvinfo displays information about the xv xorg extension. Two adaptors in my case; one bad and one good. Mplayer is unfortunately using the bad one because it always picking number #0.
Relevant parts from xvinfo:
Adaptor #0: "Intel(R) Textured Video"
port base: 97
Adaptor #1: "Intel(R) Video Overlay"
port base: 113
Port 97 is Textured Video and port 113 is Video Overlay. The -vo parameter can be used to tell mplayer which adaptor to use.
mplayer -vo xv:port=113 file.avi
And bam! That's it.