In Windows 7 I’m using Shark007’s codec pack along with the 64-bit components. Special codecs like nvidia purevideo, CoreAVC or the Cyberlink 264/AVC codec help you to get real smooth HD playback. Especially in 1080p movies the visual difference between two frames is so broad that you need to interpolate additional information between two frames by guessing the movement between the frames.
For Mac OS X is no such solution. Some players like MPlayer OSX Extended pretend to improve the visual quality, but i never really got the player working (crashes when trying to update). The VLC playback is stable and "consistently smooth" but not as smooth as I’m used from Windows 7 and the is no way to improve it by using different codecs or tweaking.
Then I found CoreCodec CorePlayer for OS X.
There is hardly any information about this player in the web. I couldn’t even find a trial version for mac. So I just bought it and tried it. Well, the first try gave me no sound (AC3, DTS). After reading in the forums I found out that many people complain about this missing feature. I tried to write some apple script that starts CorePlayer and VLC (disable video stream) simultaneously, but most HD movies went out of sync after short time. I tried to compensate this with global keyboard shortcuts for VLC’s audio delay tweaking, but its no fun.
By accident I found a forum post about using Perian‘s AudioQueue as the AC3 audio pass through for CorePlayer.
I can say its up sync most times in HD movies. Just don’t do anything else in the background (at least on my i5).
The playback is still not as smooth as in Windows 7 in extreme high motion scenes but its a lot better in slow and normal motion scenes. It plays my MKV h.264 AC3, DTS movies quite nicely and I’m using it as a VLC replacement on Snow Leopard now. Worked for me.