Beware when hooking window procedures returned by GetWindowLongPtr

I'm currently rewriting (read: improving) a bunch of stuff for Hades, and one of those things is the window management. Previously I was using a messy system in which I used window subclassing in order to 'hook' the window procedure of the display window(s) of the game.

The problem with this approach is that window subclassing is quite 'ugly', and it required me to hook several APIs, for which the implementation was also quite 'ugly'.

I solved this by moving to a detour-based system on the window procedure for the game's display window. Whilst there are some obvious problems with this approach too (which I will not get into now because they're complex and irrelevant) I believe the pros outweigh the cons.

One problem I ran into whilst implementing this new system however is the behaviour of GetWindowLongPtr when retrieving a window procedure, and how that behaviour changes depending on whether your module is ANSI or Unicode, and whether the target is ANSI or Unicode.

If the character sets don't match, GetWindowLongPtr does not return a true pointer, it returns a special internal handle which is then 'translated' by CallWindowProc. MSDN actually documents this behaviour but I accidentally skimmed over it and ended up reversing this behaviour myself. >_>

Long story short, if you want to detour the window procedure you need the 'real' pointer, so the obvious solution is to call both functions, detect which one is the handle, and throw it away (hence leaving you with the real pointer).

Here's the code to do that, thankfully the handle detection is very simple, I just wish I had read MSDN more closely the first time:

HookWindow-v20100117a