Beware when hooking window procedures returned by GetWindowLongPtr

I'm currently rewriting (read: improving) a bunch of stuff for Hades, and one of those things is the window management. Previously I was using a messy system in which I used window subclassing in order to 'hook' the window procedure of the display window(s) of the game.

The problem with this approach is that window subclassing is quite 'ugly', and it required me to hook several APIs, for which the implementation was also quite 'ugly'.

I solved this by moving to a detour-based system on the window procedure for the game's display window. Whilst there are some obvious problems with this approach too (which I will not get into now because they're complex and irrelevant) I believe the pros outweigh the cons.

One problem I ran into whilst implementing this new system however is the behaviour of GetWindowLongPtr when retrieving a window procedure, and how that behaviour changes depending on whether your module is ANSI or Unicode, and whether the target is ANSI or Unicode.

If the character sets don't match, GetWindowLongPtr does not return a true pointer, it returns a special internal handle which is then 'translated' by CallWindowProc. MSDN actually documents this behaviour but I accidentally skimmed over it and ended up reversing this behaviour myself. >_>

Long story short, if you want to detour the window procedure you need the 'real' pointer, so the obvious solution is to call both functions, detect which one is the handle, and throw it away (hence leaving you with the real pointer).

Here's the code to do that, thankfully the handle detection is very simple, I just wish I had read MSDN more closely the first time:

HookWindow-v20100117a

LuaNinja for WoW v3.3.0

I've updated LuaNinja for WoW v3.3.0. Download it here.

Have fun you dirty cheaters.

Hades Now Fully Dynamic

Hooray, I have now gotten all of Hades working in a fully dynamic manner. Previously in order to run a game under Hades you would have to inject Hades at the time of process creation. Recently though I have been rewriting large portions of Hades, and one of those portions has been the hooking lib I'm using.

Whilst my original hooking library used IAT and EAT hooks I found that far too restrictive for my needs, so I wrote an inline hooking library (similar to Microsoft Detours, except mine has full x64 support -- technically Microsoft Detours does too, but you have to pay $10000 to get your hands on it).

Not only does this mean I can now inject Hades at any time, it also means I can eject Hades at any time. This is AWESOME when it comes to testing because it means I can rapidly test new features and code without having to restart the game (which was a huge pain previously).

It also means I have full support for the Steam overlay (which was previously flaky because it doesn't like it when it's not the first module to hook), and I can implement some of my planned features much easier.

One of the next planned features I want to tackle (after some more rewriting of the core framework) is DirectX 10 support, which I've been putting off for a long time because previously I would have been forced to do a full device replacement like I did with the DirectX 9 layer.

The new hooking library though has allowed me to turn about 4 pages of code for the DX9 layer into about half a page, and the same can be applied to the DX10 layer.

Whilst it's not ready for public consumption yet, I'm considering releasing my hooking library if there's enough interest for it.

It may be a while though because before I get the hooking library ready for public use there's a lot more other stuff I want to work on in Hades like the Lua API, the .NET API, the generic anti-anti-cheat system, and better DirectInput support (which is currently quite basic due primarily to a lack of games I own which utilize it heavily).

Anyway, someone requested information on Hades from me on MSN and told me to post it here so there you go. Should be interesting to someone.