Tutorial 24: Windows Hooks

We will learn about Windows hooks in this tutorial. Windows hooks are very powerful. With them, you can poke inside other processes and sometimes alter their behaviors.
Download the example here.

Theory:

Windows hooks can be considered one of the most powerful features of Windows. With them, you can trap events that will occur, either in your own process or in other processes. By "hooking", you tell Windows about a function, filter function also called hook procedure, that will be called everytime an event you're interested in occurs. There are two types of them: local and remote hooks.

Local hooks trap events that will occur in your own process.
Remote hooks trap events that will occur in other process(es). There are two types of remote hooks

thread-specific traps events that will occur in a specific thread in other process. In short, you want to observe events in a specific thread in a specific process.
system-wide traps all events destined for all threads in all processes in the system.

When you install hooks, remember that they affect system performance. System-wide hooks are the most notorious. Since ALL related events will be routed through your filter function, your system may slow down noticeably. So if you use a system-wide hook, you should use it judiciously and unhook it as soon as you don't need it. Also, you have a higher chance of crashing the other processes since you can meddle with other processes and if something is wrong in your filter function, it can pull the other processes down to oblivion with it. Remember: Power comes with responsibility.
You have to understand how a hook works before you can use it efficiently. When you create a hook, Windows creates a data structure in memory, containing information about the hook, and adds it to a linked list of existing hooks. New hook is added in front of old hooks. When an event occurs, if you install a local hook, the filter function in your process is called so it's rather straightforward. But if it's a remote hook, the system must inject the code for the hook procedure into the address space(s) of the other process(es). And the system can do that only if the function resides in a DLL. Thus , if you want to use a remote hook, your hook procedure must reside in a DLL. There is two exceptions to this rule: journal record and journal playback hooks. The hook procedures for those two hooks must reside in the thread that installs the hooks. The reason why it must be so is that: both hooks deal with the low-level interception of hardware input events. The input events must be recorded/playbacked in the order they appeared. If the code of those two hooks is in a DLL, the input events may scatter among several threads and it is impossible to know the order of them. So the solution: the hook procedure of those two hooks must be in a single thread only i.e. the thread that installs the hooks.
There are 14 types of hooks:

WH_CALLWNDPROC called when SendMessage is called
WH_CALLWNDPROCRET called when SendMessage returns
WH_GETMESSAGE called when GetMessage or PeekMessage is called
WH_KEYBOARD called when GetMessage or PeekMessage retrieves WM_KEYUP or WM_KEYDOWN from the message queue
WH_MOUSE called when GetMessage or PeekMessage retrieves a mouse message from the message queue
WH_HARDWARE called when GetMessage or PeekMessage retrieves some hardware message that is not related to keyboard or mouse.
WH_MSGFILTER called when a dialog box, menu or scrollbar is about to process a message. This hook is local. It's specifically for those objects which have their own internal message loops.
WH_SYSMSGFILTER same as WH_MSGFILTER but system-wide
WH_JOURNALRECORD called when Windows retrieves message from the hardware input queue
WH_JOURNALPLAYBACK called when an event is requested from the system's hardware input queue.
WH_SHELL called when something interesting about the shell occurs such as when the task bar needs to redraw its button.
WH_CBT used specifically for computer-based training (CBT).
WH_FOREGROUNDIDLE used internally by Windows. Little use for general applications
WH_DEBUG used to debug the hooking procedure

Now that we know some theory, we can move on to how to install/uninstall the hooks.
To install a hook, you call SetWindowsHookEx which has the following syntax:

SetWindowsHookEx proto HookType:DWORD, pHookProc:DWORD, hInstance:DWORD, ThreadID:DWORD

HookType is one of the values listed above, e.g., WH_MOUSE, WH_KEYBOARD

pHookProc is the address of the hook procedure that will be called to process the messages for the specified hook. If the hook is a remote one, it must reside in a DLL. If not, it must be in your process.

hInstance is the instance handle of the DLL in which the hook procedure resides. If the hook is a local one, this value must be NULL

ThreadID is the ID of the thread you want to install the hook to spy on. This parameter is the one that determines whether a hook is local or remote. If this parameter is NULL, Windows will interpret the hook as a system-wide remote hook that affects all threads in the system. If you specify the thread ID of a thread in your own process, this hook is a local one. If you specify the thread ID from other process, the hook is a thread-specific remote one. There are two exceptions to this rule: WH_JOURNALRECORD and WH_JOURNALPLAYBACK are always local system-wide hooks that are not required to be in a DLL. And WH_SYSMSGFILTER is always a system-wide remote hook. Actually it is identical to WH_MSGFILTER hook with ThreadID==0.

If the call is successful, it returns the hook handle in eax. If not, NULL is returned. You must save the hook handle for unhooking later.

You can uninstall a hook by calling UnhookWindowsHookEx which accepts only one parameter, the handle of the hook you want to uninstall. If the call succeeds, it returns a non-zero value in eax. Otherwise, it returns NULL.
Now that you know how to install/uninstall hooks, we can examine the hook procedure.
The hook procedure will be called whenever an event that is associated with the type of hook you have installed occurs. For example, if you install WH_MOUSE hook, when a mouse event occurs, your hook procedure will be called. Regardless of the type of hook you installed, the hook procedure always has the following prototype:

HookProc proto nCode:DWORD, wParam:DWORD, lParam:DWORD

nCode specifies the hook code.
wParam and lParam contain additional information about the event

HookProc is actually a placeholder for the function name. You can name it anything you like so long as it has the above prototype. The interpretation of nCode, wParam and lParam is dependent on the type of hook you install. So as the return value from the hook procedure. For example:

WH_CALLWNDPROC

nCode can be only HC_ACTION which means there is a message sent to a window

wParam contains the message being sent, if it's not zero

lParam points to a CWPSTRUCT structure

return value: not used, return zero

WH_MOUSE

nCode can be HC_ACTION or HC_NOREMOVE

wParam contains the mouse message

lParam points to a MOUSEHOOKSTRUCT structure

return value: zero if the message should be processed. 1 if the message should be discarded.

The bottom line is: you must consult your win32 api reference for details about the meanings of the parameters and return value of the hook you want to install.
Now there is a little catch about the hook procedure. Remember that the hooks are chained in a linked list with the most recently installed hook at the head of the list. When an event occurs, Windows will call only the first hook in the chain. It's your hook procedure's responsibility to call the next hook in the chain. You can choose not to call the next hook but you'd better know what you're doing. Most of the time, it's a good practice to call the next procedure so other hooks can have a shot at the event. You can call the next hook by calling CallNextHookEx which has the following prototype:

CallNextHookEx proto hHook:DWORD, nCode:DWORD, wParam:DWORD, lParam:DWORD

hHook is your own hook handle. The function uses this handle to traverse the linked list and search for the hook procedure it should call next.

nCode, wParam and lParam you can just pass those three values you receive from Windows to CallNextHookEx.

An important note about remote hooks: the hook procedure must reside in a DLL which will be mapped into other processes. When Windows maps the DLL into other processes, it will not map the data section(s) into the other processes. In short, all processes share a single copy of code but they will have their own private copy of the DLL's data section! This can be a big surprise to the unwary. You may think that when you store a value into a variable in the data section of a DLL, that value will be shared among all processes that load the DLL into their process address space. It's simply not true. In normal situation, this behavior is desirable since it provides the illusion that each process has its own copy of the DLL. But not when Windows hook is concerned. We want the DLL to be identical in all processes, including the data. The solution: you must mark the data section as shared. You can do this by specifying the section(s) attribute in the linker switch. For MASM, you need to use this switch:

/SECTION:<section name>, S

The name of the initialized data section is .data and the uninitialized data is .bss. For example if you want to assemble a DLL which contains a hook procedure and you want the uninitialized data section to be shared amoung processes, you must use the following line:

link /section:.bss,S /DLL /SUBSYSTEM:WINDOWS ..........

S attribute marks the section as shared.

Example:

There are two modules: one is the main program which will do the GUI part and the other is the DLL that will install/uninstall the hook.

;--------------------------------------------- This is the source code of the main program --------------------------------------
.386
.model flat,stdcall
option casemap:none
include \masm32\include\windows.inc
include \masm32\include\user32.inc
include \masm32\include\kernel32.inc
include mousehook.inc
includelib mousehook.lib
includelib \masm32\lib\user32.lib
includelib \masm32\lib\kernel32.lib

wsprintfA proto C :DWORD,:DWORD,:VARARG
wsprintf TEXTEQU <wsprintfA>

.const
IDD_MAINDLG                   equ 101
IDC_CLASSNAME              equ 1000
IDC_HANDLE                     equ 1001
IDC_WNDPROC                 equ 1002
IDC_HOOK                         equ 1004
IDC_EXIT                           equ 1005
WM_MOUSEHOOK             equ WM_USER+6

DlgFunc PROTO :DWORD,:DWORD,:DWORD,:DWORD

.data
HookFlag dd FALSE
HookText db "&Hook",0
UnhookText db "&Unhook",0
template db "%lx",0

.data?
hInstance dd ?
hHook dd ?
.code
start:
    invoke GetModuleHandle,NULL
    mov hInstance,eax
    invoke DialogBoxParam,hInstance,IDD_MAINDLG,NULL,addr DlgFunc,NULL
    invoke ExitProcess,NULL

DlgFunc proc hDlg:DWORD,uMsg:DWORD,wParam:DWORD,lParam:DWORD
    LOCAL hLib:DWORD
    LOCAL buffer[128]:byte
    LOCAL buffer1[128]:byte
    LOCAL rect:RECT
    .if uMsg==WM_CLOSE
        .if HookFlag==TRUE
            invoke UninstallHook
        .endif
        invoke EndDialog,hDlg,NULL
    .elseif uMsg==WM_INITDIALOG
        invoke GetWindowRect,hDlg,addr rect
        invoke SetWindowPos, hDlg, HWND_TOPMOST, rect.left, rect.top, rect.right, rect.bottom, SWP_SHOWWINDOW
    .elseif uMsg==WM_MOUSEHOOK
        invoke GetDlgItemText,hDlg,IDC_HANDLE,addr buffer1,128
        invoke wsprintf,addr buffer,addr template,wParam
        invoke lstrcmpi,addr buffer,addr buffer1
        .if eax!=0
            invoke SetDlgItemText,hDlg,IDC_HANDLE,addr buffer
        .endif
        invoke GetDlgItemText,hDlg,IDC_CLASSNAME,addr buffer1,128
        invoke GetClassName,wParam,addr buffer,128
        invoke lstrcmpi,addr buffer,addr buffer1
        .if eax!=0
            invoke SetDlgItemText,hDlg,IDC_CLASSNAME,addr buffer
        .endif
        invoke GetDlgItemText,hDlg,IDC_WNDPROC,addr buffer1,128
        invoke GetClassLong,wParam,GCL_WNDPROC
        invoke wsprintf,addr buffer,addr template,eax
        invoke lstrcmpi,addr buffer,addr buffer1
        .if eax!=0
            invoke SetDlgItemText,hDlg,IDC_WNDPROC,addr buffer
        .endif
    .elseif uMsg==WM_COMMAND
        .if lParam!=0
            mov eax,wParam
            mov edx,eax
            shr edx,16
            .if dx==BN_CLICKED
                .if ax==IDC_EXIT
                    invoke SendMessage,hDlg,WM_CLOSE,0,0
                .else
                    .if HookFlag==FALSE
                        invoke InstallHook,hDlg
                        .if eax!=NULL
                            mov HookFlag,TRUE
                            invoke SetDlgItemText,hDlg,IDC_HOOK,addr UnhookText
                        .endif
                    .else
                        invoke UninstallHook
                        invoke SetDlgItemText,hDlg,IDC_HOOK,addr HookText
                        mov HookFlag,FALSE
                        invoke SetDlgItemText,hDlg,IDC_CLASSNAME,NULL
                        invoke SetDlgItemText,hDlg,IDC_HANDLE,NULL
                        invoke SetDlgItemText,hDlg,IDC_WNDPROC,NULL
                    .endif
                .endif
            .endif
        .endif
    .else
        mov eax,FALSE
        ret
    .endif
    mov eax,TRUE
    ret
DlgFunc endp

end start

;----------------------------------------------------- This is the source code of the DLL --------------------------------------
.386
.model flat,stdcall
option casemap:none
include \masm32\include\windows.inc
include \masm32\include\kernel32.inc
includelib \masm32\lib\kernel32.lib
include \masm32\include\user32.inc
includelib \masm32\lib\user32.lib

.const
WM_MOUSEHOOK equ WM_USER+6

.data
hInstance dd 0

.data?
hHook dd ?
hWnd dd ?

.code
DllEntry proc hInst:HINSTANCE, reason:DWORD, reserved1:DWORD
    .if reason==DLL_PROCESS_ATTACH
        push hInst
        pop hInstance
    .endif
    mov eax,TRUE
    ret
DllEntry Endp

MouseProc proc nCode:DWORD,wParam:DWORD,lParam:DWORD
    invoke CallNextHookEx,hHook,nCode,wParam,lParam
    mov edx,lParam
    assume edx:PTR MOUSEHOOKSTRUCT
    invoke WindowFromPoint,[edx].pt.x,[edx].pt.y
    invoke PostMessage,hWnd,WM_MOUSEHOOK,eax,0
    assume edx:nothing
    xor eax,eax
    ret
MouseProc endp

InstallHook proc hwnd:DWORD
    push hwnd
    pop hWnd
    invoke SetWindowsHookEx,WH_MOUSE,addr MouseProc,hInstance,NULL
    mov hHook,eax
    ret
InstallHook endp

UninstallHook proc
invoke UnhookWindowsHookEx,hHook
ret
UninstallHook endp

End DllEntry

;---------------------------------------------- This is the makefile of the DLL ----------------------------------------------

NAME=mousehook
$(NAME).dll: $(NAME).obj
Link /SECTION:.bss,S /DLL /DEF:$(NAME).def /SUBSYSTEM:WINDOWS /LIBPATH:c:\masm\lib $(NAME).obj
$(NAME).obj: $(NAME).asm
ml /c /coff /Cp $(NAME).asm

Analysis:

The example will display a dialog box with three edit controls that will be filled with the class name, window handle and the address of the window procedure associated with the window under the mouse cursor. There are two buttons, Hook and Exit. When you press the Hook button, the program hooks the mouse input and the text on the button changes to Unhook. When you move the mouse cursor over a window, the info about that window will be displayed in the main window of the example. When you press Unhook button, the program removes the mouse hook.
The main program uses a dialog box as its main window. It defines a custom message, WM_MOUSEHOOK which will be used between the main program and the hook DLL. When the main program receives this message, wParam contains the handle of the window that the mouse cursor is on. Of course, this is an arbitrary arrangement. I decide to send the handle in wParam for the sake of simplicity. You can choose your own method of communication between the main program and the hook DLL.

                    .if HookFlag==FALSE
                        invoke InstallHook,hDlg
                        .if eax!=NULL
                            mov HookFlag,TRUE
                            invoke SetDlgItemText,hDlg,IDC_HOOK,addr UnhookText
                        .endif

The program maintains a flag, HookFlag, to monitor the state of the hook. It's FALSE if the hook is not installed and TRUE if the hook is installed.
When the user presses Hook button, the program checks if the hook is already installed. If it is not, it call InstallHook function in the hook DLL to install it. Note that we pass the handle of the main dialog as the parameter of the function so the hook DLL can send the WM_MOUSEHOOK messages to the right window i.e. our own.
When the program is loaded, the hook DLL is loaded too. Actually, DLLs are loaded immediately after the program is in memory. The DLL entrypoint function is called before the first instruction in the main program is execute even. So when the main program executes the DLL(s) is/are initialized. We put the following code in the DLL entrypoint function of the hook DLL:

    .if reason==DLL_PROCESS_ATTACH
        push hInst
        pop hInstance
    .endif

The code just saves the instance handle of the hook DLL itself to a global variable named hInstance for use within the InstallHook function. Since the DLL entrypoint function is called before other functions in the DLL are called , hInstance is always valid. We put hInstance in .data section so that this value is kept on per-process basis. Since when the mouse cursor hovers over a window, the hook DLL is mapped into the process. Imagine that there is already a DLL that occupies the intended load address of the hook DLL, the hook DLL would be remapped to another address. The value of hInstance will be updated to those of the new load address. When the user presses Unhook button and then Hook button, SetWindowsHookEx will be called again. However, this time, it will use the new load address as the instance handle which will be wrong because in the example process, the hook DLL's load address hasn't been changed. The hook will be a local one where you can hook only the mouse events that occur in your own window. Hardly desirable.

InstallHook proc hwnd:DWORD
    push hwnd
    pop hWnd
    invoke SetWindowsHookEx,WH_MOUSE,addr MouseProc,hInstance,NULL
    mov hHook,eax
    ret
InstallHook endp

The InstallHook function itself is very simple. It saves the window handle passed as its parameter to a global variable named hWnd for future use. It then calls SetWindowsHookEx to install a mouse hook. The return value of SetWindowsHookEx is stored in a global variable named hHook for use with UnhookWindowsHookEx.
After SetWindowsHookEx is called, the mouse hook is functional. Whenever a mouse event occurs in the system, MouseProc ( your hook procedure) is called.

The first thing it does is to call CallNextHookEx to give the other hooks the chance to process the mouse event. After that, it calls WindowFromPoint function to retrieve the handle of the window at the specified screen coordinate. Note that we use the POINT structure in the MOUSEHOOKSTRUCT structure pointed to by lParam as the current mouse coordinate. After that we send the window handle to the main program via PostMessage with WM_MOUSEHOOK message. One thing you should remember is that: you should not use SendMessage inside the hook procedure, it can cause message deadlock. PostMessage is recommended. The MOUSEHOOKSTRUCT structure is defined below:

MOUSEHOOKSTRUCT STRUCT DWORD
pt            POINT <>
hwnd          DWORD      ?
wHitTestCode DWORD      ?
dwExtraInfo   DWORD      ?
MOUSEHOOKSTRUCT ENDS

pt is the current screen coordinate of the mouse cursor
hwnd is the handle of the window that will receive the mouse message. It's usually the window under the mouse cursor but not always. If a window calls SetCapture, the mouse input will be redirected to that window instead. Because of this reason, I don't use the hwnd member of this structure but choose to call WindowFromPoint instead.
wHitTestCode specifies the hit-test value. The hit-test value gives more information about the current mouse cursor position. It specifies on what part of window the mouse cursor is. For complete list, check your win32 api reference under WM_NCHITTEST message.
dwExtraInfo contains the extra information associated with the message. Normally this value is set by calling mouse_event and retrieved by calling GetMessageExtraInfo.

When the main window receives WM_MOUSEHOOK message, it uses the window handle in wParam to retrieve the information about the window.

    .elseif uMsg==WM_MOUSEHOOK
        invoke GetDlgItemText,hDlg,IDC_HANDLE,addr buffer1,128
        invoke wsprintf,addr buffer,addr template,wParam
        invoke lstrcmpi,addr buffer,addr buffer1
        .if eax!=0
            invoke SetDlgItemText,hDlg,IDC_HANDLE,addr buffer
        .endif
        invoke GetDlgItemText,hDlg,IDC_CLASSNAME,addr buffer1,128
        invoke GetClassName,wParam,addr buffer,128
        invoke lstrcmpi,addr buffer,addr buffer1
        .if eax!=0
            invoke SetDlgItemText,hDlg,IDC_CLASSNAME,addr buffer
        .endif
        invoke GetDlgItemText,hDlg,IDC_WNDPROC,addr buffer1,128
        invoke GetClassLong,wParam,GCL_WNDPROC
        invoke wsprintf,addr buffer,addr template,eax
        invoke lstrcmpi,addr buffer,addr buffer1
        .if eax!=0
            invoke SetDlgItemText,hDlg,IDC_WNDPROC,addr buffer
        .endif

To avoid flickers, we check the text already in the edit controls and the text we will put into them if they are identical. If they are, we skip them.
We retrieve the class name by calling GetClassName, the address of the window procedure by calling GetClassLong with GCL_WNDPROC and then format them into strings and put them into the appropriate edit controls.

                        invoke UninstallHook
                        invoke SetDlgItemText,hDlg,IDC_HOOK,addr HookText
                        mov HookFlag,FALSE
                        invoke SetDlgItemText,hDlg,IDC_CLASSNAME,NULL
                        invoke SetDlgItemText,hDlg,IDC_HANDLE,NULL
                        invoke SetDlgItemText,hDlg,IDC_WNDPROC,NULL

When the user presses Unhook button, the program calls UninstallHook function in the hook DLL. UninstallHook just calls UnhookWindowsHookEx. After that, it changes the text of the button back to "Hook", HookFlag to FALSE and clears the content of the edit controls.
Note the linker switch in the makefile.

Link /SECTION:.bss,S /DLL /DEF:$(NAME).def /SUBSYSTEM:WINDOWS

It specifies .bss section as a shared section to make all processes share the same uninitialized data section of the hook DLL. Without this switch, your hook DLL will not function correctly.

[Iczelion's Win32 Assembly Homepage]