Previously we discussed why it isn’t required to inject the DLL containing the callback to every application you need to set a hook on. In this post, we are going to see if it’s possible to create a hook or a global Hook with SetWindowsHookEx function without using any sort of DLL.
There is this cool trick in which you can use some known Dll to grab their handle and use that to legitimately install the hook with this function. Here is the code for doing that

HMODULE dll_handle = GetModuleHandle(L"user32.dll");
windows_exhook = SetWindowsHookExA(WH_GETMESSAGE, CallWndProcHook,
							dll_handle, tid); //0: System wide event
DWORD error = GetLastError();
if( windows_exhook==NULL )
{
	qDebug() << "Failed to set the hook. Error code: "
			 << error;
}
qDebug() << "hook was successfully installed="
         << windows_exhook << dll_handle
         << error << tid;

Although this code runs successfully and installs the hook if you install this globally by setting the thread id to 0, it will Instantly crash your whole explorer processor.
If you try that on a single application it also crashes the application.

I tried and spend some time to figure this out and it turns out that the callback will never get executed. I tried to simplify the callback do some sanity tests and it looks like the hook is installed without any problems and sometimes the callback called and most of the time it won’t but at the end in all test cases the targeted application were always crashes. Here is the simplified callback function that I used:

LRESULT CALLBACK CallWndProcHook(int nCode, WPARAM wParam,
                                 LPARAM lParam)
{
    // Call the next hook in the chain
    // The first argument is ignored so any value will work
    return CallNextHookEx(windows_exhook, nCode,
                          wParam, lParam);
}

I searched a bit about this and there were two comments around this issue

1. The first was that this is because you are skipping the phase which is putting your callback inside a Dll [link]
2. The second one is it’s because of some sort of aggressive security application that detects that this is not a legitimate behavior so it will instantly kill the effected affected application in order to prevent any data hijacking [link]

Conclusion

Although at the beginning it seems that you can get away using SetWindowsHookEx function without a Dll currently at least up to what I know there is no possible way to use SetWindowsHookEx function without having the callback inside another Dll

So exciting that even after 13 years of introducing to Qt by my great teacher, Mani Monajjemi there’s still a lot to learn.

I was trying to use WinRT with Qt today and now after so long time sticking to MinGW, I’m switching to MSVC. Here are the 4 reasons

1. MinGW is opensource but deep down if you are in Win32, their compiler always offers better API compatibility
2. WinRT is available only on MSVC
3. If you ever get around to some dll that simply doesn’t work with your project, it’s because MinGW and MSVC ABI are not the same. and probably that DLL was compiled by MSVC not MinGW. Same OS and still a different ABI, sounds too Windowsy to me
4. Because you are on Windows, show some support to the closed-source community!