Windows API C Calling Convention

Windows API was built more than decade, and most of them using stdcall calling convention. What actually is calling convention and what kind of calling conventions avaiable here?

Calling convention will define how the arguments and return value being passed to and from a function being called.

More info:

• https://learn.microsoft.com/en-us/cpp/cpp/calling-conventions?view=msvc-170
• https://en.wikipedia.org/wiki/X86_calling_conventions

Most familiar calling convention: cdecl, stdcall, thiscall, fastcall, and pascal.

CDECL

This calling convention is default C, support variable arguments.

Caller: Cdecl will PUSH arguments from the last to the first, then do CALL which actually doing PUSH return address before JMP to function. Upon return from function the caller must clean the stack.

Callee: Won’t do stack cleaning and return using RET.

Return value will also EAX register.

Also known as caller-cleaned. This way passing more arguments doesn’t matter.

The disadvantage of cdecl is multiple call to the function will increase code size because additional stack cleaning instructions.

stdcall

Monst Windows API Function, Callback Function, and COM Interface Method using stdcall.

Caller: Stdcall will PUSH arguments from the last (arg-N) to the first (arg-0), then do CALL which actually doing PUSH return address before JMP to function.

Calee: Stdcall upon returning will pop the function arguments using RET n

Return value alsa using EAX register. The variable arguments is not available as opposed with stdcall. It is also known as callee-cleanned.

This way caller won’t need to clean the stack and will produce smaller codes when multiple calls to Windows API being made becuse reduce of cleaning instruction on each call.

thiscall

This call usually for C++ method function and similar with stdcall.

The difference is the caller store the this pointer as first argument (arg-0) implicitly using ECX register.

FASTCALL

This calling convention also similar with stdcall, the difference the first argument stored in ECX register the second argument using EDX register.

pascal

This calling convention being used on Win16 era and similar with stdcall as callee-cleaned. The difference is the order of argument being stored start from first argument (arg-0) to last argument (arg-n).