5. Syscalls Flow

Windows system calls

What is a system call?

A syscall (system call) is a low-level function that allows a user-mode application to request services from the Windows Kernel (ntoskrnl.exe). It is the bridge between user mode and kernel mode, enabling programs to interact with the hardware and OS resources.

Syscalls are the core mechanism used when a WinAPI function (e.g., VirtualAlloc, CreateFile) needs to perform an operating system-level action.

Syscalls exist in the Windows Kernel, not in ntdll.dll.
ntdll.dll only provides the user-mode interface for calling syscalls.

graph TD
  A[User Application] -->|Calls WinAPI _e.g., VirtualAlloc_| B[WinAPI DLLs _kernel32.dll, user32.dll_]
  B -->|Calls NTAPI _e.g., NtAllocateVirtualMemory_| C[NTDLL.DLL]
  C -->|Triggers Syscall| D[Syscall Interface]
  D -->|Switches to Kernel Mode| E[Windows Kernel _ntoskrnl.exe_]
  E -->|Executes System Operation| F[Hardware & OS Components]
  F -->|Returns Data| E
  E -->|Returns to User Mode| D
  D -->|Passes Data to NTAPI| C
  C -->|Passes Data to WinAPI| B
  B -->|Returns Result to Application| A

Explanation of Flow

User Application Calls WinAPI
- The program calls a standard function (e.g., VirtualAlloc) from kernel32.dll or user32.dll.
WinAPI Calls NTAPI (ntdll.dll)
- The WinAPI function internally calls an NTAPI function (e.g., NtAllocateVirtualMemory).
NTAPI Calls a Syscall
- ntdll.dll executes the syscall instruction, transferring execution to the Windows Kernel.
The syscall interface switches from user mode to kernel mode (ring 3 → ring 0)
The Windows Kernel (ntoskrnl.exe) processes the request and interacts with hardware or system components. The kernel handles the request using SSDT (System Service Descriptor Table).
Result is Returned
- The kernel completes the task and returns the result through the syscall interface back to ntdll.dll.
- Finally, the result is returned to the application.

3. Example of a Syscall in Action

Example Syscall Mapping

WinAPI Function	NTAPI Function (ntdll.dll)	Syscall (SSDT)
`VirtualAlloc`	`NtAllocateVirtualMemory`	`NtAllocateVirtualMemory`
`ReadFile`	`NtReadFile`	`NtReadFile`
`CreateProcess`	`NtCreateUserProcess`	`NtCreateUserProcess`
`OpenProcess`	`NtOpenProcess`	`NtOpenProcess`
`WriteProcessMemory`	`NtWriteVirtualMemory`	`NtWriteVirtualMemory`

Many native-service entry points in ntdll.dll are user-mode stubs that place a build-specific system-service number in the expected register and execute the architecture's transition instruction.
The kernel dispatches that number to a native system-service routine. Symbols may use familiar Nt/Zw names, but the stable boundary is the service contract and dispatch number for that build, not a promise that two independently callable functions with identical names exist at both layers.
Some ntdll.dll exports are user-mode runtime helpers and do not transition to the kernel at all.

graph TD
  A[User Application] -->|Calls VirtualAlloc| B[WinAPI _kernel32.dll_]
  B -->|Calls NtAllocateVirtualMemory| C[NTAPI _ntdll.dll_]
  C -->|Triggers Syscall Instruction| D[Windows Kernel _ntoskrnl.exe_]
  D -->|Executes Kernel Function| E[Memory Allocated]
  E -->|Returns Result| A

2. User-mode stub and kernel service

Layer	Function	Purpose
WinAPI	`VirtualAlloc`	High-level API for memory allocation.
NTAPI	`NtAllocateVirtualMemory`	Low-level API in `ntdll.dll` that wraps a syscall.
System service	Build-specific service selected by the dispatch number	Kernel-mode implementation of the requested operation.

The ntdll.dll entry point supplies the user-mode ABI and transition stub. The
kernel dispatcher validates the user-mode boundary and invokes the matching
service implementation. System-service numbers are not a stable public ABI and
can change between Windows builds.

3. NTAPI vs. Syscall – Key Differences

Aspect	NTAPI (ntdll.dll)	Syscall (Kernel Mode)
Location	User mode (`ntdll.dll`)	Kernel mode (`ntoskrnl.exe`)
Execution Mode	Runs in Ring 3	Runs in Ring 0
How It Works	Calls a syscall instruction (`syscall`, `sysenter`)	Executes the requested system operation
Security Hooks?	Can be hooked by AV/EDR	Harder to hook directly
Performance	Includes the required user-to-kernel transition	Executes after dispatch in kernel mode

4. How to Manually Call a Syscall directly(Without WinAPI)

More info here: OW64 Windows Tradecraft — direct syscalls

Example: Calling a Syscall in Assembly

section .text
global _start

_start:
    mov eax, 0x18  ; Syscall number for NtAllocateVirtualMemory
    mov edx, 0x1234  ; Some parameters
    int 0x2E        ; Trigger syscall (Windows XP and older)

⚠️ Modern Windows versions no longer use int 0x2E but instead use syscall or sysenter.

For x64

section .text
global _start

_start:
    mov r10, rcx               ; Syscall convention requires RCX → R10
    mov eax, 0x18              ; Syscall number for NtAllocateVirtualMemory (example, may change)
    syscall                    ; Call the Windows Kernel
    ret                        ; Return to caller