Module 2 — User Mode and Syscalls
Module 2 — User Mode and Syscalls
Applications rarely touch the kernel directly. They call Win32 APIs, which call NT APIs in ntdll.dll, which execute syscalls. EDR hooks and telemetry often sit on this path.
Path: Module 1 · Next: Module 3
0. Purpose
Connect API names in logs to kernel actions. When you see VirtualAlloc, NtOpenProcess, or syscall, you should know they are different layers of the same story.
1. ELI5
You want a glass of water (allocate memory, open a file):
- You ask the hotel front desk (
kernel32.dll) in plain language - The desk calls internal operations (
ntdll.dll) with the building's formal procedure names (Nt*) - Operations rings the basement intercom (
syscall) — only then does the manager act
EDR is like security cameras at the desk and the intercom.
2. Deep breakdown
sequenceDiagram participant App as Application participant K32 as kernel32.dll participant NTD as ntdll.dll participant Kern as ntoskrnl.exe App->>K32: VirtualAlloc(...) K32->>NTD: NtAllocateVirtualMemory(...) NTD->>Kern: syscall (SSN) Kern-->>NTD: NTSTATUS NTD-->>K32: NTSTATUS K32-->>App: LPVOID
| Layer | DLL / surface | Naming | Documented for devs? |
|---|---|---|---|
| Win32 | kernel32, advapi32, user32, … |
CreateFile, OpenProcess |
Yes — Microsoft Learn Win32 |
| Native / NT | ntdll.dll |
NtCreateFile, NtOpenProcess |
Partial — some undocumented |
| Syscall | CPU instruction | System Service Number (SSN) | Kernel implementation |
Critical fact: the syscall executes in the kernel; ntdll holds the user-mode stub. See Syscalls Flow.
| Win32 (example) | NT API (typical) | What kernel does |
|---|---|---|
VirtualAlloc |
NtAllocateVirtualMemory |
Commit/reserve process memory |
CreateFile |
NtCreateFile |
File object + handle |
OpenProcess |
NtOpenProcess |
Handle to another process |
WOW64 note: 32-bit processes on 64-bit Windows use SysWOW64\ntdll.dll — separate syscall path. Always note process bitness in investigations.
System Service Numbers (SSN)
Each Nt* function maps to a syscall number that can change between Windows builds. Attackers may invoke syscalls directly (bypassing hooked ntdll stubs); defenders still see kernel-side effects via callbacks, ETW, and minifilters.
| Concept | User-mode view | Kernel view |
|---|---|---|
| API call | OpenProcess |
Handle grant in object manager |
| NT call | NtOpenProcess |
Security checks + handle table |
| Syscall | SSN in eax / r10 |
KiSystemCall64 dispatch |
Lab .114 gateway DLL versions (from baseline):
| DLL | Version |
|---|---|
ntdll.dll |
10.0.26100.8972 |
kernel32.dll |
10.0.26100.8972 |
kernelbase.dll |
10.0.26100.8972 |
Where telemetry attaches
flowchart LR APP[App] --> K32[kernel32] K32 --> NTD[ntdll] NTD -->|hooked stub| EDR[User-mode EDR hook] NTD -->|syscall| KERN[kernel callbacks] KERN --> FLT[minifilter / WFP]
Security engineer lens
| Detection / log source | Layer usually observed |
|---|---|
| Sysmon Event 10 (ProcessAccess) | Often triggered around OpenProcess / NtOpenProcess |
| API hooking (user mode) | ntdll, kernel32, or DLLs above |
| Direct syscall / "unhooked" tradecraft | Skips hooked ntdll stubs — still hits kernel |
| Kernel callbacks | Below all user DLLs |
Investigation habit: when a tool claims to "bypass API monitoring," ask: did it bypass user-mode hooks only, or kernel telemetry too?
Lab checkpoint — 192.168.50.114
# Confirm gateway DLLs exist and versions
'ntdll.dll','kernel32.dll','kernelbase.dll' | ForEach-Object {
$p = Join-Path $env:windir "System32\$_"
Get-Item $p | Select-Object @{N='DLL';E={$_}}, @{N='KB';E={[math]::Round($_.Length/1KB,1)}},
@{N='Ver';E={$_.VersionInfo.FileVersion}}
}
# See which DLLs PowerShell loaded (runtime — not file edit)
powershell -NoProfile -Command "'ping'" | Out-Null
Get-Process -Id $PID -Module |
Where-Object ModuleName -match 'ntdll|kernel32|amsi' |
Select-Object ModuleName, FileName
Optional (admin): Sysinternals Process Explorer → lower pane → DLLs for powershell.exe — find ntdll.dll path and version.
Record in notebook
Navigation
← Module 1 · Module 3 →