Module 2 — User Mode and Syscalls

Module 2 — User Mode and Syscalls

Summary

Applications rarely touch the kernel directly. They call Win32 APIs, which call NT APIs in ntdll.dll, which execute syscalls. EDR hooks and telemetry often sit on this path.

Path: Module 1 · Next: Module 3


0. Purpose

Connect API names in logs to kernel actions. When you see VirtualAlloc, NtOpenProcess, or syscall, you should know they are different layers of the same story.


1. ELI5

You want a glass of water (allocate memory, open a file):

  1. You ask the hotel front desk (kernel32.dll) in plain language
  2. The desk calls internal operations (ntdll.dll) with the building's formal procedure names (Nt*)
  3. Operations rings the basement intercom (syscall) — only then does the manager act

EDR is like security cameras at the desk and the intercom.


2. Deep breakdown

sequenceDiagram
  participant App as Application
  participant K32 as kernel32.dll
  participant NTD as ntdll.dll
  participant Kern as ntoskrnl.exe

  App->>K32: VirtualAlloc(...)
  K32->>NTD: NtAllocateVirtualMemory(...)
  NTD->>Kern: syscall (SSN)
  Kern-->>NTD: NTSTATUS
  NTD-->>K32: NTSTATUS
  K32-->>App: LPVOID
Layer DLL / surface Naming Documented for devs?
Win32 kernel32, advapi32, user32, … CreateFile, OpenProcess Yes — Microsoft Learn Win32
Native / NT ntdll.dll NtCreateFile, NtOpenProcess Partial — some undocumented
Syscall CPU instruction System Service Number (SSN) Kernel implementation

Critical fact: the syscall executes in the kernel; ntdll holds the user-mode stub. See Syscalls Flow.

Win32 (example) NT API (typical) What kernel does
VirtualAlloc NtAllocateVirtualMemory Commit/reserve process memory
CreateFile NtCreateFile File object + handle
OpenProcess NtOpenProcess Handle to another process

WOW64 note: 32-bit processes on 64-bit Windows use SysWOW64\ntdll.dll — separate syscall path. Always note process bitness in investigations.

System Service Numbers (SSN)

Each Nt* function maps to a syscall number that can change between Windows builds. Attackers may invoke syscalls directly (bypassing hooked ntdll stubs); defenders still see kernel-side effects via callbacks, ETW, and minifilters.

Concept User-mode view Kernel view
API call OpenProcess Handle grant in object manager
NT call NtOpenProcess Security checks + handle table
Syscall SSN in eax / r10 KiSystemCall64 dispatch

Lab .114 gateway DLL versions (from baseline):

DLL Version
ntdll.dll 10.0.26100.8972
kernel32.dll 10.0.26100.8972
kernelbase.dll 10.0.26100.8972

Where telemetry attaches

flowchart LR
  APP[App] --> K32[kernel32]
  K32 --> NTD[ntdll]
  NTD -->|hooked stub| EDR[User-mode EDR hook]
  NTD -->|syscall| KERN[kernel callbacks]
  KERN --> FLT[minifilter / WFP]

Security engineer lens

Detection / log source Layer usually observed
Sysmon Event 10 (ProcessAccess) Often triggered around OpenProcess / NtOpenProcess
API hooking (user mode) ntdll, kernel32, or DLLs above
Direct syscall / "unhooked" tradecraft Skips hooked ntdll stubs — still hits kernel
Kernel callbacks Below all user DLLs

Investigation habit: when a tool claims to "bypass API monitoring," ask: did it bypass user-mode hooks only, or kernel telemetry too?


Lab checkpoint — 192.168.50.114

# Confirm gateway DLLs exist and versions
'ntdll.dll','kernel32.dll','kernelbase.dll' | ForEach-Object {
  $p = Join-Path $env:windir "System32\$_"
  Get-Item $p | Select-Object @{N='DLL';E={$_}}, @{N='KB';E={[math]::Round($_.Length/1KB,1)}},
    @{N='Ver';E={$_.VersionInfo.FileVersion}}
}

# See which DLLs PowerShell loaded (runtime — not file edit)
powershell -NoProfile -Command "'ping'" | Out-Null
Get-Process -Id $PID -Module |
  Where-Object ModuleName -match 'ntdll|kernel32|amsi' |
  Select-Object ModuleName, FileName

Optional (admin): Sysinternals Process Explorer → lower pane → DLLs for powershell.exe — find ntdll.dll path and version.


Record in notebook


Module 1 · Module 3 →