Custom VM and bytecode reverse engineering playbook. Use when CTF challenges or protected software implement custom virtual machines with proprietary bytecode, dispatcher loops, or maze-style challenges.

SKILL: VM & Bytecode Reverse Engineering — Expert Analysis Playbook AI LOAD INSTRUCTION : Expert techniques for reversing custom virtual machines and bytecode interpreters. Covers dispatcher identification, opcode mapping, custom ISA reconstruction, disassembler/decompiler writing, maze challenges, and real world VM protector analysis. Base models often fail to recognize the fetch decode execute pattern or attempt to analyze VM bytecode as native code. 0. RELATED ROUTING [code obfuscation deobfuscation](../code obfuscation deobfuscation/SKILL.md) when the VM is a commercial protector (VMProtect/Themida) [symbolic execution tools](../symbolic execution tools/SKILL.md) when using angr to solve VM based challenges [anti debugging techniques](../anti debugging techniques/SKILL.md) when the VM includes anti debug checks Quick identification Binary Pattern Likely VM Type Start With while(1) { switch(bytecode[pc]) } Switch based dispatcher Map each case to an operation Indirect jump via table jmp [table + opcode 8] Table based dispatcher Dump jump table, analyze handlers Nested if else chain on byte value If chain dispatcher Same as switch, just different syntax Stack push/pop dominant operations Stack based VM Identify push, pop, arithmetic ops reg[X] = ... array operations Register based VM Map register indices to operations 2D grid + direction input Maze challenge Extract grid, apply BFS/DFS 1. CUSTOM VM IDENTIFICATION 1.1 Structural Indicators 1.2 IDA/Ghidra Signatures Switch dispatcher (most common in CTF): Table dispatcher (more optimized): 2. ANALYSIS METHODOLOGY Step 1: Find the Dispatcher Look for: Large switch statement (many cases) in a loop Array of function pointers indexed by a byte from a data buffer Single function with high cyclomatic complexity Cross references to a data buffer read byte by byte Step 2: Map Opcodes to Operations For each case/handler, determine: Property How to Identify Opcode value Case number or table index Operation type Register/stack modifications Operand count How many bytes consumed after opcode Operand type Immediate value, register index, or memory address Side effects Output, memory write, flag modification Step 3: Extract Bytecode Program Step 4: Write Custom Disassembler Step 5: Analyze Disassembled Program With the custom disassembly, apply standard reverse engineering: Identify input reading (read opcode) Trace data flow from input to comparison Determine success/failure conditions Extract the check logic (often XOR/ADD transformations of input compared against constants) 3. COMMON VM PATTERNS IN CTF 3.1 Stack Based VM Operations work on a stack (like JVM or Python bytecode). Opcode Operation Stack Effect PUSH imm Push immediate value [...] → [..., imm] POP Discard top [..., a] → [...] ADD Add top two [..., a, b] → [..., a+b] SUB Subtract [..., a, b] → [..., a b] MUL Multiply [..., a, b] → [..., a b] XOR Bitwise XOR [..., a, b] → [..., a^b] CMP Compare [..., a, b] → [..., (a==b)] JMP addr Unconditional jump no change JZ addr Jump if top is zero [..., a] → [...] PRINT Output top as char [..., a] → [...] READ Read char to stack [...] → [..., input] HALT Stop execution 3.2 Register Based VM Operations use register indices (like x86, ARM). Opcode Format Operation MOV r, imm 0x01 RR II II reg[R] = imm16 MOV r1, r2 0x02 R1 R2 reg[R1] = reg[R2] ADD r1, r2 0x03 R1 R2 reg[R1] += reg[R2] SUB r1, r2 0x04 R1 R2 reg[R1] = reg[R2] XOR r1, r2 0x05 R1 R2 reg[R1] ^= reg[R2] CMP r1, r2 0x06 R1 R2 flags = compare(r1, r2) JMP addr 0x07 AA AA pc = addr JE addr 0x08 AA AA if equal: pc = addr LOAD r, [addr] 0x09 RR AA reg[R] = mem[addr] STORE [addr], r 0x0A AA RR mem[addr] = reg[R] SYSCALL 0x0B I/O operation based on reg[0] HALT 0xFF stop 3.3 Brainfuck like / Esoteric VMs BF Command VM Equivalent Description INC ptr Move data pointer right < DEC ptr Move data pointer left + INC [ptr] Increment byte at pointer DEC [ptr] Decrement byte at pointer . OUTPUT [ptr] Output byte at pointer , INPUT [ptr] Input byte to pointer [ JZ forward Jump past ] if byte is zero ] JNZ back Jump back to [ if byte is nonzero 4. MAZE CHALLENGES 4.1 Identification Binary reads directional input (WASD, arrow keys, UDLR) 2D array in data section (walls, paths, start, end) Position tracking with x,y coordinates Win condition at specific coordinates 4.2 Map Extraction 4.3 Automated Solving 4.4 Direction Encoding Different challenges encode directions differently: Encoding Up Down Left Right WASD W S A D UDLR U D L R Arrow keys ↑ (0x48) ↓ (0x50) ← (0x4B) → (0x4D) Numbers 1 2 3 4 Hex opcodes 0x01 0x02 0x03 0x04 5. REAL WORLD VM PROTECTORS 5.1 VMProtect Analysis Approach 5.2 Tigress Obfuscator Academic VM obfuscator with configurable protection layers. Feature Approach Single dispatch VM Standard handler extraction Split handlers Handlers spread across multiple functions Nested VMs Outer VM handler invokes inner VM Encrypted bytecode Dynamic decryption before each fetch Polymorphic handlers Different code for same operation on each build 5.3 Common VM Protector Patterns Protector Dispatcher Style Difficulty VMProtect Table + opaque predicates High Themida (Code Virtualizer) CISC like, large handler set High Tigress Configurable, academic Medium High Custom CTF VM Simple switch Low Medium Movfuscator All mov computation Medium 6. TOOLS Tool Purpose Usage IDA Pro Identify dispatcher, reverse handlers F5 decompile, xref analysis Ghidra Free alternative with Sleigh processor modules Write custom processor for VM ISA angr Symbolic execution through VM Treat entire VM as constraint system Pin / DynamoRIO Dynamic instrumentation for tracing Record opcode handler execution sequence REVEN Full system trace recording Replay and analyze VM execution Unicorn Emulate VM execution Fast handler emulation Miasm IR based analysis Lift VM handlers to IR for analysis Custom Python Write disassembler/decompiler Per challenge custom tooling Ghidra Sleigh Processor Module For recurring VM architectures, write a Sleigh processor specification: 7. DECISION TREE 8. CTF SOLVING WORKFLOW

vm-and-bytecode-reverse