HTB — reversing a custom VM in 3 hours

The challenge binary implemented a 16-instruction stack VM in ~800 lines of C. The flag was encrypted and stored as VM bytecode. My job: reverse the ISA, write an emulator in Python, then work backwards from the encrypted output to the key.

recon

❯ file vm-ware
vm-ware: ELF 64-bit LSB executable, x86-64, statically linked

❯ checksec --file vm-ware
Arch:     amd64-64-little
RELRO:    Full RELRO
Stack:    Canary found
NX:       NX enabled
PIE:      No PIE

Static, no PIE — reversing is straightforward, no ASLR to fight. Opened in Ghidra, found dispatch_loop() immediately from the switch statement shape.

the ISA

After an hour with Ghidra I had the full opcode table:

the emulator

class VM:
    def __init__(self, bytecode):
        self.code = bytecode
        self.stack = []
        self.mem = bytearray(4096)
        self.regs = [0] * 8
        self.ip = 0

    def step(self):
        op = self.code[self.ip]; self.ip += 1
        if   op == 0x01: self.stack.append(self.code[self.ip]); self.ip += 1
        elif op == 0x04: self.regs[0] ^= self.regs[1]
        elif op == 0x0A: return False
        # ... rest of opcodes
        return True

Running the emulator against the bytecode and tracing every XOR operation gave me the key schedule. The encryption was a simple rolling-XOR with a 4-byte key. Brute-force over 2^32 possible keys took 8 seconds.

flag

HTB{vm_b4by_st3ps_t0_r3v3rs1ng}

Total time: 2h 47m. The hardest part was convincing myself the key was only 4 bytes — the VM had room for a longer one and I spent 30 minutes looking for a second key derivation step that wasn’t there.