Virtually Mad — HackTheBox

Description

Your friend loves to make pretty odd programs. This time you are given a special "machine" and you have to crack the correct code.

Given a stripped ELF 64-bit PIE executable (virtually.mad) that implements a custom virtual machine. The program takes a hex string as input, interprets it as VM opcodes, executes them, and checks the final register state.

Analysis

Initial Reconnaissance

$ file virtually.mad
virtually.mad: ELF 64-bit LSB pie executable, x86-64, stripped

$ strings virtually.mad
Give me code to execute:
Invalid code.
Executing %d opcodes.
Skipping opcode #%d, value too high (0x%x).
This is the right answer! Validate the challenge with HTB{%s}

Key strings immediately indicate:

• Input of "code" for execution
• Opcode validation (checking for values that are too high)
• Answer verification with flag output

VM Architecture

Decompilation in Ghidra revealed a custom virtual machine with the following architecture:

Registers:

• 4 general-purpose registers: a (0), b (1), c (2), d (3) — 32-bit each
• Flags register flags — set by the CMP instruction

Instruction Set (5 instructions):

Code	Mnemonic	Description
0x01	MOV	Load value/register into register
0x02	ADD	Addition
0x03	SUB	Subtraction
0x04	CMP	Compare (sets flags)
0x05	EXIT	Terminate execution

Opcode Format

Each opcode is a 32-bit value, entered as an 8-character hex string. Bit layout:

31-24  23-20  19-16  15-12  11-0
 inst   type   dst    mode   operand

Bits	Field	Description
31-24	instruction	Instruction index (1=MOV, 2=ADD, 3=SUB, 4=CMP, 5=EXIT)
23-20	type_check	Must equal 1
19-16	dst_reg	Destination register (0=a, 1=b, 2=c, 3=d)
15-12	mode	0=immediate (value), 1=register (source register)
11-0	operand	Immediate value or source register index (bits 11-8)

Input Validation

• Input string is parsed as a sequence of 8-character hex blocks
• Each block is converted to a 32-bit number
• Lower 12 bits of each opcode must be ≤ 0x100
• Exactly 5 opcodes (40 hex characters input)

Required Final State

After executing all 5 opcodes, the VM checks the registers:

a     = 0x200
b     = 0xFFFFFFFF  (-1 in signed 32-bit)
c     = 0xFFFFFFFF  (-1 in signed 32-bit)
d     = 0x0
flags = 0x10000000

If all values match — the flag is output (the input code itself).

Solution

Constructing Opcodes

Initial state: all registers = 0, flags = 0.

Need to bring registers to the required state in 5 instructions:

Opcode 1: 02100100 — ADD a, 0x100

inst=0x02 (ADD), type=1, dst=0 (a), mode=0 (imm), operand=0x100
a = 0 + 0x100 = 0x100

Opcode 2: 02100100 — ADD a, 0x100

inst=0x02 (ADD), type=1, dst=0 (a), mode=0 (imm), operand=0x100
a = 0x100 + 0x100 = 0x200 ✓

Opcode 3: 03110001 — SUB b, 1

inst=0x03 (SUB), type=1, dst=1 (b), mode=0 (imm), operand=0x001
b = 0 - 1 = 0xFFFFFFFF ✓

Opcode 4: 01121100 — MOV c, reg_b

inst=0x01 (MOV), type=1, dst=2 (c), mode=1 (reg), operand=0x100 (src_reg = bits 11-8 = 1 = b)
c = b = 0xFFFFFFFF ✓

Opcode 5: 04130000 — CMP d, 0

inst=0x04 (CMP), type=1, dst=3 (d), mode=0 (imm), operand=0x000
d == 0 → equal → flags = 0x10000000 ✓

Why ADD Twice Instead of MOV 0x200?

Constraint: lower 12 bits (operand) ≤ 0x100. The value 0x200 doesn't fit in a 12-bit immediate, so two additions of 0x100 are needed.

Verification

$ echo "0210010002100100031100010112110004130000" | ./virtually.mad
Give me code to execute: Executing 5 opcodes.
=====
a: 0x200
b: 0xffffffff
c: 0xffffffff
d: 0x0
flags: 0x10000000
=====
This is the right answer! Validate the challenge with HTB{0210010002100100031100010112110004130000}