Virtually Mad — HackTheBox

Description

Your friend loves to make pretty odd programs. This time you are given a special "machine" and you have to crack the correct code.

Given a stripped ELF 64-bit PIE executable (virtually.mad) that implements a custom virtual machine. The program takes a hex string as input, interprets it as VM opcodes, executes them, and checks the final register state.

Analysis

Initial Reconnaissance

$ file virtually.mad
virtually.mad: ELF 64-bit LSB pie executable, x86-64, stripped

$ strings virtually.mad
Give me code to execute:
Invalid code.
Executing %d opcodes.
Skipping opcode #%d, value too high (0x%x).
This is the right answer! Validate the challenge with HTB{%s}

Key strings immediately indicate:

• Input of "code" for execution
• Opcode validation (checking for values that are too high)
• Answer verification with flag output

VM Architecture

Decompilation in Ghidra revealed a custom virtual machine with the following architecture:

Registers:

• 4 general-purpose registers: a (0), b (1), c (2), d (3) — 32-bit each
• Flags register flags — set by the CMP instruction

Instruction Set (5 instructions):

Code	Mnemonic	Description
0x01	MOV	Load value/register into register
0x02	ADD	Addition
0x03	SUB	Subtraction
0x04	CMP	Compare (sets flags)
0x05	EXIT	Terminate execution

Opcode Format

Each opcode is a 32-bit value, entered as an 8-character hex string. Bit layout:

31-24  23-20  19-16  15-12  11-0
 inst   type   dst    mode   operand

...