Behind the Scenes — HackTheBox

Description

"After struggling to secure our secret strings for a long time, we finally figured out the solution to our problem: Make decompilation harder. It should now be impossible to figure out how our programs work!"

The behindthescenes binary is an ELF 64-bit file that uses an anti-decompilation technique to protect the password verification logic.

Analysis

Initial Analysis

$ file behindthescenes
behindthescenes: ELF 64-bit LSB pie executable, x86-64, dynamically linked, not stripped

$ strings behindthescenes
./challenge <password>
> HTB{%s}

The binary takes a password as an argument and outputs the flag in the format HTB{password}.

Anti-Decompilation: ud2 + SIGILL

The key protection mechanism uses the ud2 instruction (undefined instruction, opcode 0f 0b):

1. Handler registration: At the start of main, a signal handler for SIGILL is registered (segill_sigaction)
2. Scattered ud2: The code contains scattered ud2 instructions that trigger an "illegal instruction" exception
3. Handler skips: The signal handler intercepts SIGILL and increments the instruction pointer by 2 bytes (the size of ud2)
4. Decompiler breaks: Static analyzers (IDA, Ghidra) don't understand that ud2 will be skipped and incorrectly build the CFG

; Typical pattern in the code:
mov    rdi, [rbp-0x10]
ud2                      ; <- decompiler thinks this crashes
add    rdi, 0x3          ; <- this code is not analyzed

Password Verification Logic

The logic extracted from the disassembler (objdump -d):

1. Check argc == 2 (argument required)
2. Check password length == 12 characters
3. Compare password in chunks using strncmp:

...