Reverse Engineer a Simple Custom File Format

Reverse Engineering Unknown File Formats

When you encounter an undocumented binary file, you must deduce its structure without specifications. This skill applies to malware analysis (custom protocols), game modding (proprietary assets), and data recovery (legacy formats).

Deduction Techniques

Magic bytes at offset 0 identify the type. Even custom formats often have a version identifier there.

Structure fields reveal themselves through patterns:

• A 4-byte value matching file size minus header suggests a length field
• Two values that look like width and height suggest image dimensions
• Offsets into the file are usually smaller than total file size

For example, if bytes at offset 8 show 34 12 and the expected value is 0x1234, the file is little-endian.

Hex editing workflow: dump the first 256 bytes, identify magic, guess field sizes, validate by checking that offsets point to plausible data.

For This Exercise

You will document techniques for identifying magic bytes, structure fields, and endianness in unknown formats. This is the same process Wireshark dissectors and game mod tools use when no specification exists.

Validating Field Guesses

After guessing field layout, check that offsets point to plausible data: strings should be printable, lengths should not exceed file size, and checksums should validate if present. Change one field at a time in a hex editor and observe whether the target program still parses the file. This iterative hypothesis-testing approach is how modding communities reverse game asset formats without official documentation.