HPI / UFO / CCX / GP3 — HAPI Archive Files
HAPI ("HPI") is the encrypted, compressed container Cavedog used to ship every game asset. The same on-disk layout is reused with different extensions:
Extension Purpose TA 3.1c mount order .gp3The 3.1 patch overlay ( rev31.gp3)Mounted first; other *.gp3are ignored.ccxCore Contingency / Battle Tactics data ( btdata.ccx,ccdata.ccx)After rev31.gp3.ufoUnit/feature add-ons ( AFark.ufo)After every .ccx; no count limit.hpiBase game archives ( totala1.hpi,tactics1.hpi)Last; only the first ten that open All four are byte-identical formats — only the mount order differs. The game folds them into a single virtual filesystem at boot, and the first archive mounted that holds a path wins.
Important
Two on-disk dialects. Total Annihilation writes HPI v1
(Version == 0x00010000), the encrypted multi-chunk layout this page
documents top-to-bottom. Total Annihilation: Kingdoms writes
HPI v2 (Version == 0x00020000) — a different, simpler layout
with no XOR cipher and a single SQSH chunk per file or directory
block. The v2 specifics live in their own section near the bottom of
this page: TA: Kingdoms HPI v2. TA 3.1c mounts
v1 archives only.
Load-order layering
TA 3.1c scans only the top level of its game directory (archives in subfolders such as
Backup/are never mounted) and mounts, in this order:
rev31.gp3;- every
*.ccx;- every
*.ufo;- the first ten
*.hpithat open;- the
*.hpifiles at the top level of the game CD, with no count limit.Within each group the game takes names in ASCII upper-case order (the order its directory listing returns:
_sorts after the letters,AFark.ufobeforezeta.ufo). When it resolves a path likeunits/ARMCOM.fbi, loose files in the install win, then the first mounted archive that holds the path. An archive only counts as mounted if it is a version 1 archive ending with the Cavedog copyright trailer (see Trailer) whose directory reads; any other file is skipped and does not use up one of the ten*.hpislots.Query: units/ARMCOM.fbi priority ↑ loose files in the install root if found → use this │ ┌─────────────────────────────────────┐ │ │ rev31.gp3 (3.1 patch) │ else fall through │ ├─────────────────────────────────────┤ │ │ *.ccx in name order │ else fall through │ ├─────────────────────────────────────┤ │ │ *.ufo in name order │ else fall through │ ├─────────────────────────────────────┤ │ │ first ten *.hpi in name order │ else fall through │ ├─────────────────────────────────────┤ │ │ *.hpi on the disc (no limit) │ else fall through │ └─────────────────────────────────────┘ priority ↓ "not found"On the GOG release the ten
*.hpislots are taken bytactics1–8.hpi,totala1.hpiandtotala2.hpi.totala3.hpi,totala4.hpi(the retail maps) andworlds.hpi(the map sections) are the disc archives: a GOG install keeps them in the game directory, which stands in for the disc, so the game mounts them after every other archive. The same applies to any*.hpia mod adds: one whose name sorts after the tenth slot is reached only through that disc scan, below everything else. Ship mods as.ufo(no limit, above every.hpi) to avoid this.kbot mounts Total Annihilation contexts the same way, everywhere: the studio,
kbot mount,kbot mount flatten,kbot pack,kbot tntand the MCP server. The Files tab's Home page and Layering tab,kbot mount'sarchivesandstatscommands and the MCPvfs_game_datatool list the mount order and every archive that was not mounted, with the reason. TA: Kingdoms contexts keep kbot's overlay order (every archive by extension and name, a later one overriding an earlier one), which that game'sIPData.hpioverdata.hpilayering needs.The "loose files win" rule is why
kbot mount flattenproduces a directory that behaves identically to the layered set: every file the engine would have resolved through the archive stack is present as a loose file at the same path, so the resolution short-circuits at the top of the stack.
Which definition wins
Features, weapons and units are loaded by listing a directory:
features/**/*.tdf(subfolders included),weapons/*.tdfandunits/*.fbi(top level only — aweapons/old/subfolder is never read). The listing runs loose files first, then each mounted archive in mount order with its entries in stored order, and the first definition of a name wins; later ones are ignored. RetailCarScar05is defined infeatures/urban/cars2.tdfand again incars.tdf, andcars2.tdfis stored first, so the game draws it fromanims/cars2.gaf. The studio (feature palette, sandbox, packs, unit and weapon lists),kbot tnt previewandkbot tnt lint, and the MCP tools apply the same rule.
At a glance
┌─── 20-byte header (clear) ───┐ ┌─── encrypted directory ───┐ ┌─── encrypted file chunks ───┐
│ "HAPI" version dirSize ... │ │ tree of names + offsets │ │ SQSH chunks, ≤64 KiB each │
└──────────────────────────────┘ └───────────────────────────┘ └─────────────────────────────┘
▲
trailer (required by TA 3.1c):
"Copyright 1997 Cavedog Entertainment"
The flow to read an HPI is:
- Read the 20-byte header. Verify the
"HAPI"magic. - Derive the per-file XOR key from
DecryptKey. - Decrypt-and-read the directory tree.
- For each file you want, seek to its compressed chunks, decrypt them,
then dispatch on
CompMethod(LZ77, ZLib, or none).
Tip
Inspect any HPI without writing a parser.
kbot hpi info totala1.hpi # header, counts, "TA 3.1c would mount: yes/no"
kbot hpi list totala1.hpi -v # files, sizes, compression
kbot hpi list totala1.hpi -p "*.bos"
kbot hpi extract totala1.hpi -p "units/ARMCOM.fbi" -t ./out
See the CLI kbot hpi reference
for every flag.
From Go. Use formats/hpi:
import (
"io"
"github.com/coreprime/kbot/formats/hpi"
)
r, _ := hpi.OpenReader("totala1.hpi")
defer r.Close()
_ = r.Walk(func(e *hpi.Entry) error {
if !e.IsDir { fmt.Println(e.FullPath(), e.Size) }
return nil
})
rc, _ := r.Open("units/ARMCOM.fbi")
defer rc.Close()
data, _ := io.ReadAll(rc) // decompressed bytes
Header (20 bytes, clear)
typedef struct {
uint32 Marker; // 'HAPI' = 0x49504148, "BANK" if save-game
uint32 Version; // Usually 0x00010000
uint32 DirectorySize; // Total directory size in bytes (includes header)
uint32 DecryptKey; // Seed for the XOR cipher (0 = no encryption)
uint32 Start; // File offset of the directory (almost always 0x14)
} HPIHeader;
| Field | Notes |
|---|---|
Marker |
'HAPI' little-endian. The 'BANK' variant is used by save games and is not covered here. |
Version |
Retail Total Annihilation archives use 0x00010000 (v1, this section). TA: Kingdoms archives use 0x00020000 (v2); see TA: Kingdoms HPI v2. |
DirectorySize |
Includes the 20 header bytes, so the actual directory blob occupies DirectorySize − Start bytes starting at Start. |
DecryptKey |
Pass through the transform below to get the byte used as the XOR key. The key bytes 0 and 0xFF both mean "not encrypted" to TA 3.1c. Cavedog's retail archives use 0xBF. |
Start |
Where the directory begins. Has always been 0x14 in observed files. |
Deriving the cipher key
key = (DecryptKey << 2) | (DecryptKey >> 6) // rotate-left by 2 bits
Every encrypted byte is then unscrambled with its absolute file position:
plain[i] = cipher[i] ^ (uint8(pos + i) ^ key)
where pos is the absolute offset of cipher[0] in the file. Yes — the
seed advances with the file position, so you can't decrypt a buffer in
isolation; you have to know where it came from.
Important
If the key byte is 0 or 0xFF, do not apply the cipher at all.
TA 3.1c reads both as plaintext (for 0xFF the rotation above would
give 0xFF, but the game does not decrypt). Some custom archives (and a
handful of community mods) are written with no encryption — applying the
XOR cipher in that case will scramble what is already plaintext.
kbot hpi pack refuses --key 255 for this reason; use --key 0.
Directory tree
A directory is a count plus a pointer to its first entry:
typedef struct { uint32 EntryCount; uint32 EntriesOffset; } HPIDir;
Each entry is exactly 9 bytes:
typedef struct {
uint32 NameOffset; // → NUL-terminated string
uint32 DataOffset; // → HPIDir (subdir) or HPIFile (leaf)
uint8 Flag; // 1 = subdirectory, 0 = file
} HPIEntry;
For files, DataOffset points at:
typedef struct {
uint32 DataOffset; // Absolute file offset of the first compressed chunk
uint32 FileSize; // **Decompressed** size in bytes
uint8 CompMethod; // 0 = stored, 1 = LZ77, 2 = ZLib
} HPIFile;
All offsets are absolute within the file. That's why the canonical
parser allocates a DirectorySize-byte buffer and writes the directory at
position Start — that way pointers can index the buffer directly without
arithmetic.
Reading the directory
buf = bytearray(header.DirectorySize)
file.seek(header.Start)
encrypted = file.read(header.DirectorySize - header.Start)
buf[header.Start:] = decrypt(encrypted, key, seed=header.Start)
walk(buf, header.Start) # recurses on Flag == 1 entries
Note
Filename casing is preserved on write but matched case-insensitively
on read. Total Annihilation grew up on Windows 9x; mods commonly mix
armcom.fbi and ARMCOM.FBI even within the same archive. The game
folds ASCII letter case only, splits paths on both \ and /, and when
one directory stores several entries whose names differ only in case it
reads the last of them; the others are unreachable. It also treats
. and .. as ordinary names. kbot hpi extract therefore writes only
the reachable entry of such a group, and skips (with a message) any
entry whose path could not be written inside the target folder as
stored, such as ../../x.txt; kbot mount flatten never writes outside
its target either.
Compressed chunks (SQSH)
A file's data is sliced into chunks of up to 64 KiB decompressed. Each chunk has its own 19-byte header:
typedef struct {
uint32 Marker; // 'SQSH' = 0x48535153
uint8 Version; // Always 0x02 in observed files
uint8 CompMethod; // 1 = LZ77, 2 = ZLib
uint8 Encrypted; // 1 = chunk body is XOR-scrambled, 0 = plaintext
uint32 CompressedSize; // Length of the encrypted+compressed payload
uint32 DecompressedSize; // Length after CompMethod is undone
uint32 Checksum; // Sum of the encrypted payload bytes (uint32)
uint8 data[CompressedSize];
} HPIChunk;
The number of chunks is ceil(FileSize / 65536). Immediately after the
file's first HPIFile.DataOffset you'll find an array of uint32
compressed-sizes, one per chunk, followed by the chunks themselves
back-to-back.
The second XOR layer
If Encrypted == 1, each chunk body has a second XOR pass applied:
for (i = 0; i < CompressedSize; i++)
data[i] = (data[i] - i) ^ i;
Note that the checksum is over the still-encrypted bytes — verify before you undo the cipher.
LZ77 (CompMethod = 1)
A textbook sliding-window LZ77 with a 4 KiB history:
- Tag byte — 8 flag bits, LSB-first.
0⇒ literal,1⇒ back-reference. - Literal — copy 1 byte verbatim to the output and history.
- Match — read a
uint16. The upper 12 bits are the window offset (1–4095), the lower 4 bits encode the length aslen − 2(so matches are 2–17 bytes). - Terminator — a match with
windowOffset == 0ends the stream.
ZLib (CompMethod = 2)
Standard zlib raw stream. Feed the chunk body to inflate() after the
encryption layer has been removed; DecompressedSize is the expected
length.
Stored (CompMethod = 0)
Bytes are written verbatim — no chunking, no checksum, no second XOR pass. Useful for files that don't compress (already-compressed PCX/GAF), and the only mode the legacy "unit viewer" understands.
Trailer
Every archive the game mounts ends with a 36-byte ASCII copyright line:
Copyright 1997 Cavedog Entertainment
Nothing in the header points at it, but it is not optional: before
mounting an archive, TA 3.1c checks that its last 36 bytes read
Copyright ____ Cavedog Entertainment (the four year characters are not
compared). An archive without it, or with any bytes appended after it, is
skipped, and does not use up one of the ten *.hpi slots — a mod packed
without the trailer is silently absent in the game.
Every archive kbot writes keeps it: kbot hpi pack and the studio's
exports (a workspace's Export mod and the map editor's archive download).
kbot hpi info, kbot hpi list -v, the MCP hpi_info and hpi_list
tools and the Files tab's metadata for an archive print a
TA 3.1c would mount: yes / no, <reason> line.
TA: Kingdoms — HPI v2
Note
This section is the TA:K-only delta. Everything above applies to
Total Annihilation (Version == 0x00010000). TA: Kingdoms ships
the same .hpi extension but a different on-disk layout — read this
section in full before reaching for the v1 parser.
TA: Kingdoms keeps the 'HAPI' magic and the directory-tree concept but
otherwise overhauls the wrapper. The differences in one breath:
| Aspect | v1 (TA) | v2 (TA: Kingdoms) |
|---|---|---|
| Version field | 0x00010000 |
0x00020000 |
| Header size | 20 bytes (5 × uint32) | 32 bytes (8 bytes magic+version + 24-byte sub-header) |
| Encryption | XOR cipher seeded by DecryptKey, advances with file position |
None — everything is plaintext |
| Directory + name pool location | Right after the header (encrypted) | At arbitrary offsets near the tail of the file (typically), each optionally wrapped in a single SQSH chunk |
| Per-file chunking | Sliced into ≤64 KiB chunks with a length table; each chunk has its own SQSH header + checksum + optional second XOR pass | Single SQSH chunk with the whole payload, or stored bytes if CompressedSize == 0; no XOR pass, no length table |
Per-chunk cipher (Encrypted byte in SQSH) |
Used | Not used (we still respect the byte if it's set, but every TAK file we've sampled has it 0) |
v2 header
After the 8-byte prologue ('HAPI' + 0x00020000) come six 32-bit
little-endian fields — total header on disk is 32 bytes:
// At offset 0 — same as v1.
typedef struct { uint32 Marker; uint32 Version; } HPIPrologue;
// At offset 8 — TA: Kingdoms only.
typedef struct {
int32 DirectoryBlock; // Absolute offset of the directory blob
int32 DirectorySize; // Length of the directory blob on disk
int32 NameBlock; // Absolute offset of the string pool
int32 NameSize; // Length of the string pool on disk
int32 Data; // Offset of the first file payload (typically 0x20)
int32 Last78; // 0, or an offset to a trailing 78-byte block
} HPIHeaderV2;
The directory blob and name blob both may be SQSH-compressed — if
the first four bytes at DirectoryBlock / NameBlock are 'SQSH',
treat the block as a single chunk (header + payload) and decompress it.
If not, the Size bytes at the given offset are the raw decompressed
content.
v2 directory entries
The decompressed directory blob is a packed array of two record types
(no separate offset tables, no EntryCount prefix — children are
located via the parent's FirstSubDirectory / FirstFile pointers):
typedef struct { // 20 bytes — HpiDir2
int32 NamePtr; // Offset into NameBlock (NUL-terminated string)
int32 FirstSubDirectory; // Byte offset into the directory blob
int32 SubCount; // Number of subdirectories
int32 FirstFile; // Byte offset into the directory blob
int32 FileCount; // Number of files
} HpiDir2;
typedef struct { // 24 bytes — HpiEntry2
int32 NamePtr; // Offset into NameBlock
int32 Start; // Absolute file offset of the payload
int32 DecompressedSize;
int32 CompressedSize; // 0 ⇒ payload is stored verbatim, no SQSH wrapper
int32 Date; // time_t when packed (informational)
int32 Checksum; // Informational — the SQSH chunk carries its own
} HpiEntry2;
Per-file payloads
For a file entry at offset Start:
- If
CompressedSize == 0: readDecompressedSizebytes verbatim. - Otherwise: the next
CompressedSizebytes form a single SQSH chunk (same 19-byte header as v1) — read it, verify its checksum against the byte-sum of the payload, decode with the chunk'sCompMethod(1= LZ77,2= ZLib), and you have the file.
The v1 multi-chunk scheme (size table, ≤64 KiB chunks, encryption pass) is gone. v2 trades the compactness for a vastly simpler reader.
kbot's handling
kbot hpi list, extract, and info all transparently support both
dialects — they dispatch on the Version word in the header. Round-trip
writing (kbot hpi pack) currently emits v1 only; we don't yet author
v2 archives.
The internal v2 reader is in formats/hpi/hpi_v2.go.
hpi.Reader.Version() reports whether the file you loaded is v1 or v2.
Tip
Confirm a TAK install loads.
kbot hpi list ~/games/takingdoms/data.hpi -v | head
kbot ctx add ~/games/takingdoms --alias tak --game takingdoms
kbot mount # walks every TAK .hpi as a single VFS
Worked example — aflakker.ufo
The original Joe D. walkthrough used the Arm Flakker unit add-on; we'll reproduce its dissected header here. Hex dump of the first 32 bytes:
00000000 48 41 50 49 00 00 01 00 24 02 00 00 7D 00 00 00 HAPI....$...}...
00000010 14 00 00 00 ... ...
Field by field:
| Offset | Bytes | Field | Value |
|---|---|---|---|
0x00 |
48 41 50 49 |
Marker |
'HAPI' ✓ |
0x04 |
00 00 01 00 |
Version |
0x00010000 ✓ |
0x08 |
24 02 00 00 |
DirectorySize |
0x224 = 548 bytes |
0x0C |
7D 00 00 00 |
DecryptKey |
0x7D → key = (0x7D << 2) | (0x7D >> 6) = 0xF5 |
0x10 |
14 00 00 00 |
Start |
0x14 |
Decrypting [0x14, 0x224) and walking the tree yields:
.
├── anims/
│ └── armflak_gadget.gaf
├── download/
│ └── ARMFLAK.TDF
├── features/corpses/
│ └── armflak_dead.tdf
├── objects3d/
│ ├── armflak.3do
│ └── armflak_dead.3do
├── scripts/
│ └── ARMFLAK.COB
├── unitpics/
│ └── ARMFLAK.PCX
├── units/
│ └── ARMFLAK.FBI
└── weapons/
└── armflak_weapon.tdf
This nine-file layout is the canonical shape of a single-unit .ufo
add-on; the base totala*.hpi archives use the same scheme at scale (~30k
files).
Tip
Reproduce this dissection on your own machine.
kbot hpi info path/to/aflakker.ufo
kbot hpi list path/to/aflakker.ufo -v
kbot hpi extract path/to/aflakker.ufo -t ./aflakker-out
Repacking
To build a new archive from a directory tree:
kbot hpi pack ./aflakker-out --target aflakker-new.ufo
kbot defaults to LZ77 chunked compression with the standard 0xBF
header key and the Cavedog trailer — which produces a binary the retail
engine accepts. The flags keep it that way:
| Flag | Effect |
|---|---|
--method lz77|zlib|none |
Chunk compression; none writes stored (uncompressed) entries. |
--key N |
Header key byte; 0 writes an unencrypted archive. 255 is refused: the game reads it as "not encrypted". |
--trailer "Copyright 1998 Cavedog Entertainment" |
Another year in the trailer; any other text is refused. |
--no-trailer |
Omit the trailer — the game will not mount the result (a warning is printed). |
--format v2 |
Write a TA: Kingdoms archive (--key, --trailer, --no-trailer and --encode-chunks do not apply). |
Directory names that differ only in letter case (Units/ and units/ on a
case-sensitive disk) are merged the way the game compares names; two files
whose paths differ only in case are both stored, the game reads the one
packed last, and pack prints a warning.
Gotchas
Warning
Pointers are absolute to file start, not directory start. Subtract
nothing. The standard implementation trick is to allocate a buffer of
DirectorySize bytes and read the decrypted directory into it at offset
Start so the pointers index the buffer directly.
- Chunk count formula —
ceil(FileSize / 65536). There is no explicit count field; you have to derive it fromFileSize.
Typical sizes
| Archive | Size | Files | Notes |
|---|---|---|---|
totala1.hpi (sounds, anims) |
~30 MB | ~1900 | Base game, biggest single archive. |
totala2.hpi (units) |
~13 MB | ~600 | Most unit metadata + scripts. |
totala4.hpi (maps) |
~50 MB | ~300 | All retail maps. |
worlds.hpi (sections) |
~20 MB | ~5000 | Editor SCT libraries. |
ccdata.ccx (Core Contingency) |
~25 MB | ~1500 | Expansion units/maps. |
Typical single-unit .ufo |
10–150 KB | 8–12 | Most third-party unit add-ons. |
Map-only .ufo (with .tnt + .ota) |
0.5–5 MB | 2–4 | Per-map distribution. |
| Compression ratio | 35–55% | — | LZ77 most files; ZLib for the largest. |
- Some chunks expand when compressed. A "compressed" chunk can be
larger than 64 KiB if LZ77 found no matches. Treat
CompressedSizeas the authoritative length to copy. - The checksum is over encrypted bytes. If you decrypt first, you'll fail the checksum.
- Save games (
BANKmarker) are a different beast — they reuse the HPI wrapper but the encoded contents are serialised game state, not a filesystem. kbot does not decode them. .gp3requires the rev3.1 patch to load. It's still an HPI inside, but the engine refuses to mount it on unpatched installs.
Appendix — Full hex walk of AFark.ufo
AFark.ufo is the Arm Fark add-on — a small 9-file archive that's
ideal for end-to-end demonstration. We'll walk the header, decrypt
the directory, dissect a couple of entries, then peek at the
opening bytes of a compressed chunk.
1. Header (clear)
Raw bytes 0x00..0x14:
00000000: 48 41 50 49 00 00 01 00 24 02 00 00 a6 00 00 00 HAPI....$.......
00000010: 14 00 00 00 ....
Decoded:
| Offset | Bytes | Field | Value |
|---|---|---|---|
0x00 |
48 41 50 49 |
Marker |
'HAPI' ✓ |
0x04 |
00 00 01 00 |
Version |
0x00010000 |
0x08 |
24 02 00 00 |
DirectorySize |
0x224 = 548 bytes |
0x0C |
a6 00 00 00 |
DecryptKey |
byte 0xA6 |
0x10 |
14 00 00 00 |
Start |
0x14 |
Derive the cipher key:
key = (0xA6 << 2) | (0xA6 >> 6)
= 0x98 | 0x02
= 0x9A
2. Decrypted directory (offsets 0x14..0x224)
After running plain[i] = cipher[i] ^ (uint8(0x14 + i) ^ 0x9A) over
the directory blob — and inserting it into a 548-byte buffer at offset
0x14 so pointers index correctly — the first 96 bytes read:
00000010 08 00 00 00 1c 00 00 00 64 00 00 00
^^^^^^^^^^^ ^^^^^^^^^^^
00000020 6a 00 00 00 01 97 00 00 00 a0 00 00 00 01 c6 00
00000030 00 00 cf 00 00 00 01 13 01 00 00 1d 01 00 00 01
00000040 66 01 00 00 6e 01 00 00 01 94 01 00 00 9d 01 00
00000050 00 01 c3 01 00 00 c9 01 00 00 01 ef 01 00 00 f7
00000060 01 00 00 01 61 6e 69 6d 73 00 01 00 00 00 72 00
^^^^^^^^^^^^^^^^^ "anims"
At 0x14 we have 0x00000008 — the root entry count: 8
top-level subdirectories. At 0x18, 0x0000001C — the offset of
the first directory entry.
3. The first entry
The 9 bytes at 0x1C decode as:
| Field | Bytes | Value |
|---|---|---|
NameOffset |
64 00 00 00 |
0x64 → "anims" string |
DataOffset |
6a 00 00 00 |
0x6A → subdirectory descriptor |
Flag |
01 |
1 ⇒ subdirectory |
At 0x64:
00000060 .... 61 6e 69 6d 73 00 ....anims.
^^^^^^^^^^^^^^^^^
The string "anims\0" — confirming the directory name.
At 0x6A:
00000060 .... 01 00 00 00 72 00 ........r.
^^^^^^^^^^^ ^^^^
00000070 00 00 7b 00 00 00 ... ..{...
^^^^^
| Field | Bytes | Value |
|---|---|---|
Subdirectory EntryCount |
01 00 00 00 |
1 |
Subdirectory EntriesOffset |
72 00 00 00 |
0x72 |
So the anims/ directory contains 1 file, described starting at
0x72:
| Field | Bytes | Value |
|---|---|---|
NameOffset |
7b 00 00 00 |
0x7B → file name |
DataOffset |
8e 00 00 00 |
0x8E → file descriptor |
Flag |
00 |
0 ⇒ file |
The name at 0x7B:
00000070 .. .. .. 61 72 6d 66 61 armfa
00000080 72 6b 5f 67 61 64 67 65 74 2e 67 61 66 00 rk_gadget.gaf.
→ "armfark_gadget.gaf".
The file descriptor at 0x8E is a 9-byte HPIFile:
00000080 24 02 $.
00000090 00 00 48 48 00 00 01 ..HH...
| Field | Bytes | Value |
|---|---|---|
DataOffset |
24 02 00 00 |
0x0224 — first byte after the directory! |
FileSize |
48 48 00 00 |
0x4848 = 18504 bytes |
CompMethod |
01 |
1 ⇒ LZ77 |
This matches kbot hpi list -v:
anims/armfark_gadget.gaf 18504
4. Reading the full tree
Continuing through the remaining 7 top-level entries produces:
.
├── anims/
│ └── armfark_gadget.gaf (18504 B, LZ77)
├── download/
│ └── ARMFARK.TDF (84 B, LZ77)
├── features/corpses/
│ └── armfark_dead.tdf (619 B, LZ77)
├── objects3d/
│ ├── armfark.3do (7337 B, LZ77)
│ └── armfark_dead.3do (3551 B, LZ77)
├── scripts/
│ └── ARMFARK.COB (18988 B, LZ77)
├── unitpics/
│ └── ARMFARK.PCX (8728 B, LZ77)
├── units/
│ └── ARMFARK.FBI (1466 B, LZ77)
└── weapons/
└── armfark_weapon.tdf (430 B, LZ77)
Nine files, the canonical shape of a single-unit .ufo.
5. A compressed chunk
The first byte after the 548-byte directory is at 0x224. Decrypting
that region (the chunk header is also encrypted by the directory
cipher) reveals:
00000220 .. 53 51 53 48 02 01 01 cc .SQSH....
| Offset (from 0x224) | Bytes | Field | Value |
|---|---|---|---|
+0 |
53 51 53 48 |
Marker |
'SQSH' ✓ |
+4 |
02 |
Version |
0x02 (always) |
+5 |
01 |
CompMethod |
1 ⇒ LZ77 |
+6 |
01 |
Encrypted |
1 ⇒ apply the second XOR pass |
+7.. |
… | CompressedSize, DecompressedSize, Checksum, then data[] |
… |
Once you unscramble the chunk body with data[i] = (data[i] - i) ^ i
and run the resulting bytes through the LZ77 decoder described above,
you get the decompressed GAF — ready to hand off to a GAF
parser.
Tip
Reproduce this walk on your own machine.
# Show counts + key
kbot hpi info AFark.ufo
# Walk the tree
kbot hpi list AFark.ufo -v
# Pull a single file out, decompressed and ready to inspect
kbot hpi extract AFark.ufo -p "anims/armfark_gadget.gaf" -t ./out
kbot gaf list ./out/anims/armfark_gadget.gaf
Appendix — Full hex walk of Jersey.hpi (HPI v2)
The TA: Kingdoms equivalent of the AFark walkthrough. Jersey.hpi is
a 67 KB voice-pack HPI that ships in the GoG TA:K install — six
sound-effect WAVs and nothing else. Small enough to walk end-to-end,
big enough to exercise every v2 mechanic (32-byte header, name pool,
SQSH-wrapped directory blob, single-chunk file payloads). The v1
walkthrough above covers everything that remains the same; this
section calls out only the v2 deltas.
1. The 32-byte header
00000000: 48 41 50 49 00 00 02 00 92 06 01 00 ad 00 00 00 HAPI............
00000010: 40 06 01 00 5b 00 00 00 20 00 00 00 3f 07 01 00 @...[... ...?...
| Offset | Bytes | Field | Value |
|---|---|---|---|
0x00 |
48 41 50 49 |
Marker |
'HAPI' ✓ |
0x04 |
00 00 02 00 |
Version |
0x00020000 — v2 |
0x08 |
92 06 01 00 |
DirectoryBlock |
0x00010692 — offset of the directory blob |
0x0C |
ad 00 00 00 |
DirectorySize |
0xAD = 173 bytes |
0x10 |
40 06 01 00 |
NameBlock |
0x00010640 — offset of the name pool |
0x14 |
5b 00 00 00 |
NameSize |
0x5B = 91 bytes |
0x18 |
20 00 00 00 |
Data |
0x20 — file payloads start here, right after the header |
0x1C |
3f 07 01 00 |
Last78 |
0x0001073F — end-of-archive marker (0x1073F = 67391 = file size − 78 footer bytes) |
No XOR cipher, no decryption transforms, no per-position seed. The
header values point at absolute file offsets you can seek to
directly.
2. The name pool (raw text, near the tail)
At DirectoryBlock minus a few hundred bytes (the engine writes the
name pool before the directory) sits the string table:
00010640: 00 73 6f 75 6e 64 73 00 57 68 6f 64 69 65 64 2e .sounds.Whodied.
00010650: 7761 7600 53 75 72 65 2e 77 61 76 00 57 68 6f wav.Sure.wav.Who
00010660: 6168 5061 6c 6c 79 2e 77 61 76 00 46 6f 72 67 ahPally.wav.Forg
00010670: 6574 6162 6f 75 74 69 74 2e 77 61 76 00 59 6f etaboutit.wav.Yo
00010680: 2e77 6176 00 49 67 6f 74 63 68 65 72 2e 77 61 .wav.Igotcher.wa
00010690: 76 00 v.
Every directory and file entry stores a NamePtr byte offset into
this block; following the pointer and reading until the next \0
gives you the name. Walking byte by byte:
Offset (relative to NameBlock) |
NUL-terminated string |
|---|---|
0x00 |
"" (root directory — empty name) |
0x01 |
"sounds" (the only sub-directory) |
0x08 |
"Whodied.wav" |
0x14 |
"Sure.wav" |
0x1D |
"WhoahPally.wav" |
0x2C |
"Forgetaboutit.wav" |
0x3E |
"Yo.wav" |
0x45 |
"Igotcher.wav" |
Note
The name pool's nominal size includes a few unused trailing
bytes. NameSize=0x5B runs to offset 0x1069B, but the last
meaningful byte (the NUL after Igotcher.wav) is at 0x10691.
The 9 trailing bytes overlap into the start of the SQSH directory
chunk and are never indexed by any NamePtr — readers tolerate them
simply by never looking at them.
3. The directory blob (SQSH-wrapped)
At 0x10692 the directory blob starts with a SQSH chunk header — v2
re-uses the v1 chunk format:
00010692: 53 51 53 48 02 02 01 9a 00 00 00 b8 00 00 00 9b SQSH............
000106a2: 4f 00 00 78 9e 63 … O..x.c…
| Offset | Bytes | Field | Value |
|---|---|---|---|
+0 |
53 51 53 48 |
Marker |
'SQSH' ✓ |
+4 |
02 |
Version |
2 |
+5 |
02 |
CompMethod |
2 ⇒ ZLib |
+6 |
01 |
Encrypted |
1 — chunk body has the second XOR pass applied (rare in TAK; this archive has it) |
+7 |
9a 00 00 00 |
CompressedSize |
0x9A = 154 bytes |
+11 |
b8 00 00 00 |
DecompressedSize |
0xB8 = 184 bytes |
+15 |
9b 4f 00 00 |
Checksum |
0x4F9B |
+19..+172 |
… | data[] |
154 bytes of XOR-scrambled ZLib stream |
19 + 154 = 173 = 0xAD = DirectorySize, so the whole chunk fits the
header's claim. After undoing the XOR pass (same algorithm as v1) and
running through inflate, you get 184 bytes of packed directory
records.
4. Decoding the directory records
The decompressed blob holds a packed array of two record types: 20-byte
directory entries (HpiDir2) and 24-byte file entries
(HpiEntry2). There's no count prefix or offset table — the root
directory always sits at offset 0 of the blob, and its
FirstSubDirectory / FirstFile fields point at the children.
The root HpiDir2 (at decompressed offset 0, 20 bytes):
| Offset | Bytes (decompressed) | Field | Value |
|---|---|---|---|
+0 |
00 00 00 00 |
NamePtr |
0 → "" (root) |
+4 |
14 00 00 00 |
FirstSubDirectory |
0x14 — single 20-byte sub-dir record at this offset |
+8 |
01 00 00 00 |
SubCount |
1 (the sounds/ directory) |
+12 |
00 00 00 00 |
FirstFile |
0 — no files at the root |
+16 |
00 00 00 00 |
FileCount |
0 |
Following the pointer to 0x14 lands on the sounds HpiDir2:
NamePtr = 0x01→"sounds"FirstSubDirectory = 0,SubCount = 0— no nested subdirsFirstFile = 0x28— first of sixHpiEntry2records (24 bytes each)FileCount = 6
Each file record at 0x28, 0x40, 0x58, … carries:
NamePtr // → name pool offset
Start // → absolute file offset of the SQSH chunk (or stored bytes)
DecompressedSize
CompressedSize // 0 ⇒ stored verbatim (no SQSH wrapper)
Date // time_t, informational
Checksum // informational
Decoded, the six entries correspond exactly to what kbot hpi list
reports:
sounds/Whodied.wav 56474 bytes
sounds/Sure.wav 15248 bytes
sounds/WhoahPally.wav 24826 bytes
sounds/Forgetaboutit.wav 24826 bytes
sounds/Yo.wav 19368 bytes
sounds/Igotcher.wav 35980 bytes
5. What's NOT here (vs. v1)
If you have the v1 walkthrough fresh in mind, the missing pieces are the whole point of v2's simplification:
- No XOR cipher derivation —
DecryptKeyis always0for v2. Even when a SQSH chunk hasEncrypted == 1, the second-pass XOR inside the chunk is still applied, but the directory/payload envelope is plaintext. - No per-position seed — the byte-by-byte cipher that complicated v1 readers (decryption depends on absolute file offset) doesn't exist at the v2 envelope level.
- No chunk size table — v1 files chunked into ≤64 KiB blocks with a leading uint32-size array. v2 stores each file as a single SQSH chunk (or stored bytes), so there's nothing to size-table.
- No
0x14directory-immediately-after-header convention — the directory and name blocks live wherever the writer parked them (typically near the file tail). The header points at them.
Tip
Reproduce this walk on your own machine.
kbot ctx add /path/to/tak --alias tak --game takingdoms
kbot hpi info /path/to/tak/Jersey.hpi
kbot hpi list /path/to/tak/Jersey.hpi -v
kbot hpi extract /path/to/tak/Jersey.hpi -t ./jersey-out
# And to peek at the raw bytes:
xxd -l 64 /path/to/tak/Jersey.hpi # 32-byte v2 header
xxd -s 0x10640 -l 96 /path/to/tak/Jersey.hpi # name pool
xxd -s 0x10692 -l 32 /path/to/tak/Jersey.hpi # SQSH directory chunk header