Recovering MinIO Data From Raw Drives, Without a Server

MinIO is an S3-compatible object store that many organisations run on their own hardware for backups, logs and data lakes. When its server dies, the data is still on the drives, but not as ordinary files: each object is split into shards, spread across drives, hashed and described by a binary metadata file called xl.meta. In this post we recover MinIO data from raw drives, following one case from start to finish: we read the on-disk format byte by byte, rebuild objects by hand, lose two drives and rebuild them from parity, look at what deleted versions leave behind, and finish with a small Python tool that does all of it. Every hex dump and line of output comes from a lab we built for this post.

The scenario

A client's storage server has failed. It ran a single MinIO node with four data drives holding a bucket of case material. The operating system disk is gone, and with it the MinIO binary, its configuration and the root credentials. We receive a forensic copy of each data drive, and a question: what was in the bucket, including anything deleted, and can it be recovered exactly?

Starting a fresh MinIO server on the copies is the wrong first move. A server writes to its drives as soon as it starts: it heals objects, runs its scanner and empties its trash folder, which can destroy the deleted data we most want to see. So we read the drives as data, without MinIO.

To make every claim checkable, we rebuilt the scenario in a lab. The MinIO community edition is now distributed as source, so we built RELEASE.2025-10-15T17-29-55Z from its tagged commit and started it on four folders:

minio server --address 127.0.0.1:9100 "/lab/d{1...4}"

INFO: Formatting 1st pool, 1 set(s), 4 drives per set.
Version: RELEASE.2025-10-15T17-29-55Z (go1.27.3 linux/amd64)
API: http://127.0.0.1:9100

Four drives make one erasure set (the group of drives an object is spread across) with the default parity of EC:2, meaning two data shards and two parity shards per object. With the MinIO client, mc, we created a versioned bucket called case-files and uploaded:

A 152-byte text file, small enough to be stored inside its metadata.
A 5,000,050-byte CSV file, stored as erasure-coded shards.
A 12,595,257-byte archive, sent as a three-part multipart upload with 5 MiB parts.
A log file, compressed by MinIO on the way in.
A text file encrypted with SSE-S3 (server-side encryption with a MinIO-managed key).
Two versions of a report, followed by a delete, which leaves a delete marker.

We recorded a SHA-256 hash of every original, then stopped the server and copied the drive folders. Everything below runs against those copies.

How MinIO stores an object

MinIO keeps no database. The layout on each drive is a plain directory tree, and the object's key becomes a directory path:

Four MinIO drives, each with format.json, an object folder with xl.meta and part.1, and the shard it holds; EcDist 3, 4, 1, 2 maps drive slots to shards

Each drive holds .minio.sys/format.json, which identifies the drive, and one directory per object, named after its key. Inside it are an xl.meta file and, for larger objects, a data directory named by a UUID that holds part.1, part.2 and so on.

Each object is cut into blocks of 1 MiB (1,048,576 bytes). Each block is split into two equal data shards and two parity shards are calculated from them, and each of the four shards goes to a different drive. Any two of the four shards are enough to rebuild the block. Small objects skip the data directory entirely and store their shards inside xl.meta itself. The test is on the shard, not the object: a shard of up to 128 KiB is inlined, or up to 16 KiB in a versioned bucket, so in our versioned EC:2 bucket any object up to about 32 KiB is stored inline.

First triage of the drives

The first job is to find out which drive is which, and what the set looked like. Each drive's format.json names the deployment, the drive itself and the order of every drive in the set:

python3 -m json.tool d3/.minio.sys/format.json

{
    "version": "1",
    "format": "xl",
    "id": "8cbce3df-fd3c-4f55-a038-6b86d9fe1232",
    "xl": {
        "version": "3",
        "this": "38f32057-88f5-4001-9527-6b1a4758eb20",
        "sets": [
            [
                "b3fdf080-4d99-4eda-9f4f-f5fa76ee30b0",
                "d5057020-0f87-495f-82a5-0949e2eced8c",
                "38f32057-88f5-4001-9527-6b1a4758eb20",
                "d13bd3bb-59c5-4d01-b25f-05f9e3633229"
            ]
        ],
        "distributionAlgo": "SIPMOD+PARITY"
    }
}

The id is the deployment ID, and it must be the same on every drive; a drive with a different id came from another deployment. The this value is the drive's own UUID, and its position in sets is its slot in the erasure set. Here this is the third entry, so this copy is drive 3 of 4, whatever the evidence bag or the mount point says. Drives get moved between bays, so record this mapping before anything else.

Next, list the objects. Below .minio.sys, every directory that holds an xl.meta is an object:

find d1 -path d1/.minio.sys -prune -o -type f -print | sort

d1/case-files/evidence/evidence-bundle.tar/ace0551a-65fd-42ff-a007-e1002c5c8b2e/part.1
d1/case-files/evidence/evidence-bundle.tar/ace0551a-65fd-42ff-a007-e1002c5c8b2e/part.2
d1/case-files/evidence/evidence-bundle.tar/ace0551a-65fd-42ff-a007-e1002c5c8b2e/part.3
d1/case-files/evidence/evidence-bundle.tar/xl.meta
d1/case-files/finance/invoices-2026.csv/6e569f90-5270-4094-ade8-9a1c347ec445/part.1
d1/case-files/finance/invoices-2026.csv/xl.meta
d1/case-files/logs/auth.log/1cebb0fa-da36-4e61-ac36-ff70ee66a7ed/part.1
d1/case-files/logs/auth.log/xl.meta
d1/case-files/notes/handover.txt/xl.meta
d1/case-files/reports/q3.txt/xl.meta
d1/case-files/secure/keys.txt/xl.meta

The multipart archive has three part files, the text files have no data directory, and q3.txt has a single xl.meta even though we wrote two versions and deleted it: every version lives in that one file.

Inside xl.meta

The format is defined in cmd/xl-storage-format-v2.go in the MinIO source. Here is the whole of xl.meta for notes/handover.txt on drive 1:

00000000  58 4c 32 20 01 00 03 00  c6 00 00 01 6d 03 03 01  |XL2 ........m...|
00000010  c4 26 97 c4 10 54 83 4c  21 38 c1 4f 0f a8 32 ca  |.&...T.L!8.O..2.|
00000020  1d 89 62 4f 70 d3 18 e4  8c 77 ec 40 84 57 c4 04  |[email protected]..|
00000030  f3 8e 4a 3b 01 06 02 02  c5 01 3f 83 a4 54 79 70  |..J;......?..Typ|
00000040  65 01 a5 56 32 4f 62 6a  de 00 11 a2 49 44 c4 10  |e..V2Obj....ID..|
00000050  54 83 4c 21 38 c1 4f 0f  a8 32 ca 1d 89 62 4f 70  |T.L!8.O..2...bOp|
00000060  a4 44 44 69 72 c4 10 92  d1 42 9b 5c 0e 41 7a a3  |.DDir....B.\.Az.|
00000070  b6 f4 20 05 a0 af 27 a6  45 63 41 6c 67 6f 01 a3  |.. ...'.EcAlgo..|
00000080  45 63 4d 02 a3 45 63 4e  02 a7 45 63 42 53 69 7a  |EcM..EcN..EcBSiz|
00000090  65 d2 00 10 00 00 a7 45  63 49 6e 64 65 78 01 a6  |e......EcIndex..|
000000a0  45 63 44 69 73 74 94 01  02 03 04 a8 43 53 75 6d  |EcDist......CSum|
000000b0  41 6c 67 6f 01 a8 50 61  72 74 4e 75 6d 73 91 01  |Algo..PartNums..|
000000c0  a9 50 61 72 74 45 54 61  67 73 c0 a9 50 61 72 74  |.PartETags..Part|
000000d0  53 69 7a 65 73 91 d1 00  98 aa 50 61 72 74 41 53  |Sizes.....PartAS|
000000e0  69 7a 65 73 91 d1 00 98  a4 53 69 7a 65 d1 00 98  |izes.....Size...|
000000f0  a5 4d 54 69 6d 65 d3 18  e4 8c 77 ec 40 84 57 a7  |[email protected].|
00000100  4d 65 74 61 53 79 73 81  bc 78 2d 6d 69 6e 69 6f  |MetaSys..x-minio|
00000110  2d 69 6e 74 65 72 6e 61  6c 2d 69 6e 6c 69 6e 65  |-internal-inline|
00000120  2d 64 61 74 61 c4 04 74  72 75 65 a7 4d 65 74 61  |-data..true.Meta|
00000130  55 73 72 82 a4 65 74 61  67 d9 20 31 64 30 32 32  |Usr..etag. 1d022|
00000140  62 61 61 33 34 61 36 64  35 62 31 30 35 34 63 37  |baa34a6d5b1054c7|
00000150  64 37 64 33 32 66 66 33  34 63 38 ac 63 6f 6e 74  |d7d32ff34c8.cont|
00000160  65 6e 74 2d 74 79 70 65  aa 74 65 78 74 2f 70 6c  |ent-type.text/pl|
00000170  61 69 6e a1 76 ce 68 ef  da 13 ce 41 ed 6a ed 01  |ain.v.h....A.j..|
00000180  81 d9 24 35 34 38 33 34  63 32 31 2d 33 38 63 31  |..$54834c21-38c1|
00000190  2d 34 66 30 66 2d 61 38  33 32 2d 63 61 31 64 38  |-4f0f-a832-ca1d8|
000001a0  39 36 32 34 66 37 30 c4  6c 6a d5 a1 3b 30 44 a4  |9624f70.lj..;0D.|
000001b0  ea fa e0 a8 bf d2 08 9f  ab e0 92 d6 b6 f0 c0 19  |................|
000001c0  23 4d 12 d6 1a 36 44 ad  18 49 6e 63 69 64 65 6e  |#M...6D..Inciden|
000001d0  74 20 32 30 32 36 2d 30  33 31 20 68 61 6e 64 6f  |t 2026-031 hando|
000001e0  76 65 72 0a 41 6e 61 6c  79 73 74 3a 20 6a 2e 64  |ver.Analyst: j.d|
000001f0  6f 65 0a 41 66 66 65 63  74 65 64 20 68 6f 73 74  |oe.Affected host|
00000200  3a 20 31 39 32 2e 30 2e  32 2e 34 34 20 28 66 69  |: 192.0.2.44 (fi|
00000210  6c 65 73 68 61                                    |lesha|

Most of the file is MessagePack, a compact binary form of JSON in which a type byte comes before each value (c4 26 is "binary, 38 bytes", a4 is "string, 4 bytes", d3 is "signed 64-bit integer"). Around it sits a small fixed frame:

Anatomy of xl.meta: XL2 magic, version 1.3, bin32 length, metadata block with version header and V2Obj fields, xxhash64 CRC, and inline data with a hash and shard

Taking the dump apart in order:

Offset	Bytes	Meaning
`0x00`	`58 4c 32 20`	Magic, `XL2`.
`0x04`	`01 00 03 00`	Format version 1.3, two little-endian 16-bit numbers.
`0x08`	`c6 00 00 01 6d`	MessagePack bin32: 365 bytes of metadata follow.
`0x0d`	`03 03 01`	Header version 3, metadata version 3, one version.
`0x10`	`c4 26 97 ...`	Version header: a seven-item array (below).
`0x38`	`c5 01 3f 83 ...`	Version metadata: a 319-byte map.
`0x17a`	`ce 41 ed 6a ed`	CRC of the metadata block.
`0x17f`	`01 81 d9 24 ...`	Inline data: version 1, then a map.

The version header is a fast index, so MinIO can list versions without decoding everything. Its seven items are the version ID (54834c21-38c1-4f0f-a832-ca1d89624f70), the modification time in nanoseconds since 1970 (0x18e48c77ec408457, which is 2026-11-03 13:36:38.292948055 UTC), a 4-byte signature, the type (1 for an object, 2 for a delete marker), flags, and the parity and data shard counts. The flags byte 06 sets "uses data dir" (2) and "inline data" (4). The signature (f3 8e 4a 3b) is a hash of the version's metadata with the drive-specific EcIndex zeroed, so it is identical on all four drives; it gives a quick way to check that the copies agree.

The CRC at 0x17a is the lower 32 bits of an xxHash64 of the 365-byte metadata block, stored as a MessagePack uint32. We checked it against all 24 xl.meta files, six objects on four drives, and it matched every time. A failed CRC is a strong sign of a truncated or damaged file.

Decoding the version metadata gives the fields we need for recovery. show.py is a few lines that call the parse_xlmeta() function from the tool at the end of this post and print the result:

python3 show.py d1/case-files/notes/handover.txt/xl.meta

format 1.3 crc ok: True
{'Type': 1,
 'V2Obj': {'ID': b'T\x83L!8\xc1O\x0f\xa82\xca\x1d\x89bOp',
           'DDir': b"\x92\xd1B\x9b\\\x0eAz\xa3\xb6\xf4 \x05\xa0\xaf'",
           'EcAlgo': 1,
           'EcM': 2,
           'EcN': 2,
           'EcBSize': 1048576,
           'EcIndex': 1,
           'EcDist': [1, 2, 3, 4],
           'CSumAlgo': 1,
           'PartNums': [1],
           'PartETags': None,
           'PartSizes': [152],
           'PartASizes': [152],
           'Size': 152,
           'MTime': 1793712998292948055,
           'MetaSys': {'x-minio-internal-inline-data': b'true'},
           'MetaUsr': {'etag': '1d022baa34a6d5b1054c7d7d32ff34c8', 'content-type': 'text/plain'}},
 'v': 1760549395}

The names are terse, so here they are in full:

ID and DDir – the version ID and the data directory UUID, both as 16 raw bytes.
EcAlgo – the erasure algorithm; 1 is Reed-Solomon.
EcM and EcN – the number of data shards and parity shards. The MinIO source comments confirm that M is data and N is parity.
EcBSize – the erasure block size, 1 MiB.
EcIndex – which shard this drive holds, counted from 1. It is the only field that differs between drives.
EcDist – the distribution: the shard number for each drive slot, in format.json order.
CSumAlgo – the bitrot hash algorithm; 1 is HighwayHash.
PartNums, PartSizes and PartASizes – the part numbers, stored sizes and actual (uncompressed) sizes.
MTime, MetaSys and MetaUsr – the modification time, MinIO's internal metadata and the user metadata, including the S3 ETag.
v – the release that wrote the version, as a Unix time. 1760549395 is 2025-10-15 17:29:55 UTC, the MinIO release we ran.

EcDist is not random. MinIO calculates it with hashOrder(), which takes the CRC32 of bucket/object modulo the number of drives as a starting slot. For case-files/notes/handover.txt that gives [1, 2, 3, 4]; for case-files/finance/invoices-2026.csv it gives [3, 4, 1, 2], so on that object drives 3 and 4 hold the data and drives 1 and 2 hold the parity.

Recovering an inline object by hand

The bytes after the CRC are the inline data: a version byte (01), then a MessagePack map from version ID to bytes. The value for our one version is 108 bytes long (c4 6c), and the readable text starts 32 bytes into it. Those 32 bytes are a bitrot hash, a checksum MinIO writes before every shard block so it can detect silent corruption.

So even a 152-byte object is erasure coded. Each drive stores a 76-byte shard, half the file, plus its hash. Printing the first 20 bytes after the hash on each drive shows two halves of the text and two blocks of parity:

d1 EcIndex 1 108 bytes -> 49 6e 63 69 64 65 6e 74 20 32 30 32 36 2d 30 33 31 20 68 61
d2 EcIndex 2 108 bytes -> 72 65 2d 30 32 29 0a 46 69 72 73 74 20 73 65 65 6e 3a 20 32
d3 EcIndex 3 108 bytes -> 3f 78 ff db c8 fd a6 10 b2 b2 b6 be 1a 91 9a 9f 8f 14 f8 c7
d4 EcIndex 4 108 bytes -> 04 73 b1 82 9e b1 c2 22 fb f2 f5 f8 0c cf cf c9 d0 0e b0 94

To rebuild the object, strip the 32-byte hash from shards 1 and 2, join them in shard order and cut the result to Size:

from minio_recover import parse_xlmeta

shards = {}
for d in ("d1", "d2"):
    m = parse_xlmeta(open(f"{d}/case-files/notes/handover.txt/xl.meta", "rb").read())
    idx = m["versions"][0]["V2Obj"]["EcIndex"]
    shards[idx] = list(m["inline"].values())[0][32:]
print((shards[1] + shards[2])[:152].decode())

Incident 2026-031 handover
Analyst: j.doe
Affected host: 192.0.2.44 (fileshare-02)
First seen: 2026-10-28 09:14 UTC
Status: contained, awaiting imaging

The trim matters. A shard is the block size divided by the number of data shards, rounded up, and Reed-Solomon pads the last shard with zeros, so the joined shards can be a byte or so longer than the object.

Recovering an erasure-coded object

The CSV file is too big for inline storage, so its shards are in part.1 in the data directory. Each drive's part.1 is 2,500,185 bytes, and that number can be worked out from xl.meta alone:

5,000,050 bytes in 1 MiB blocks is four full blocks and a final block of 805,746 bytes.
Each full block gives a 524,288-byte shard per drive, and the last gives a 402,873-byte shard (805,746 divided by two).
That is 2,500,025 bytes of shard data, plus a 32-byte hash in front of each of the five blocks: 2,500,185.

The 32-byte hash is HighwayHash-256 with a fixed key, magicHighwayHash256Key, set in cmd/bitrot.go. There is no HighwayHash package for Python, so we wrote one from Google's reference implementation and tested it against the stored hashes: it matched on a full block, a short final block and an inline shard. It is in the tool below.

A part file as 32-byte hashes and shard blocks; data shards 1 and 2 are joined and trimmed, and Reed-Solomon parity rebuilds missing or damaged shards

The diagram shows how a part file is laid out and how blocks come back together. For this object EcDist is [3, 4, 1, 2], so the data shards should be on drives 3 and 4. Reading the end of the first block on drive 3 and the start of the first block on drive 4 shows one CSV row cut across the two shards:

tail -c +33 d3/case-files/finance/invoices-2026.csv/6e569f90-5270-4094-ade8-9a1c347ec445/part.1 | head -c 524288 | tail -c 48 | hexdump -C

00000000  2d 34 30 39 39 2c 38 33  33 2e 31 30 2c 31 39 38  |-4099,833.10,198|
00000010  2e 35 31 2e 31 30 30 2e  34 37 0a 49 4e 56 2d 31  |.51.100.47.INV-1|
00000020  30 39 38 30 36 2c 32 30  32 36 2d 30 38 2d 30 33  |09806,2026-08-03|

tail -c +33 d4/case-files/finance/invoices-2026.csv/6e569f90-5270-4094-ade8-9a1c347ec445/part.1 | head -c 48 | hexdump -C

00000000  2c 43 55 53 54 2d 34 38  37 30 2c 37 38 30 2e 39  |,CUST-4870,780.9|
00000010  36 2c 31 39 38 2e 35 31  2e 31 30 30 2e 32 36 0a  |6,198.51.100.26.|
00000020  49 4e 56 2d 31 30 39 38  30 37 2c 32 30 32 36 2d  |INV-109807,2026-|

INV-109806,2026-08-03 on drive 3 continues as ,CUST-4870,780.96 on drive 4. The rule for every block is the same: shard 1 then shard 2, with no gap.

A multipart upload adds one more layer. The archive's xl.meta lists PartNums: [1, 2, 3] and PartSizes: [5242880, 5242880, 2109497], and each part is erasure coded on its own, with its own block grid, as part.1, part.2 and part.3. On each drive the first two are 2,621,600 bytes (five blocks of 32 plus 524,288 bytes) and the third is 1,054,845 bytes. To rebuild the object, rebuild each part and join them in part-number order.

Losing drives: parity reconstruction

MinIO uses klauspost/reedsolomon (version 1.12.4 in this release) with its default matrix. That library starts with a Vandermonde matrix over GF(2⁸), the 256-value finite field defined by the polynomial x⁸ + x⁴ + x³ + x² + 1, and multiplies it by the inverse of its top square. This makes the data shards pass through unchanged, and leaves rows that produce the parity. For two data and two parity shards our code produces this matrix:

So parity shard 1 is 3·D1 + 2·D2, and parity shard 2 is 2·D1 + 3·D2, where + is XOR and · is multiplication in GF(2⁸). To confirm that we had the same matrix as MinIO, we re-encoded the first block of the CSV file from drives 3 and 4 and compared the result with drives 1 and 2:

parity 1 matches: True
parity 2 matches: True

Rebuilding from any two shards works backwards: take the matrix rows for the shards you have, invert that 2×2 matrix and multiply. To test it, we deleted drives 3 and 4 from the working copy, which removed every data shard of the CSV file and left only parity:

python3 minio_recover.py recover --object case-files/finance/invoices-2026.csv -o out/invoices-2026.csv --expect 87bd4c8bb13e6d4cd550bae07b04e72ab34943bf20027e65a8d6c39bdb03ea45 d1 d2 d3 d4

version 6af706f5-65d9-4aaf-a17f-ae54866d2d4d  2026-11-03 13:36:38.394871 UTC  5000050 bytes  EC 2+2  shards found: [3, 4]
wrote out/invoices-2026.csv: 5000050 bytes, 10 data shard block(s) rebuilt from parity
etag 11088e17e51dd99b01444db857b7543e (stored 11088e17e51dd99b01444db857b7543e)
sha256 87bd4c8bb13e6d4cd550bae07b04e72ab34943bf20027e65a8d6c39bdb03ea45
MATCH

All ten data shard blocks were rebuilt from parity, and the result matches the SHA-256 of the original. The multipart archive rebuilt from the same two drives in the same way, with 26 data shard blocks rebuilt.

The bitrot hashes earn their place when a drive is present but damaged. We put back all four drives and changed a single byte in the third block of drive 3's shard. Without hash checks, the tool joins the damaged shard and the file is silently wrong:

etag 0c55ea9e81d4f74c2dafd51bb4103313 (stored 11088e17e51dd99b01444db857b7543e)
sha256 7b810dbd293f03fa239b2c38b191ffff565c6d00f48be052d8cee849fe131a8f

With the checks on, the damaged block fails its HighwayHash, the tool treats that one shard block as missing, and parity fills the gap:

wrote out/v.csv: 5000050 bytes, 1 data shard block(s) rebuilt from parity
etag 11088e17e51dd99b01444db857b7543e (stored 11088e17e51dd99b01444db857b7543e)
sha256 87bd4c8bb13e6d4cd550bae07b04e72ab34943bf20027e65a8d6c39bdb03ea45

Note the ETag line. For an unencrypted single-part object, the S3 ETag is the MD5 of the content; for a multipart upload it is the MD5 of the parts' MD5s plus - and the part count (7e7816229a484b6a1ccb6376262983d3-3 for the archive). Because it is stored in xl.meta, it is an integrity check from the evidence itself, even when nobody hashed the original.

Versions and delete markers

The bucket had versioning on, so deleting reports/q3.txt did not remove anything. It added a third version, a delete marker, to the same xl.meta:

format 1.3 crc ok: True
{'Type': 2,
 'DelObj': {'ID': b'\xab\x83!\x86\x10~Iu\x8e\x9b\xb6\xd2=]-\xee',
            'MTime': 1793713002802809197,
            'MetaSys': {}},
 'v': 1760549395}

A delete marker has a version ID and a time, and no data. The two earlier versions are still there, newest first, and their inline shards are still in the inline data map, keyed by version ID. An S3 client sees the object as deleted, but each version can be rebuilt by passing its ID:

wrote out/q3-cf9c7b44-dbdc-4e4a-a13d-b179c64e3f0a.txt: 44 bytes, 0 data shard block(s) rebuilt from parity
Q3 report draft v1
Revenue figures pending.
wrote out/q3-b7771e91-0cf1-4294-8b10-c10fa3cc9cd0.txt: 57 bytes, 0 data shard block(s) rebuilt from parity
Q3 report draft v2
Revenue figures confirmed by finance.

A permanent delete (removing a specific version ID, or deleting with versioning off) is different. To see what it leaves, we started MinIO again on the lab drives, deleted the oldest q3.txt version and the only CSV version by ID, and stopped it after about 25 seconds. The q3.txt version was gone from xl.meta, along with its inline data. The CSV folder was gone completely. But MinIO deletes by renaming into a trash folder first:

find d1/.minio.sys/tmp/.trash -type f -not -name ".writable*" -printf "%s %p\n"

352 d1/.minio.sys/tmp/.trash/93ae19a8-33e4-4032-b398-8809929ed677
2500185 d1/.minio.sys/tmp/.trash/8187caf9-3eac-429d-b8d7-6ebe101ff1c7/part.1

The 352-byte file starts 58 4c 32 20 01 00 03 00: it is the CSV's xl.meta under a new name. The part.1 beside it is the 2,500,185-byte shard. Both are renamed to random UUIDs, so the link between them has to come from sizes and from the DDir in the metadata. We copied each drive's pair back into the normal layout, renamed the data directory to the DDir UUID, and the tool rebuilt the deleted CSV with a matching SHA-256.

This window is short. MinIO empties .minio.sys/tmp/.trash on a timer, delete_cleanup_interval, which defaults to 5 minutes, and some deletes, or a drive that is nearly full, purge it at once. On drives taken from a server that stopped soon after a deletion, check the trash folder first. It is worth checking even when nothing was deleted: on our untouched copy it already held an xl.meta.bkp, an earlier copy of a 757-byte internal metadata object that MinIO had rewritten. After it has been emptied, what remains is whatever the filesystem itself has not yet overwritten, which is a job for filesystem carving rather than MinIO parsing.

Where recovery stops

Two features change the stored bytes: compression and encryption. Both show up in MetaSys, and the tool refuses to write either object as if it were the original.

Compression. MinIO compresses objects whose extension or content type matches its settings, using S2, an extension of the Snappy format. Our auth.log is 300,084 bytes, and xl.meta records X-Minio-Internal-compression: klauspost/compress/s2, PartSizes: [52818] and PartASizes: [300084]. The erasure layer works on the 52,818 compressed bytes, so we rebuild those first with --raw:

00000000  ff 06 00 00 53 32 73 54  77 4f 00 44 ce 00 2c 85  |....S2sTwO.D..,.|
00000010  d9 3f b4 a8 12 f0 6b 4f  63 74 20 32 38 20 30 39  |.?....kOct 28 09|

ff 06 00 00 S2sTwO is the S2 stream identifier, and readable log text already shows through the first block. The s2d tool from the same klauspost/compress version MinIO uses (1.18.0) decompresses it to the original:

s2d -c out/auth.log.s2 | sha256sum

0261e9e1d6a4fcbea8314dadbad4e5d52a7e11e2cec16499b9df98e43d2f4c14  -

That is the SHA-256 of the original auth.log, so compression adds one step and nothing more. The ETag of a compressed object is not the MD5 of the original, so use the decompressed hash instead.

Encryption. secure/keys.txt was stored with SSE-S3. Its MetaSys holds a sealed object key, the KMS key name (lab-key) and X-Minio-Internal-Server-Side-Encryption-Seal-Algorithm: DAREv2-HMAC-SHA256. The stored object is 71 bytes for 39 bytes of text:

00000000  20 00 26 00 aa 75 98 49  47 3a 22 40 5a cc 67 69  | .&..u.IG:"@Z.gi|
00000010  00 7c b0 20 cf 1f 5e ec  09 70 d5 38 29 9f 7b 58  |.|. ..^..p.8).{X|
00000020  83 a9 2a 39 ab 5f 1b 48  72 bf 92 b6 cb 79 06 ad  |..*9._.Hr....y..|
00000030  b5 81 94 f8 2f ba c0 9f  7a 58 74 20 2a e0 cc 9d  |..../...zXt *...|
00000040  ad 2c fd 7f 0c 27 ac                              |.,...'.|

This is a DARE 2.0 package: version 0x20, cipher 0x00 (AES-256-GCM), payload length minus one (0x0026, so 39 bytes), a 12-byte nonce whose top bit marks the final package, the ciphertext and a 16-byte authentication tag. 16 + 39 + 16 = 71. The erasure layer still works, so the ciphertext can be rebuilt exactly, but reading it needs the object key, which is sealed with a key from the KMS. Without the KMS key or the client's SSE-C key, recovery stops at intact ciphertext. Record the key ID and the sealed key; they tell whoever holds the KMS which key the object needs.

The recovery script

minio_recover.py puts all of this into one file of about 340 lines. It needs Python 3 with msgpack and xxhash (pip install msgpack xxhash, ideally in a virtual environment). It has two commands:

list – walks the drives, decodes every xl.meta with a valid CRC and prints each version with its time, size and storage type.
recover – rebuilds one version, checking each block's HighwayHash, rebuilding missing or damaged shards from parity, and printing the ETag and SHA-256. --expect compares the result with a known hash, and --raw writes compressed or encrypted objects as stored.

Run against the four drive copies, list gives the whole bucket, including the versions an S3 client would hide:

python3 minio_recover.py list d1 d2 d3 d4

case-files/evidence/evidence-bundle.tar
  9353e6a0-82d0-4cce-aa53-5386027b4707  2026-11-03 13:36:38.522514 UTC  12595257 bytes  3 part(s)
case-files/finance/invoices-2026.csv
  6af706f5-65d9-4aaf-a17f-ae54866d2d4d  2026-11-03 13:36:38.394871 UTC  5000050 bytes   1 part(s)
case-files/logs/auth.log
  61a9f45f-673f-4b4a-927c-a431c725c8dc  2026-11-03 13:36:38.582528 UTC  52818 bytes     1 part(s), compressed
case-files/notes/handover.txt
  54834c21-38c1-4f0f-a832-ca1d89624f70  2026-11-03 13:36:38.292948 UTC  152 bytes       inline
case-files/reports/q3.txt
  ab832186-107e-4975-8e9b-b6d23d5d2dee  2026-11-03 13:36:42.802809 UTC  delete marker
  b7771e91-0cf1-4294-8b10-c10fa3cc9cd0  2026-11-03 13:36:40.746320 UTC  57 bytes        inline
  cf9c7b44-dbdc-4e4a-a13d-b179c64e3f0a  2026-11-03 13:36:38.690016 UTC  44 bytes        inline
case-files/secure/keys.txt
  591e8459-a235-4ae9-bcf2-42f6f7269dc0  2026-11-03 13:36:38.636100 UTC  71 bytes        inline, encrypted

The multipart archive from all four drives:

python3 minio_recover.py recover --object case-files/evidence/evidence-bundle.tar -o out/evidence-bundle.tar --expect 2d31f8c8f1a8e9284c6536575217eb1dc386111026c9ab523c693b4086758000 d1 d2 d3 d4

version 9353e6a0-82d0-4cce-aa53-5386027b4707  2026-11-03 13:36:38.522514 UTC  12595257 bytes  EC 2+2  shards found: [1, 2, 3, 4]
wrote out/evidence-bundle.tar: 12595257 bytes, 0 data shard block(s) rebuilt from parity
etag 7e7816229a484b6a1ccb6376262983d3-3 (stored 7e7816229a484b6a1ccb6376262983d3-3)
sha256 2d31f8c8f1a8e9284c6536575217eb1dc386111026c9ab523c693b4086758000
MATCH

We tested it on every object with all four drives, with drives 3 and 4 removed, and with a corrupted shard, and every plain object matched its original SHA-256. It is written to be read rather than to be fast: the pure-Python HighwayHash checks about 2 MB a second, so the 12 MB archive takes about 13 seconds.

Know its limits before using it on a real case. It reads one erasure set; a multi-pool or multi-set deployment needs its drives grouped first, using format.json. It handles format 1.3 V2Obj versions only, not the older xl.json layout. It assumes the drives agree about each version (the header signature is how to check), and it does not decrypt.

#!/usr/bin/env python3
"""Recover objects from MinIO drive folders without a MinIO server.

Reads xl.meta (format XL2 1.3) and part.N files from the drives of one
erasure set, checks HighwayHash bitrot hashes, rebuilds missing shards with
Reed-Solomon and writes the object out. Tested against MinIO
RELEASE.2025-10-15T17-29-55Z. Needs: pip install msgpack xxhash

  minio_recover.py list DRIVE [DRIVE ...]
  minio_recover.py recover --object BUCKET/KEY [--version ID] -o OUT DRIVE ...
"""
import argparse
import hashlib
import os
import struct
import sys
import uuid
from datetime import datetime, timezone

import msgpack
import xxhash

# --- HighwayHash-256, used by MinIO for bitrot hashes (cmd/bitrot.go) -------
M64 = (1 << 64) - 1
BITROT_KEY = bytes.fromhex(
    "4be734fa8e238acd263e83e6bb968552040f935da39f441497e09d1322de36a0")


def _zipper(v1, v0):
    a0 = ((((v0 & 0xff000000) | (v1 & 0xff00000000)) >> 24)
          | (((v0 & 0xff0000000000) | (v1 & 0xff000000000000)) >> 16)
          | (v0 & 0xff0000) | ((v0 & 0xff00) << 32)
          | ((v1 & 0xff00000000000000) >> 8) | (v0 << 56)) & M64
    a1 = ((((v1 & 0xff000000) | (v0 & 0xff00000000)) >> 24)
          | (v1 & 0xff0000) | ((v1 & 0xff0000000000) >> 16)
          | ((v1 & 0xff00) << 24) | ((v0 & 0xff000000000000) >> 8)
          | ((v1 & 0xff) << 48) | (v0 & 0xff00000000000000)) & M64
    return a1, a0


def highwayhash256(data, key=BITROT_KEY):
    k = struct.unpack("<4Q", key)
    m0 = [0xdbe6d5d5fe4cce2f, 0xa4093822299f31d0, 0x13198a2e03707344, 0x243f6a8885a308d3]
    m1 = [0x3bd39e10cb0ef593, 0xc0acf169b5f18a8c, 0xbe5466cf34e90c6c, 0x452821e638d01377]
    v0 = [m0[i] ^ k[i] for i in range(4)]
    v1 = [m1[i] ^ (((k[i] >> 32) | (k[i] << 32)) & M64) for i in range(4)]

    def update(lanes):
        for i in range(4):
            v1[i] = (v1[i] + m0[i] + lanes[i]) & M64
            m0[i] ^= ((v1[i] & 0xffffffff) * (v0[i] >> 32)) & M64
            v0[i] = (v0[i] + m1[i]) & M64
            m1[i] ^= ((v0[i] & 0xffffffff) * (v1[i] >> 32)) & M64
        for hi, lo, src, dst in ((1, 0, v1, v0), (3, 2, v1, v0), (1, 0, v0, v1), (3, 2, v0, v1)):
            x1, x0 = _zipper(src[hi], src[lo])
            dst[hi] = (dst[hi] + x1) & M64
            dst[lo] = (dst[lo] + x0) & M64

    whole = len(data) - len(data) % 32
    for off in range(0, whole, 32):
        update(struct.unpack_from("<4Q", data, off))
    size = len(data) - whole
    if size:
        def rot(h):
            return ((h << size) | (h >> (32 - size))) & 0xffffffff
        for i in range(4):
            v0[i] = (v0[i] + (size << 32) + size) & M64
            v1[i] = rot(v1[i] & 0xffffffff) | (rot(v1[i] >> 32) << 32)
        rem, packet, mod4 = data[whole:], bytearray(32), size & 3
        packet[:size & ~3] = rem[:size & ~3]
        if size & 16:
            packet[28:32] = rem[size - 4:size]
        elif mod4:
            tail = rem[size & ~3:]
            packet[16:19] = bytes((tail[0], tail[mod4 >> 1], tail[mod4 - 1]))
        update(struct.unpack("<4Q", packet))
    for _ in range(10):
        update([((v0[j] >> 32) | (v0[j] << 32)) & M64 for j in (2, 3, 0, 1)])
    out = []
    for hi, lo in ((1, 0), (3, 2)):
        a3 = (v1[hi] + m1[hi]) & 0x3FFFFFFFFFFFFFFF
        a2 = (v1[lo] + m1[lo]) & M64
        a1 = (v0[hi] + m0[hi]) & M64
        a0 = (v0[lo] + m0[lo]) & M64
        out += [(a0 ^ (a2 << 1) ^ (a2 << 2)) & M64,
                (a1 ^ ((a3 << 1) | (a2 >> 63)) ^ ((a3 << 2) | (a2 >> 62))) & M64]
    return struct.pack("<4Q", *out)


# --- GF(2^8) Reed-Solomon, matching klauspost/reedsolomon's default matrix --
EXP, LOG = [0] * 512, [0] * 256
_x = 1
for _i in range(255):
    EXP[_i] = EXP[_i + 255] = _x
    LOG[_x] = _i
    _x <<= 1
    if _x & 0x100:
        _x ^= 0x11d          # x^8 + x^4 + x^3 + x^2 + 1 (generating polynomial 29)


def gmul(a, b):
    return 0 if a == 0 or b == 0 else EXP[LOG[a] + LOG[b]]


def ginv(a):
    return EXP[255 - LOG[a]]


MUL_TABLE = [bytes(gmul(c, b) for b in range(256)) for c in range(256)]


def mat_mul(a, b):
    return [[_dot(row, [b[k][j] for k in range(len(b))]) for j in range(len(b[0]))] for row in a]


def _dot(xs, ys):
    acc = 0
    for x, y in zip(xs, ys):
        acc ^= gmul(x, y)
    return acc


def mat_invert(m):
    n = len(m)
    a = [list(row) + [int(i == j) for j in range(n)] for i, row in enumerate(m)]
    for col in range(n):
        pivot = next(r for r in range(col, n) if a[r][col])
        a[col], a[pivot] = a[pivot], a[col]
        inv = ginv(a[col][col])
        a[col] = [gmul(v, inv) for v in a[col]]
        for r in range(n):
            if r != col and a[r][col]:
                f = a[r][col]
                a[r] = [v ^ gmul(f, p) for v, p in zip(a[r], a[col])]
    return [row[n:] for row in a]


def encoding_matrix(data, total):
    """Vandermonde matrix with its top square turned into the identity."""
    vm = [[1 if c == 0 else (0 if r == 0 else EXP[(LOG[r] * c) % 255]) for c in range(data)]
          for r in range(total)]
    return mat_mul(vm, mat_invert(vm[:data]))


def combine(coeffs, shards):
    """Sum of coeff * shard over GF(2^8), done a whole shard at a time."""
    acc = 0
    for c, s in zip(coeffs, shards):
        if c:
            acc ^= int.from_bytes(s.translate(MUL_TABLE[c]), "little")
    return acc.to_bytes(len(shards[0]), "little")


def rebuild_data(shards, data, parity):
    """Return the data shards, rebuilding any that are None from the others."""
    if all(s is not None for s in shards[:data]):
        return shards[:data]
    have = [i for i, s in enumerate(shards) if s is not None][:data]
    if len(have) < data:
        raise ValueError(f"only {len(have)} of {data} shards needed are readable")
    matrix = encoding_matrix(data, data + parity)
    decode = mat_invert([matrix[i] for i in have])
    return [shards[i] if shards[i] is not None else combine(decode[i], [shards[h] for h in have])
            for i in range(data)]


# --- xl.meta -----------------------------------------------------------------
def parse_xlmeta(raw):
    if raw[:4] != b"XL2 ":
        raise ValueError("not an xl.meta file (no XL2 magic)")
    major, minor = struct.unpack("<HH", raw[4:8])
    if raw[8] != 0xc6:
        raise ValueError("expected a msgpack bin32 metadata block at offset 8")
    (length,) = struct.unpack(">I", raw[9:13])
    meta = raw[13:13 + length]
    stored_crc = struct.unpack(">I", raw[14 + length:18 + length])[0]
    crc_ok = stored_crc == xxhash.xxh64(meta).intdigest() & 0xffffffff
    up = msgpack.Unpacker(raw=False, strict_map_key=False)
    up.feed(meta)
    up.unpack()                       # header version (3)
    up.unpack()                       # metadata version (3)
    versions = []
    for _ in range(up.unpack()):
        up.unpack()                   # version header: a summary of the version below
        versions.append(msgpack.unpackb(up.unpack(), raw=False, strict_map_key=False))
    inline = {}
    tail = raw[18 + length:]
    if tail[:1] == b"\x01":           # inline data: version byte, then map[version-id]bytes
        inline = msgpack.unpackb(tail[1:], raw=False)
    return {"version": f"{major}.{minor}", "crc_ok": crc_ok, "versions": versions, "inline": inline}


def describe(v):
    obj = v.get("V2Obj") or v.get("DelObj")
    vid = str(uuid.UUID(bytes=obj["ID"])) if any(obj["ID"]) else "null"
    when = datetime.fromtimestamp(obj["MTime"] / 1e9, timezone.utc).strftime("%Y-%m-%d %H:%M:%S.%f UTC")
    if v["Type"] == 2:
        return vid, when, "delete marker", ""
    sysmeta = {k.lower(): val for k, val in (obj.get("MetaSys") or {}).items()}
    notes = ["inline" if "x-minio-internal-inline-data" in sysmeta else f"{len(obj['PartNums'])} part(s)"]
    if "x-minio-internal-compression" in sysmeta:
        notes.append("compressed")
    if any(k.startswith("x-minio-internal-server-side-encryption") for k in sysmeta):
        notes.append("encrypted")
    return vid, when, f"{obj['Size']} bytes", ", ".join(notes)


def object_keys(drives):
    keys = set()
    for drive in drives:
        for root, dirs, files in os.walk(drive):
            dirs[:] = [d for d in dirs if d != ".minio.sys"]
            if "xl.meta" in files:
                keys.add(os.path.relpath(root, drive))
    return sorted(keys)


def read_meta(drive, key):
    try:
        with open(os.path.join(drive, key, "xl.meta"), "rb") as f:
            meta = parse_xlmeta(f.read())
    except (OSError, ValueError):
        return None
    return meta if meta["crc_ok"] else None


def cmd_list(args):
    for key in object_keys(args.drives):
        meta = next(m for m in (read_meta(d, key) for d in args.drives) if m)
        print(key)
        for v in meta["versions"]:
            print("  %-36s  %s  %-15s %s" % describe(v))


# --- recovery ----------------------------------------------------------------
def shard_streams(drives, key, vid):
    """Map shard index (1-based) to (version, inline bytes or data dir path)."""
    found = {}
    for drive in drives:
        meta = read_meta(drive, key)
        for v in (meta or {"versions": []})["versions"]:
            obj = v.get("V2Obj")
            if not obj or str(uuid.UUID(bytes=obj["ID"])) != vid:
                continue
            if vid in meta["inline"]:
                found[obj["EcIndex"]] = (obj, meta["inline"][vid])
            else:
                found[obj["EcIndex"]] = (obj, os.path.join(drive, key, str(uuid.UUID(bytes=obj["DDir"]))))
    return found


def read_part(source, part_num, part_size, data, block_size, verify):
    """Yield (block_length, shard bytes or None) for each erasure block of one part."""
    if isinstance(source, bytes):
        blob = source
    else:
        try:
            with open(os.path.join(source, f"part.{part_num}"), "rb") as f:
                blob = f.read()
        except OSError:
            blob = b""
    pos, left = 0, part_size
    while left > 0:
        block_len = min(block_size, left)
        n = -(-block_len // data)
        digest, shard = blob[pos:pos + 32], blob[pos + 32:pos + 32 + n]
        ok = len(shard) == n and (not verify or highwayhash256(shard) == digest)
        yield block_len, shard if ok else None
        pos, left = pos + 32 + n, left - block_len


def etag_of(data, part_sizes):
    """S3 ETag: MD5 of the object, or MD5 of the part MD5s plus -N for multipart."""
    if len(part_sizes) == 1:
        return hashlib.md5(data).hexdigest()
    md5s, pos = b"", 0
    for size in part_sizes:
        md5s += hashlib.md5(data[pos:pos + size]).digest()
        pos += size
    return f"{hashlib.md5(md5s).hexdigest()}-{len(part_sizes)}"


def cmd_recover(args):
    meta = next((m for m in (read_meta(d, args.object) for d in args.drives) if m), None)
    if not meta:
        sys.exit(f"no readable xl.meta for {args.object}")
    versions = meta["versions"]
    v = versions[0] if not args.version else next(
        (x for x in versions if describe(x)[0] == args.version), None)
    if v is None or v["Type"] != 1:
        sys.exit("that version is a delete marker or does not exist; run 'list' and pass --version")
    vid, when, size, notes = describe(v)
    obj = v["V2Obj"]
    stops = [n for n in ("compressed", "encrypted") if n in notes]
    if stops and not args.raw:
        sys.exit(f"object is {' and '.join(stops)}: the stored bytes are not the original "
                 "(use --raw to write them as stored)")
    data, parity, block = obj["EcM"], obj["EcN"], obj["EcBSize"]
    streams = shard_streams(args.drives, args.object, vid)
    print(f"version {vid}  {when}  {size}  EC {data}+{parity}  shards found: {sorted(streams)}")
    rebuilt = 0
    with open(args.output, "wb") as out:
        for part_num, part_size in zip(obj["PartNums"], obj["PartSizes"]):
            readers = [read_part(streams[i][1], part_num, part_size, data, block, not args.no_verify)
                       if i in streams else None for i in range(1, data + parity + 1)]
            for _ in range(-(-part_size // block)):
                blocks = [next(r) if r else (None, None) for r in readers]
                block_len = next(b[0] for b in blocks if b[0])
                shards = [b[1] for b in blocks]
                rebuilt += sum(s is None for s in shards[:data])
                out.write(b"".join(rebuild_data(shards, data, parity))[:block_len])
    with open(args.output, "rb") as f:
        result = f.read()
    digest = hashlib.sha256(result).hexdigest()
    print(f"wrote {args.output}: {len(result)} bytes, "
          f"{rebuilt} data shard block(s) rebuilt from parity")
    if not stops:
        print(f"etag {etag_of(result, obj['PartSizes'])} (stored {obj['MetaUsr'].get('etag')})")
    print(f"sha256 {digest}")
    if args.expect:
        print("MATCH" if digest == args.expect.lower() else "MISMATCH")
        sys.exit(0 if digest == args.expect.lower() else 1)


def main():
    p = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
    sub = p.add_subparsers(dest="cmd", required=True)
    ls = sub.add_parser("list", help="list objects and their versions")
    ls.add_argument("drives", nargs="+")
    rc = sub.add_parser("recover", help="rebuild one object version")
    rc.add_argument("--object", required=True, help="bucket/key, e.g. case-files/notes/handover.txt")
    rc.add_argument("--version", help="version ID (default: latest)")
    rc.add_argument("-o", "--output", required=True)
    rc.add_argument("--expect", help="SHA-256 the result must match")
    rc.add_argument("--raw", action="store_true", help="write compressed or encrypted bytes as stored")
    rc.add_argument("--no-verify", action="store_true", help="skip HighwayHash bitrot checks")
    rc.add_argument("drives", nargs="+")
    args = p.parse_args()
    cmd_list(args) if args.cmd == "list" else cmd_recover(args)


if __name__ == "__main__":
    main()

Forensic notes

Recovering the bytes is only part of the job. A MinIO deployment holds evidence about itself, and the order in which you collect it matters.

Work on copies. Image each drive and hash the image before anything else, as for any disk. Never start MinIO on original drives: it heals, scans and empties .trash as soon as it runs. Our post on chain of custody for digital evidence covers the records that go with each step.
Record the drive map. Note the deployment id and each drive's this UUID and slot from format.json, against the exhibit number of each image. The slot, not the bay or device name, decides which shard a drive holds.
Use the timestamps that MinIO wrote. MTime is the S3 last-modified time in nanoseconds. Filesystem times on xl.meta change whenever MinIO rewrites the file (a new version, a delete marker, healing), so they date the last metadata write, not the object.
Note the writer. The v field records the MinIO release that wrote each version, which can reveal an upgrade.
Read the history. Every version and delete marker of an object, with its time, is in one xl.meta. A delete marker with no earlier versions behind it can point to a permanent delete; check .minio.sys/tmp/.trash for the pieces.
Look at the rest of .minio.sys. buckets/<bucket>/.metadata.bin holds bucket settings such as versioning, and config/config.json holds the server configuration. Both are objects with their own xl.meta, so the same parser reads them. The configuration and the identity and access management data beside it can include credentials, so treat them as sensitive.
Verify twice. Check each recovered object against the ETag in xl.meta and, where the client has them, against independent hashes. Record both, along with the tool and its version.