|
| 1 | +# Raw image input |
| 2 | + |
| 3 | +For maintainers, and shared by iLEAPP, ALEAPP, RLEAPP, VLEAPP and DLEAPP. |
| 4 | + |
| 5 | +`-t raw` takes a disk image (`.img`, `.dd`, `.bin`, or any numbered `.001` |
| 6 | +segment of a split set) or an EnCase/EWF acquisition (`.E01` and its segments) |
| 7 | +and reads it in place: no mounting, no administrator rights, and no copy of the |
| 8 | +image or of its files anywhere but the files an artifact asks for. Its NTFS, |
| 9 | +FAT32, exFAT, ext2/3/4, HFS+, APFS, QNX6, QNX4, ETFS, EFS and QNX IFS volumes |
| 10 | +are searched directly. |
| 11 | + |
| 12 | +## Where the pieces are |
| 13 | + |
| 14 | +- `scripts/vendor/qnxprobe.py` and `scripts/vendor/ewfprobe.py` are the reader, |
| 15 | + copied verbatim from their repositories and guarded by |
| 16 | + `admin/scripts/check_vendored.py` (see `scripts/vendor/README.md`). Fix them |
| 17 | + upstream and re-vendor; an edit here is reverted by the next sync. |
| 18 | +- `scripts/raw_image.py` holds everything of ours: `FileSeekerRaw`, the constants |
| 19 | + the GUI uses to recognise an image by extension, and `split_image_sibling`. |
| 20 | + **This file is byte-identical in all five cores.** Change it in one, land the |
| 21 | + same bytes in the other four in the same round, and let the parity scanner in |
| 22 | + leapps-org/leapps-parity confirm they match. |
| 23 | +- The entry point adds one `-t` choice, one dispatch branch and a `finally` that |
| 24 | + calls `seeker.cleanup()` on every exit. The GUI maps the conventional image |
| 25 | + extensions onto `raw` and lists them in the file dialog. |
| 26 | +- `admin/test/scripts/test_raw_image_seeker.py` checks staged bytes against |
| 27 | + independent hash lists over the fixtures in `admin/test/data/raw_images/`, |
| 28 | + including an E01 set and a split set built at test time. |
| 29 | + |
| 30 | +## How it reads |
| 31 | + |
| 32 | +`FileSeekerRaw` opens the image through the reader's `open_image()` (which joins |
| 33 | +a split set and reads an EWF set), asks `volumes()` for every volume the reader's |
| 34 | +own report would name, walks each readable volume once for its directory tree, |
| 35 | +and offers the run a member list in the shape the zip seeker offers: one name per |
| 36 | +file and one per directory (with a trailing slash), each prefixed by the volume's |
| 37 | +extraction name, `p<partition>_lba<start>[_<label>]` or `lba0` for a bare volume. |
| 38 | +`search()` matches a pattern against that list with the same `fnmatch` rules as |
| 39 | +the zip seeker and reads only the matches out of the image, so the cost of a run |
| 40 | +is the walk plus the files the artifacts wanted. |
| 41 | + |
| 42 | +Measured on a 238.5 GiB Windows acquisition (PC-MUS-001, 253,032 files): the |
| 43 | +walk took 9 seconds, a pattern resolves in under half a second, and DLEAPP's |
| 44 | +patterns select 694 MiB of the 103.3 GiB of live files. Staging every file to a |
| 45 | +temporary folder first, the design this replaced, would have written those 103 GiB |
| 46 | +before the first artifact ran. |
| 47 | + |
| 48 | +The run log names every volume with its filesystem and size, says which it |
| 49 | +cannot read and why, and warns when the image is shorter than the volumes its |
| 50 | +partition table describes, which is what a lone first segment of a split set |
| 51 | +looks like. A file that runs past the end of the image is named in the log and |
| 52 | +not staged, because a truncated database parses as a smaller one rather than as |
| 53 | +an error. |
| 54 | + |
| 55 | +## Times |
| 56 | + |
| 57 | +NTFS, ext, HFS+ and APFS store instants, and those reach the staged copy's |
| 58 | +mtime and the `FileInfo` the run records. FAT32 and exFAT store a wall-clock |
| 59 | +reading with no zone; the reader hands those back as text. The staged copy's |
| 60 | +mtime is set from that reading in the machine's local zone, exactly as the zip |
| 61 | +seeker sets it from a member's DOS stamp, and the `FileInfo` records no instant |
| 62 | +for it, so no report field carries a zone the evidence never had. |
| 63 | + |
| 64 | +## What it does not do |
| 65 | + |
| 66 | +- It reads live files. Deleted records the reader can recover on NTFS, FAT32 |
| 67 | + and exFAT are not staged. |
| 68 | +- It does not re-root a bare partition image. A raw image of an Android |
| 69 | + `userdata` partition has `data/`, `media/` and `system/` at its root rather |
| 70 | + than under `data/`, and an iOS Data volume has `mobile/` and `containers/` |
| 71 | + rather than `private/var/`. Patterns that begin with `*/` match either way; |
| 72 | + the few that spell out `private/var/` or `data/media/` do not match on a |
| 73 | + bare partition image. Two measured consequences: on a bare Android `userdata` |
| 74 | + image the storage-view dedupe recognises only the `data/data`, `data/user/N` |
| 75 | + and `data_mirror` spellings, so a partition read at its own root is not |
| 76 | + collapsed and per-app row counts multiply; on a bare iOS Data volume the |
| 77 | + `iosfilesystemevents` family and `diagnosticlogdevents` find nothing. A |
| 78 | + full-disk or full-`/data` image carries the expected root and neither happens. |
| 79 | +- F2FS, the filesystem most current Android userdata partitions use, is not |
| 80 | + one the reader walks; such a volume is listed as not recognised. |
| 81 | +- An encrypted volume (Android file-based encryption, iOS data protection, |
| 82 | + FileVault, BitLocker) reads, but its names or contents are ciphertext. |
| 83 | + |
| 84 | +## Comparing a raw run against a zip run |
| 85 | + |
| 86 | +The same extraction read as a zip and as a raw image of the same file tree does |
| 87 | +not produce byte-identical reports, and the differences are in the input format |
| 88 | +and in a few artifacts, not in what the seeker staged. A raw run over a full-disk |
| 89 | +or full-`/data` image stages every file an artifact asked for, so the run log's |
| 90 | +`Not staged` count is zero. The reports still differ in three ways worth knowing, |
| 91 | +none of them a file the seeker got wrong: |
| 92 | + |
| 93 | +- Timestamps. A zip member carries an MS-DOS stamp: two-second granularity, no |
| 94 | + sub-second, no zone. A filesystem carries the real instant. So a column that |
| 95 | + surfaces a staged file's own time reads a second or two apart between the two |
| 96 | + routes, and the raw route is the more faithful of the two. |
| 97 | +- First-match order. A few artifacts read one file out of several equivalent |
| 98 | + copies and take the first the seeker returns. The zip, tar and raw seekers each |
| 99 | + list a directory in their own order, so which copy wins can differ. On Android |
| 100 | + `emulatedSmeta` reads one user's `external.db`, `installedappsGass` labels its |
| 101 | + source by the view it read, and `walstrings` numbers its rows by arrival. This |
| 102 | + is a property of those artifacts and shows between the zip and tar routes too. |
| 103 | +- Companion files beside the zip. Some artifacts read a file a tool exported next |
| 104 | + to the acquisition rather than inside it. iOS keychain artifacts read a |
| 105 | + `<udid>_keychain.plist` sitting beside the zip on disk, found only when the |
| 106 | + input is that zip. A raw image is the filesystem itself and carries no such |
| 107 | + companion, so those artifacts report nothing from it. |
| 108 | + |
| 109 | +## Frozen builds |
| 110 | + |
| 111 | +The reader is imported as a module, so a PyInstaller build carries it like any |
| 112 | +other module; nothing is spawned. `test_builds.yml` runs the frozen CLI with |
| 113 | +`-t raw` on the NTFS fixture and requires the run log to show the walk. An |
| 114 | +earlier design that ran the reader as a subprocess through `sys.executable` |
| 115 | +could not work frozen, because in a bundle that is the tool itself. |
| 116 | + |
| 117 | +## The corpus validator |
| 118 | + |
| 119 | +`admin/scripts/validate_sample_data.py --run` runs a registry entry through |
| 120 | +`-t raw` when the entry carries `"input_type": "raw"`. A raw image is never |
| 121 | +guessed from its extension, because a registry can hold `.bin` files that are |
| 122 | +readable disk images and `.bin` files that are chip-level dumps no walker opens. |
0 commit comments