go-diskfs

go-diskfs is a go library for performing manipulation of disks, disk images and filesystems natively in go.

You can do nearly everything that go-diskfs provides using shell tools like gdisk/fdisk/mkfs.vfat/mtools/sgdisk/sfdisk/dd. However, these have the following limitations:

• they need to be installed on your system
• you need to fork/exec to the command (and possibly a shell) to run them
• some are difficult to run without mounting disks, which may not be possible or may be risky in your environment, and almost certainly will require root privileges
• you do not want to launch a VM to run the excellent libguestfs and it may not be installed

go-diskfs performs all modifications natively in go, without mounting any disks.

Usage

Note: detailed go documentation is available at godoc.org.

Concepts

go-diskfs has a few basic concepts:

• Backend
• Disk
• Partition
• Filesystem

Backend

Backend is a (relatively) thin layer which abstracts low-level read/write operations. Through a backend you can seamlessly operate different disk formats.

Currently there is only one implementation - file.

Use file backend to access block devices and raw image files.

Disk

A disk represents either a file or block device that you access and manipulate. With access to the disk, you can:

• read, modify or create a partition table
• open an existing or create a new filesystem

Partition

A partition is a slice of a disk, beginning at one point and ending at a later one. You can have multiple partitions on a disk, and a partition table that describes how partitions are laid out on the disk.

Filesystem

A filesystem is a construct that gives you access to create, read and write directories and files.

You do not need a partitioned disk to work with a filesystem; filesystems can be an entire disk, just as they can be an entire block device. However, they also can be in a partition in a disk

Working With a Disk

Before you can do anything with a disk - partitions or filesystems - you need to access it.

• If you have an existing disk or image file, you Open() it
• If you are creating a new one, usually just disk image files, you Create() it

The disk will be opened read-write, with exclusive access. If it cannot do either, it will fail.

Once you have a Disk, you can work with partitions or filesystems in it.

Partitions on a Disk

The following are the partition actions you can take on a disk:

• GetPartitionTable() - if one exists. Will report the table layout and type.
• Partition() - partition the disk, overwriting any previous table if it exists

As of this writing, supported partition formats are Master Boot Record (mbr) and GUID Partition Table (gpt).

Filesystems on a Disk

Once you have a valid disk, and optionally partition, you can access filesystems on that disk image or partition.

• CreateFilesystem() - create a filesystem in an individual partition or the entire disk
• GetFilesystem() - access an existing filesystem in a partition or the entire disk

As of this writing, supported filesystems include FAT32 and ISO9660 (a.k.a. .iso).

With a filesystem in hand, you can create, access and modify directories and files.

• Mkdir() - make a directory in a filesystem
• Readdir() - read all of the entries in a directory
• OpenFile() - open a file for read, optionally write, create and append

Note that OpenFile() is intended to match os.OpenFile and returns a godiskfs.File that closely matches os.File

With a File in hand, you then can:

• Write(p []byte) to the file
• Read(b []byte) from the file
• Seek(offset int64, whence int) to set the next read or write to an offset in the file

Read-Only Filesystems

Some filesystem types are intended to be created once, after which they are read-only, for example ISO9660/.iso and squashfs.

godiskfs recognizes read-only filesystems and limits working with them to the following:

• You can GetFilesystem() a read-only filesystem and do all read activities, but cannot write to them. Any attempt to Mkdir() or OpenFile() in write/append/create modes or Write() to the file will result in an error.
• You can CreateFilesystem() a read-only filesystem and write anything to it that you want. It will do all of its work in a "scratch" area, or temporary "workspace" directory on your local filesystem. When you are ready to complete it, you call Finalize(), after which it becomes read-only. If you forget to Finalize() it, you get... nothing. The Finalize() function exists only on read-only filesystems.

FAT: Loading the Table or Reading It on Demand

fat12.Read, fat16.Read and fat32.Read load the whole file allocation table (FAT) into memory when they read the filesystem. ReadWithOptions with WithLazyFAT() reads each FAT entry from the backend when it is needed instead:

fs, err := fat32.ReadWithOptions(b, size, start, blocksize, fat32.WithLazyFAT())

Both tables read and write the same files, and the same changes made through either leave byte-identical images. They differ in what they cost and in what they see:

• The in-memory table costs a full read of the FAT when the filesystem is read, and then every cluster lookup is a slice index. It suits a filesystem that stays open for a lot of work: walking many files, reading large files in small pieces, writing many files.
• The lazy table costs a few small reads when the filesystem is read, and then one small read per cluster lookup. It suits opening a filesystem to read or write a file or two, and reading a volume that something else writes: every lookup sees the FAT as it is on the backend at that moment. Nothing coordinates two writers, so do not write a filesystem that something else writes at the same time.

File.Read rebuilds the file's cluster chain on every call, so with the lazy table read a large file in as few calls as you can.

Measured on a Raspberry Pi Zero W, with a 5 GiB FAT32 volume with 4 KiB clusters in an image file on its SD card:

	in-memory table	lazy table
read the filesystem	309 ms	0.5 ms
read the filesystem, then write a 100 kB file	339 ms	49 ms
already read: read a 50 MiB file in one Read	1.1 s	2.3 s
already read: read 4 MiB of it in 32 KiB Reads	0.7 s	45.5 s
already read: walk 2000 files in 20 directories	346 ms	373 ms
already read: write 50 files of 100 kB, FAT 90% allocated	1.8 s	4.4 s

FAT12 and FAT16 tables are small, at most about 6 KiB and 128 KiB, so for them the lazy table matters for what it sees rather than for speed.

Eager reads (fat12.Read, fat16.Read and fat32.Read) compare the two copies of the FAT and refuse a filesystem where they differ. Lazy reads skip that full comparison to keep opening fast. CheckFAT() compares the two copies on the backend at any time without loading them.

Example

There are examples in the examples/ directory. See for example how to create a fully bootable EFI disk image.

Tests

There are two ways to run tests: unit and integration (somewhat loosely defined).

• Unit: these tests run entirely within the go process, primarily test unexported and some exported functions, and may use pre-defined test fixtures in a directory's testdata/ subdirectory. By default, these are run by running go test ./... or just make unit_test.
• Integration: these test the exported functions and their ability to create or manipulate correct files. They are validated by running a docker container with the right utilities to validate the output. These are run by running TEST_IMAGE=diskfs/godiskfs go test ./... or just make test. The value of TEST_IMAGE will be the image to use to run tests.

For integration tests to work, the correct docker image must be available. You can create it by running make image. Check the Makefile to see the docker build command used to create it. Running make test automatically creates the image for you.

Integration Test Image

The integration test image contains the various tools necessary to test images: mtools, fdisk, gdisk, etc. It works on precisely one file at a time. In order to avoid docker volume mounting limitations with various OSes, instead of mounting the image -v, it expects to receive the image as a stdin stream, and saves it internally to the container as /file.img.

For example, to test the existence of directory /abc on file $PWD/foo.img:

cat $PWD/foo.img | docker run -i --rm $INT_IMAGE mdir -i /file.img /abc

Plans

Future plans are to add the following:

• embed boot code in mbr e.g. altmbr.bin (no need for gpt since an ESP with /EFI/BOOT/BOOT<arch>.EFI will boot)
• Rock Ridge sparse file support - supports the flag, but not yet reading or writing
• squashfs sparse file support - currently treats sparse files as regular files
• qcow disk format