{{ message }}
Repository navigation
Tags: firecracker-microvm/firecracker
Tags
Added * [#5891](#5891): Added support for virtio device reset. * [#5983](#5983): Add two optional metrics fields, set via `PUT /metrics` or a config file: `emit_id` emits the microVM instance id under a top-level `id` field, and `properties` emits operator-defined key-value pairs under a top-level `properties` field. Each is opt-in and independent. See [metrics documentation](docs/metrics.md). * [#6003](#6003): Added a new option `Transparent` for the `huge_pages` setting. If set, Firecracker will use transparent huge pages for the guest memory via `madvise(MADV_HUGEPAGE)`. Guest memory must be a multiple of 2MB when using this option. * [#6013](#6013), [#6017](#6017), [#6021](#6021): Added official support for 6.18 microVM guest kernels. * [#6037](#6037): Added support for booting `bzImage` guest kernels on x86_64, in addition to the existing uncompressed ELF (`vmlinux`) images. The kernel image format is detected automatically, so no configuration change is required. * [#6064](#6064): Added the `huge_pages` field to `PUT /snapshot/load`, allowing the restored microVM to reuse the snapshot's host page configuration or select `None`, `Transparent`, or `2M`. * [#6109](#6109): Added an optional `sync_snapshot_files` field to the `PUT /snapshot/create` API, defaulting to `true`. It controls whether the snapshot state and guest memory files are synced to disk (`fsync`) before the request returns. Setting it to `false` returns without waiting for the writeback, making snapshot creation faster at the cost of durability across a host crash; the data stays in the host page cache, so same-host reads still see the full contents. Block device backing files are always `fsync`'d regardless. See [snapshot documentation](docs/snapshotting/snapshot-support.md). * [#6116](#6116): On aarch64, enable `KVM_CAP_ARM_WRITABLE_IMP_ID_REGS` when the host kernel offers it (Linux 6.15 and later), adding support for custom CPU templates that modify the implementation ID registers (`MIDR_EL1`, `REVIDR_EL1`, `AIDR_EL1`). Such templates previously failed at boot with `Failed to set register ... Invalid argument` because KVM rejects the write unless the capability is enabled on the VM. * [#6145](#6145): Added official support for AWS Graviton5 (m9g.metal-48xl) instances. * [#6098](#6098): Added new VIRTIO_BLK_F_BLK_SIZE and VIRTIO_BLK_F_TOPOLOGY features to the virtio-block device. More information is in the new [block](docs/block.md) documentation. * [#6142](#6142): Add opt-in virtio-blk discard support for writable `Sync` IO engine drives through the `discard` drive configuration field. See the [block discard documentation](docs/api_requests/block-discard.md). Changed * [#6115](#6115): Changed the T2S CPU template to set the `FB_CLEAR` bit of `MSR_IA32_ARCH_CAPABILITIES` to 1. This lets guest kernels recognize that the `VERW` instruction clears fill buffers, including on host kernels before v6.4 that cannot expose `FLUSH_L1D`. * [#6172](#6172): Gate CLIRD_EL1 override to only happen on host kernels equal or newer than 6.10 release. This aliviates the guest performance regression seen on host kernels in range \[6.3..6.10) correlated to incorrect cpu cache topology presented to the guest. Deprecated Removed Fixed * [#6100](#6100): Fixed the vsock device permanently suppressing RX (host-to-guest) delivery after a bare pause/resume cycle (`PATCH /vm` with `Paused` then `Resumed`, without a snapshot). The resume kick armed the `TRANSPORT_RESET` RX gate even though no reset event had been sent, so the guest could never acknowledge it and every new host-initiated connection made after the resume hung forever. The gate is now part of the persisted device state and the resume kick respects it instead of arming it. * [#6174](#6174): Fixed `virtio-mem` leaving unplugged memory writable by the VMM, and potentially unmapped, on microVMs restored from a snapshot memory file. Firecracker now maps each slot straight to the protection it should have, with the `MAP_NORESERVE` flag, and aborts if the re-map fails. * [#6174](#6174): Fixed `virtio-mem` discarding the whole hotpluggable region on every `UNPLUG_ALL` request, even when nothing was plugged. The discard is now skipped when the range has no plugged blocks. * [#6176](#6176): Fixed `virtio-mem` leaving its block accounting inconsistent with the KVM memory slots if a plug or unplug request failed part way through. Firecracker now commits each slot's state only after its KVM update succeeds, so a partial failure leaves the block state and the KVM slots reflecting exactly the slots that were updated. * [#5956](#5956): Fixed a TOCTOU race in the aarch64 jailer when setting ownership of the CPU cache and `MIDR_EL1` information files copied into the chroot. * [#6076](#6076): Fixed memory hotplug sizes silently wrapping when converted from MiB to bytes. `requested_size_mib` on `PATCH /hotplug/memory` and `total_size_mib`, `block_size_mib` and `slot_size_mib` on `PUT /hotplug/memory` are now bounded to 32 bits. Values above that previously wrapped to a smaller byte count, so a large enough request was accepted as a 0-byte region instead of being rejected. * [#6031](#6031), [#6041](#6041), [#6077](#6077): Fixed the vsock device re-arming its host-stream `EPOLLIN` interest while received data was still awaiting a guest RX buffer, which could busy-spin the event thread until the guest posted buffers. The device now drains the host stream fully across successive RX operations instead of one packet per event loop iteration, reducing the host-side CPU cost per gigabit of host-to-guest traffic by up to ~50% and improving host-to-guest throughput by ~44% (median) in most configurations. On single-vcpu microVMs on the newest Intel hosts (m7i, m8i), host-to-guest throughput may instead decrease by ~15% (median), where the single guest vCPU rather than the host becomes the bottleneck. Terminating a connection now also discards its TX buffer, so the device stops advertising `EPOLLOUT` for a host stream it will never write to again, which could otherwise busy-spin the event thread indefinitely. * [#6086](#6086), [#6143](#6143): Fixed a deadlock in the logger: a signal handler that logs while the interrupted thread is already logging would hang the VMM and its API socket. This is reachable whenever a `SIGPIPE` is raised by Firecracker's own log write, for example when logging to `stdout` and the reader of that pipe exits. The logger now uses an `RwLock`, and `sigpipe_handler` only increments the `signals.sigpipe` metric instead of logging, so it no longer emits a `Received signal 13, code 0.` line. * [#6120](#6120): Fixed a bug caused by a KVM behavior change introduced in Linux 6.13, which requires guest CPUID to be set before userspace reads CPUID-dependent MSRs. On x86_64 with Linux 6.18 host kernels, Firecracker read the base values of those MSRs before setting guest CPUID, so KVM returned zero for such MSRs and CPU templates consequently used zero for bits configured for passthrough. Firecracker now sets guest CPUID before reading MSRs to be modified by CPU templates, so passthrough bits retain their KVM-provided values and features such as eIBRS remain exposed to the guest.
Added Changed * [#6172](#6172): Gate CLIRD_EL1 override to only happen on host kernels equal or newer than 6.10 release. This aliviates the guest performance regression seen on host kernels in range \[6.3..6.10) correlated to incorrect cpu cache topology presented to the guest. * [#6100](#6100): Bumped the snapshot format version. Snapshots created by this version cannot be loaded by 1.16.0 or 1.16.1, and snapshots created by those versions cannot be loaded by this one. Deprecated Removed Fixed * [#6100](#6100): Fixed the vsock device permanently suppressing RX (host-to-guest) delivery after a bare pause/resume cycle (`PATCH /vm` with `Paused` then `Resumed`, without a snapshot). The resume kick armed the `TRANSPORT_RESET` RX gate even though no reset event had been sent, so the guest could never acknowledge it and every new host-initiated connection made after the resume hung forever. The gate is now part of the persisted device state and the resume kick respects it instead of arming it. * [#6174](#6174): Fixed `virtio-mem` leaving unplugged memory writable by the VMM, and potentially unmapped, on microVMs restored from a snapshot memory file. Firecracker now maps each slot straight to the protection it should have, with the `MAP_NORESERVE` flag, and aborts if the re-map fails. * [#6174](#6174): Fixed `virtio-mem` discarding the whole hotpluggable region on every `UNPLUG_ALL` request, even when nothing was plugged. The discard is now skipped when the range has no plugged blocks. * [#6176](#6176): Fixed `virtio-mem` leaving its block accounting inconsistent with the KVM memory slots if a plug or unplug request failed part way through. Firecracker now commits each slot's state only after its KVM update succeeds, so a partial failure leaves the block state and the KVM slots reflecting exactly the slots that were updated. * [#5956](#5956): Fixed a TOCTOU race in the aarch64 jailer when setting ownership of the CPU cache and `MIDR_EL1` information files copied into the chroot. * [#6076](#6076): Fixed memory hotplug sizes silently wrapping when converted from MiB to bytes. `requested_size_mib` on `PATCH /hotplug/memory` and `total_size_mib`, `block_size_mib` and `slot_size_mib` on `PUT /hotplug/memory` are now bounded to 32 bits. Values above that previously wrapped to a smaller byte count, so a large enough request was accepted as a 0-byte region instead of being rejected. * [#6031](#6031), [#6041](#6041), [#6077](#6077): Fixed the vsock device re-arming its host-stream `EPOLLIN` interest while received data was still awaiting a guest RX buffer, which could busy-spin the event thread until the guest posted buffers. The device now drains the host stream fully across successive RX operations instead of one packet per event loop iteration, reducing the host-side CPU cost per gigabit of host-to-guest traffic by up to ~50% and improving host-to-guest throughput by ~44% (median) in most configurations. On single-vcpu microVMs on the newest Intel hosts (m7i, m8i), host-to-guest throughput may instead decrease by ~15% (median), where the single guest vCPU rather than the host becomes the bottleneck. Terminating a connection now also discards its TX buffer, so the device stops advertising `EPOLLOUT` for a host stream it will never write to again, which could otherwise busy-spin the event thread indefinitely. * [#6086](#6086), [#6143](#6143): Fixed a deadlock in the logger: a signal handler that logs while the interrupted thread is already logging would hang the VMM and its API socket. This is reachable whenever a `SIGPIPE` is raised by Firecracker's own log write, for example when logging to `stdout` and the reader of that pipe exits. The logger now uses an `RwLock`, and `sigpipe_handler` only increments the `signals.sigpipe` metric instead of logging, so it no longer emits a `Received signal 13, code 0.` line. * [#6120](#6120): Fixed a bug caused by a KVM behavior change introduced in Linux 6.13, which requires guest CPUID to be set before userspace reads CPUID-dependent MSRs. On x86_64 with Linux 6.18 host kernels, Firecracker read the base values of those MSRs before setting guest CPUID, so KVM returned zero for such MSRs and CPU templates consequently used zero for bits configured for passthrough. Firecracker now sets guest CPUID before reading MSRs to be modified by CPU templates, so passthrough bits retain their KVM-provided values and features such as eIBRS remain exposed to the guest.
Fixed * [#5959](#5959): Reverted the use of `O_NOFOLLOW` for the jailer's cgroup and network namespace file operations, so symlinks are again allowed in these paths. * [#5958](#5958): Fixed a bug that caused vsock guest-to-host connections to time out after snapshot restore, triggered by taking a snapshot with a TX descriptor in-flight. On restore the device now replays the TX queue notification so in-flight TX descriptors are re-processed and notification suppression is re-armed.
Added * [#5786](#5786): Added developer preview support for hotplugging and hot-unplugging PCI virtio devices (block, pmem, net) on a running microVM. The guest must manually rescan the PCI bus after hotplug and remove the device before unplug since no automatic notification mechanism is implemented yet. More information can be found in the [Device Hotplugging](docs/device-hotplug.md) documentation page. * [#5323](#5323): Add support for Vsock Unix domain socket path overriding on snapshot restore. More information can be found in the [docs](docs/vsock.md/#unix-domain-socket-renaming). * [#5824](#5824): Add optional rate limiting to serial console output, configurable via the `rate_limiter` field on `PUT /serial`. A new metric is exposed under `uart`: `rate_limiter_dropped_bytes`. * [#5799](#5799): Add per-callsite rate limiting for error, warn, and info level log messages. Each callsite independently allows up to 10 messages per 5-second window. When logging resumes after suppression, a warn-level summary reports the count of suppressed messages. A new `rate_limited_log_count` metric tracks the total number of suppressed messages. * [#5789](#5789): Add rate-limiter support to virtio-pmem device to allow control over I/O bandwidth generated by the FLUSH requests from the guest. * [#5872](#5872): Add notification suppression support in the virtio-vsock device via the EVENT_IDX virtio feature to reduce device overhead. * [#5828](#5828): Advertise MTU to the guest via `VIRTIO_NET_F_MTU` using a new optional `mtu` field in the `network-interfaces` API. When set, a compatible guest driver will configure the interface with the specified MTU. * [#5906](#5906): Add `rng-seed` FDT node for aarch64 guests which provides an initial random seed for the guest to use. This helps older aarch64 machines which do not have hardware random generators. * Added support for Linux 6.18 host kernels alongside the existing 5.10 and 6.1 host kernels. See the [kernel support policy](docs/kernel-policy.md) for details. Changed Deprecated Removed Fixed * [#5762](#5762): Cap virtio-rng per-request entropy to 64 KiB. Previously, a guest could construct a descriptor chain that caused Firecracker to allocate more host memory than the guest actually provided, potentially leading to excessive host memory consumption. * [#5760](#5760): Fixed HID (Hardware ID) of VMGenID device so that it aligns with the upstream Linux kernel. This caused the driver not to be bound correctly to the device prior to Linux kernel 6.10. * [#5764](#5764): Fixed a bug that caused the guest UART driver to get stuck and stop transmitting after snapshot restore. The bug was triggered by taking a snapshot while a serial transmission was taking place. On restore the driver would wait for a TX interrupt that would never arrive and no output would appear in the serial console. * [#5780](#5780): Fixed missing `/sys/devices/system/cpu/cpu*/cache/*` in aarch64 guests when running on host kernels >= 6.3 with guest kernels >= 6.1.156. * [#5793](#5793): Fixed virtio-mem plug/unplug skipping KVM slot updates for memory blocks not aligned to a slot boundary. On plug, this could leave hotplugged memory inaccessible to the guest. On unplug, the guest could retain access to memory that Firecracker considered freed. * [#5794](#5794): Bound balloon statistics descriptor length to prevent a guest-controlled oversized descriptor from temporarily stalling the VMM event loop. Only affects microVMs with `stats_polling_interval_s > 0`. * [#5809](#5809): Fixed a bug on host Linux >= 5.16 for x86_64 guests using the `kvm-clock` clock source causing the monotonic clock to jump on restore by the wall-clock time elapsed since the snapshot was taken. Users using `kvm-clock` that want to explicitly advance the clock with `KVM_CLOCK_REALTIME` can opt back in using the new `clock_realtime` flag in `LoadSnapshot` API. * [#5738](#5738): Fixed x86_64 snapshot serialization to cover the full KVM custom MSR range (0x4b564d00-0x4b564dff) instead of a small subset. Previously, some KVM MSRs such as MSR_KVM_ASYNC_PF_INT and MSR_KVM_ASYNC_PF_ACK were missing from snapshots, which could cause issues on restore. * [#5818](#5818): Enforce the virtio device initialization sequence in the PCI transport, matching the existing MMIO transport behavior. The PCI transport now validates device status transitions, rejects queue configuration writes outside the FEATURES_OK to DRIVER_OK window, rejects feature negotiation outside the DRIVER state, blocks re-initialization after a failed reset, and sets DEVICE_NEEDS_RESET when device activation fails. * [#5818](#5818): Reject device status writes that clear previously set bits in the MMIO transport, except for reset. * [#5884](#5884): Corrected the OpenAPI spec for `PATCH /balloon/hinting/start` and `PATCH /balloon/hinting/stop` to declare `204 No Content` instead of `200`, matching the actual runtime response. * [#5882](#5882): Fixed a race in the vsock device where, after snapshot restore, the RX queue could deliver data to the guest before it had acknowledged the TRANSPORT_RESET event, causing established connections to break.
Fixed * [#5762](#5762): Cap virtio-rng per-request entropy to 64 KiB. Previously, a guest could construct a descriptor chain that caused Firecracker to allocate more host memory than the guest actually provided, potentially leading to excessive host memory consumption. * [#XXXX](https://github.com/firecracker-microvm/firecracker/pull/XXXX): Enforce the virtio device initialization sequence in the PCI transport, matching the existing MMIO transport behavior. The PCI transport now validates device status transitions, rejects queue configuration writes outside the FEATURES_OK to DRIVER_OK window, rejects feature negotiation outside the DRIVER state, blocks re-initialization after a failed reset, and sets DEVICE_NEEDS_RESET when device activation fails. * [#XXXX](https://github.com/firecracker-microvm/firecracker/pull/XXXX): Reject device status writes that clear previously set bits in the MMIO transport, except for reset. * [#5780](#5780): Fixed missing `/sys/devices/system/cpu/cpu*/cache/*` in aarch64 guests when running on host kernels >= 6.3 with guest kernels >= 6.1.156. * [#5793](#5793): Fixed virtio-mem plug/unplug skipping KVM slot updates for memory blocks not aligned to a slot boundary. On plug, this could leave hotplugged memory inaccessible to the guest. On unplug, the guest could retain access to memory that Firecracker considered freed. * [#5794](#5794): Bound balloon statistics descriptor length to prevent a guest-controlled oversized descriptor from temporarily stalling the VMM event loop. Only affects microVMs with `stats_polling_interval_s > 0`. * [#5809](#5809): Fixed a bug on host Linux >= 5.16 for x86_64 guests using the `kvm-clock` clock source causing the monotonic clock to jump on restore by the wall-clock time elapsed since the snapshot was taken. Users using `kvm-clock` that want to explicitly advance the clock with `KVM_CLOCK_REALTIME` can opt back in using the new `clock_realtime` flag in `LoadSnapshot` API.
Fixed * [#5762](#5762): Cap virtio-rng per-request entropy to 64 KiB. Previously, a guest could construct a descriptor chain that caused Firecracker to allocate more host memory than the guest actually provided, potentially leading to excessive host memory consumption. * [#XXXX](https://github.com/firecracker-microvm/firecracker/pull/XXXX): Enforce the virtio device initialization sequence in the PCI transport, matching the existing MMIO transport behavior. The PCI transport now validates device status transitions, rejects queue configuration writes outside the FEATURES_OK to DRIVER_OK window, rejects feature negotiation outside the DRIVER state, blocks re-initialization after a failed reset, and sets DEVICE_NEEDS_RESET when device activation fails. * [#XXXX](https://github.com/firecracker-microvm/firecracker/pull/XXXX): Reject device status writes that clear previously set bits in the MMIO transport, except for reset. * [#5780](#5780): Fixed missing `/sys/devices/system/cpu/cpu*/cache/*` in aarch64 guests when running on host kernels >= 6.3 with guest kernels >= 6.1.156. * [#5793](#5793): Fixed virtio-mem plug/unplug skipping KVM slot updates for memory blocks not aligned to a slot boundary. On plug, this could leave hotplugged memory inaccessible to the guest. On unplug, the guest could retain access to memory that Firecracker considered freed. * [#5794](#5794): Bound balloon statistics descriptor length to prevent a guest-controlled oversized descriptor from temporarily stalling the VMM event loop. Only affects microVMs with `stats_polling_interval_s > 0`. * [#5809](#5809): Fixed a bug on host Linux >= 5.16 for x86_64 guests using the `kvm-clock` clock source causing the monotonic clock to jump on restore by the wall-clock time elapsed since the snapshot was taken. Users using `kvm-clock` that want to explicitly advance the clock with `KVM_CLOCK_REALTIME` can opt back in using the new `clock_realtime` flag in `LoadSnapshot` API.
Added * [#5510](#5510), [#5593](#5593), [#5564](#5564): Add support for the [VMClock device](https://uapi-group.org/specifications/specs/vmclock). The implementation supports the snapshot safety features proposed [here](https://lore.kernel.org/lkml/20260107132514.437-1-bchalios@amazon.es/), but doesn't provide currently any clock-specific information for helping the guest synchronize its clocks. More information can be found in [docs](docs/snapshotting/snapshot-support.md#userspace-notifications-of-loading-virtual-machine-snapshots). Changed * [#5564](#5564): which added support for VMClock, uses one extra GSI for the VMClock device itself which reduces the available GSIs for VirtIO devices. New maximum values is 92 devices on Aarch64 and 17 devices on x86. * [#5631](#5631): Update binary copy process inside Jailer to disallow symlinks and hardlinks at the destination path and change ownership of the copied binary to the specified uid/gid. Deprecated Removed Fixed * [#5698](#5698): Fixed the possible ENXIO error which could occur during file open operation if the underlying file is FIFO without active readers already attached. * [#5688](#5688): Fixed vsock local port reuse across snapshot restore by saving the last used local port into the snapshot, so users need to regenerate snapshots. * [#5705](#5705): Fixed a bug that caused Firecracker to corrupt the memory files of differential snapshots for VMs with multiple memory slots. This affected VMs using memory hot-plugging or any x86 VMs with a memory size larger than 3GiB. * [#5739](#5739): Fixed validation of TCP SYN options length when MMDS is enabled.
Fixed * [#5698](#5698): Fixed the possible ENXIO error which could occur during file open operation if the underlying file is FIFO without active readers already attached. * [#5705](#5705): Fixed a bug that caused Firecracker to corrupt the memory files of differential snapshots for VMs with multiple memory slots. This affected VMs using memory hot-plugging or any x86 VMs with a memory size larger than 3GiB.
PreviousNext