🚀 firecracker-microvm/firecracker - Release Notes

### Added

* [#4987](https://github.com/firecracker-microvm/firecracker/pull/4987): Reset physical counter register (`CNTPCT_EL0`) on VM startup. This avoids VM reading the host physical counter value. This is only possible on 6.4 and newer kernels. For older kernels physical counter will still be passed to the guest unmodified. See more info [here](https://github.com/firecracker-microvm/firecracker/blob/main/docs/prod-host-setup.md#arm-only-vm-physical-counter-behaviour)
* [#5088](https://github.com/firecracker-microvm/firecracker/pull/5088): Added AMD Genoa as a supported and tested platform for Firecracker.

### Changed

* [#4913](https://github.com/firecracker-microvm/firecracker/pull/4913): Removed unnecessary fields (`max_connections` and `max_pending_resets`) from the snapshot format, bumping the snapshot version to 5.0.0. Users need to regenerate snapshots.
* [#4926](https://github.com/firecracker-microvm/firecracker/pull/4926): Replace underlying implementation for seccompiler from in house one in favor of `libseccomp` which produces smaller and more optimized BPF code.

### Fixed

* [#4921](https://github.com/firecracker-microvm/firecracker/pull/4921): Fixed swagger `CpuConfig` definition to include missing aarch64-specific fields.
* [#4916](https://github.com/firecracker-microvm/firecracker/pull/4916): Fixed `IovDeque` implementation to work with any host page size. This fixes virtio-net device on non 4K host kernels.
* [#4991](https://github.com/firecracker-microvm/firecracker/pull/4991): Fixed `mem_size_mib` and `track_dirty_pages` being mandatory for all `PATCH /machine-config` requests. Now, they can be omitted which leaves these parts of the machine configuration unchanged.
* [#5007](https://github.com/firecracker-microvm/firecracker/pull/5007): Fixed watchdog softlockup warning on x86_64 guests when a vCPU is paused during GDB debugging.
* [#5021](https://github.com/firecracker-microvm/firecracker/pull/5021) If a balloon device is inflated post UFFD-backed snapshot restore, Firecracker now causes `remove` UFFD messages to be sent to the UFFD handler. Previously, no such message would be sent.
* [#5034](https://github.com/firecracker-microvm/firecracker/pull/5034): Fix an integer underflow in the jailer when computing the value it passes to Firecracker's `--parent-cpu-time-us` values, which caused development builds of Firecracker to crash (but production builds were unaffected as underflows do not panic in release mode).
* [#5045](https://github.com/firecracker-microvm/firecracker/pull/5045): Fixed an issue where firecracker intermittently receives SIGHUP when using jailer with `--new-pid-ns` but without `--daemonize`.
* [#4995](https://github.com/firecracker-microvm/firecracker/pull/4995): Firecracker no longer overwrites CPUID leaf 0x80000000 when running AMD hardware, meaning the guest can now discover a greater range of CPUID leaves in the extended function range (this range is host kernel dependent).
* [#5046](https://github.com/firecracker-microvm/firecracker/pull/5046): Retry KVM_CREATE_VM on EINTR that occasionally happen on heavily loaded hosts to improve reliability of microVM creation.
* [#5052](https://github.com/firecracker-microvm/firecracker/pull/5052): Build the empty seccomp policy as default for debug builds to avoid crashes on syscalls introduced by debug assertions from Rust 1.80.0.

Changed

* [#4907](https://github.com/firecracker-microvm/firecracker/pull/4907): Bump snapshot version to 4.0.0. Ensure Firecracker v1.10.1 is incompatible with snapshots from other releases.

### Added

* [#4834](https://github.com/firecracker-microvm/firecracker/pull/4834): Add `VIRTIO_NET_F_RX_MRGBUF` support to the `virtio-net` device.  When this feature is negotiated, guest `virtio-net` driver can perform more efficient memory management which in turn improves RX and TX performance.
* [#4460](https://github.com/firecracker-microvm/firecracker/pull/4460): Add a call to [`KVM_KVMCLOCK_CTRL`](https://docs.kernel.org/virt/kvm/api.html#kvm-kvmclock-ctrl) after pausing vCPUs on x86_64 architectures. This ioctl sets a flag in the KVM state of the vCPU indicating that it has been paused by the host userspace. In guests that use kvmclock, the soft lockup watchdog checks this flag. If it is set, it won't trigger the lockup condition. Calling the ioctl for guests that don't use kvmclock will fail. These failures are not fatal. We log the failure and increase the `vcpu.kvmclock_ctrl_fails` metric.
* [#4869](https://github.com/firecracker-microvm/firecracker/pull/4869): Added support for Aarch64 systems which feature CPU caches with a number of sets higher than `u16::MAX`.
* [#4797](https://github.com/firecracker-microvm/firecracker/pull/4797), [#4854](https://github.com/firecracker-microvm/firecracker/pull/4854): Added GDB debugging support for a microVM guest kernel. Please see our [GDB debugging documentation](docs/gdb-debugging.md) for more information.

### Changed

* [#4844](https://github.com/firecracker-microvm/firecracker/pull/4844): Upgrade `virtio-net` device to use `readv` syscall to avoid unnecessary memory copies on RX path, increasing the RX performance.

### Removed

* [#4804](https://github.com/firecracker-microvm/firecracker/pull/4804): Drop Support for guest kernel 4.14. Linux 4.14 reached end-of-life in [January 2024](https://lore.kernel.org/lkml/2024011046-ecology-tiptoeing-ce50@gregkh/) The minimum supported guest kernel now is 5.10.

### Fixed

* [#4796](https://github.com/firecracker-microvm/firecracker/pull/4796): Fixed Vsock not notifying guest about `TRANSPORT_RESET_EVENT` event after snapshot restore. This resulted in guest waiting indefinitely on a connection which was reset during snapshot creation.
* [#4790](https://github.com/firecracker-microvm/firecracker/pull/4790): v1.9.0 was missing most of the debugging information in the debuginfo file, due to a change in the Cargo defaults. This has been corrected.
* [#4826](https://github.com/firecracker-microvm/firecracker/pull/4826): Add missing configuration of tap offload features when restoring from a snapshot. Setting the features was previously [moved](https://github.com/firecracker-microvm/firecracker/pull/4680/commits/49ed5ea4b48ccd98903da037368fa3108f58ac1f) from net device creation to device activation time, but it was not reflected in the restore path. This was leading to inability to connect to the restored VM if the offload features were used.

### Fixed

* [#4824](https://github.com/firecracker-microvm/firecracker/pull/4824): Add missing configuration of tap offload features when restoring from a snapshot. Setting the features was previously [moved](https://github.com/firecracker-microvm/firecracker/pull/4680/commits/49ed5ea4b48ccd98903da037368fa3108f58ac1f) from net device creation to device activation time, but it was not reflected in the restore path. This was leading to inability to connect to the restored VM if the offload features were used.
* [#4829](https://github.com/firecracker-microvm/firecracker/pull/4829): v1.9.0 was missing most of the debugging information in the debuginfo file, due to a change in the Cargo defaults. This has been corrected.

### Added

* [#4687](https://github.com/firecracker-microvm/firecracker/pull/4687): Added VMGenID support for microVMs running on ARM hosts with 6.1 guest kernels. Support for VMGenID via DeviceTree bindings exists only on mainline 6.10 Linux onwards. Users of Firecracker will need to backport the relevant patches on top of their 6.1 kernels to make use of the feature.
* [#4732](https://github.com/firecracker-microvm/firecracker/pull/4732), [#4733](https://github.com/firecracker-microvm/firecracker/pull/4733), [#4741](https://github.com/firecracker-microvm/firecracker/pull/4741), [#4746](https://github.com/firecracker-microvm/firecracker/pull/4746): Added official support for 6.1 microVM guest kernels.

### Changed

### Deprecated

* Support for guest kernel 4.14 is now deprecated. We will completely remove 4.14 support with Firecracker version v1.10

### Removed

* [#4689](https://github.com/firecracker-microvm/firecracker/pull/4689): Drop support for host kernel 4.14. Linux 4.14 reached end-of-life in [January 2024](https://lore.kernel.org/lkml/2024011046-ecology-tiptoeing-ce50@gregkh/). The minimum supported kernel now is 5.10. Guest kernel 4.14 is still supported.

### Fixed

* [4680](https://github.com/firecracker-microvm/firecracker/pull/4680): Fixed an issue ([#4659](https://github.com/firecracker-microvm/firecracker/issues/4659)) where the virtio-net device implementation would always assume the guest accepts all VirtIO features the device offers. This is always true with the Linux guest kernels we are testing but other kernels, like FreeBSD make different assumptions. This PR fixes the emulation code to set the TAP features based on the features accepted by the guest.

### Added

* [#4428](https://github.com/firecracker-microvm/firecracker/pull/4428): Added ACPI support to Firecracker for x86_64 microVMs. Currently, we pass ACPI tables with information about the available vCPUs, interrupt controllers, VirtIO and legacy x86 devices to the guest. This allows booting kernels without MPTable support. Please see our [kernel policy documentation](docs/kernel-policy.md) for more information regarding relevant kernel configurations.
* [#4487](https://github.com/firecracker-microvm/firecracker/pull/4487): Added support for the Virtual Machine Generation Identifier (VMGenID) device on x86_64 platforms. VMGenID is a virtual device that allows VMMs to notify guests when they are resumed from a snapshot. Linux includes VMGenID support since version 5.18. It uses notifications from the device to reseed its internal CSPRNG. Please refer to [snapshot support](docs/snapshotting/snapshot-support.md) and [random for clones](docs/snapshotting/random-for-clones.md) documention for more info on VMGenID. VMGenID state is part of the snapshot format of Firecracker. As a result, Firecracker snapshot version is now 2.0.0.

### Changed

* [#4492](https://github.com/firecracker-microvm/firecracker/pull/4492): Changed `--config` parameter of `cpu-template-helper` optional. Users no longer need to prepare kernel, rootfs and Firecracker configuration files to use `cpu-template-helper`.
* [#4537](https://github.com/firecracker-microvm/firecracker/pull/4537) Changed T2CL template to pass through bit 27 and 28 of `MSR_IA32_ARCH_CAPABILITIES` (`RFDS_NO` and `RFDS_CLEAR`) since KVM consider they are able to be passed through and T2CL isn't designed for secure snapshot migration between different processors.
* [#4537](https://github.com/firecracker-microvm/firecracker/pull/4537) Changed T2S template to set bit 27 of `MSR_IA32_ARCH_CAPABILITIES` (`RFDS_NO`) to 1 since it assumes that the fleet only consists of processors that are not affected by RFDS.
* [#4388](https://github.com/firecracker-microvm/firecracker/pull/4388): Avoid setting `kvm_immediate_exit` to 1 if are already handling an exit, or if the vCPU is stopped. This avoids a spurious KVM exit upon restoring snapshots.
* [#4567](https://github.com/firecracker-microvm/firecracker/pull/4567): Do not initialize vCPUs in powered-off state upon snapshot restore. No functional change, as vCPU initialization is only relevant for the booted case (where the guest expects CPUs to be powered off).

### Deprecated

* Firecracker's `--start-time-cpu-us` and `--start-time-us` parameters are deprecated and will be removed in v2.0 or later. They are used by the jailer to pass the value that should be subtracted from the (CPU) time, when emitting the `start_time_us` and `start_time_cpu_us` metrics. These parameters were never meant to be used by end customers, and we recommend doing any such time adjustments outside Firecracker.
* Booting with microVM kernels that rely on MPTable on x86_64 is deprecated and support will be removed in v2.0 or later. We suggest to users of Firecracker to use guest kernels with ACPI support. For x86_64 microVMs, ACPI will be the only way Firecracker passes hardware information to the guest once MPTable support is removed.

### Fixed

* [#4526](https://github.com/firecracker-microvm/firecracker/pull/4526): Added a check in the network TX path that the size of the network frames the guest passes to us is not bigger than the maximum frame the device expects to handle. On the TX path, we copy frames destined to MMDS from guest memory to Firecracker memory. Without the check, a mis-behaving virtio-net driver could cause an increase in the memory footprint of the Firecracker process. Now, if we receive such a frame, we ignore it and increase `Net::tx_malformed_frames` metric.
* [#4536](https://github.com/firecracker-microvm/firecracker/pull/4536): Make the first differential snapshot taken after a full snapshot contain only the set of memory pages changed since the full snapshot. Previously, these differential snapshots would contain all memory pages. This will result in potentially much smaller differential snapshots after a full snapshot.
* [#4578](https://github.com/firecracker-microvm/firecracker/pull/4578): Fix UFFD support not being forward-compatible with new ioctl options introduced in Linux 6.6. See also https://github.com/bytecodealliance/userfaultfd-rs/issues/61.
* [#4630](https://github.com/firecracker-microvm/firecracker/pull/4630): On x86_64, when taking a snapshot, if a vCPU has MSR_IA32_TSC_DEADLINE set to 0, Firecracker will replace it with the MSR_IA32_TSC value from the same vCPU. This is to guarantee that the vCPU will continue receiving TSC interrupts after restoring from the snapshot even if an interrupt is lost when taking a snapshot.
* [#4666](https://github.com/firecracker-microvm/firecracker/pull/4666): Fixed Firecracker sometimes restoring `MSR_IA32_TSC_DEADLINE` before `MSR_IA32_TSC`. Now it always restores `MSR_IA32_TSC_DEADLINE` MSR after `MSR_IA32_TSC`, as KVM relies on the guest TSC for correct restoration of `MSR_IA32_TSC_DEADLINE`. This fixed guests using the `TSC_DEADLINE` hardware feature receiving incorrect timer interrupts after snapshot restoration, which could lead to them seemingly getting stuck in sleep-related syscalls (see also https://github.com/firecracker-microvm/firecracker/pull/4099).

### Added

* [#4346](https://github.com/firecracker-microvm/firecracker/pull/4346): Added support to emit aggregate (minimum/maximum/sum) latency for `VcpuExit::MmioRead`, `VcpuExit::MmioWrite`, `VcpuExit::IoIn` and `VcpuExit::IoOut`. The average for these VM exits is not emitted since it can be deduced from the available emitted metrics.
* [#4360](https://github.com/firecracker-microvm/firecracker/pull/4360): Added dev-preview support for backing a VM's guest memory by 2M hugetlbfs pages. Please see the [documentation](docs/hugepages.md) for more information
* [#4490](https://github.com/firecracker-microvm/firecracker/pull/4490): Added block and net device metrics for file/tap access latencies and queue backlog lengths, which can be used to analyse saturation of the Firecracker VMM thread and underlying layers. Queue backlog length metrics are flushed periodically. They can be used to esimtate an average queue length by request by dividing its value by the number of requests served.

### Changed

* [#4230](https://github.com/firecracker-microvm/firecracker/pull/4230): Changed microVM snapshot format version strategy. Firecracker snapshot format now has a version that is independent of Firecracker version. The current version of the snapshot format is v1.0.0. From now on, the Firecracker binary will define the snapshot format version it supports and it will only be able to load snapshots with format that is backwards compatible with that version. Users can pass the `--snapshot-version` flag to the Firecracker binary to see its supported snapshot version format. This change renders all previous Firecracker snapshots (up to Firecracker version v1.6.0) incompatible with the current Firecracker version.
* [#4449](https://github.com/firecracker-microvm/firecracker/pull/4449): Added information about page size to the payload Firecracker sends to the UFFD handler. Each memory region object now contains a `page_size_kib` field. See also the [hugepages documentation](docs/hugepages.md).
* [#4501](https://github.com/firecracker-microvm/firecracker/pull/4501): Only use memfd to back guest memory if a vhost-user-blk device is configured, otherwise use anonymous private memory. This is because serving page faults of shared memory used by memfd is slower and may impact workloads.

### Fixed

* [#4409](https://github.com/firecracker-microvm/firecracker/pull/4409): Fixed a bug in the cpu-template-helper that made it panic during conversion of cpu configuration with SVE registers to the cpu template on aarch64 platform. Now cpu-template-helper will print warnings if it encounters SVE registers during the conversion process. This is because cpu templates are limited to only modify registers less than 128 bits.
* [#4413](https://github.com/firecracker-microvm/firecracker/pull/4413): Fixed a bug in the Firecracker that prevented it to restore snapshots of VMs that had SVE enabled.
* [#4414](https://github.com/firecracker-microvm/firecracker/pull/4360): Made `PATCH` requests to the `/machine-config` endpoint transactional, meaning Firecracker's configuration will be unchanged if the request returns an error. This fixes a bug where a microVM with incompatible balloon and guest memory size could be booted, due to the check for this condition happening after Firecracker's configuration was updated.
* [#4259](https://github.com/firecracker-microvm/firecracker/pull/4259): Added a double fork mechanism in the Jailer to avoid setsid() failures occurred while running Jailer as the process group leader. However, this changed the behaviour of Jailer and now the Firecracker process will always have a different PID than the Jailer process.
*  [#4436](https://github.com/firecracker-microvm/firecracker/pull/4436): Added a "Known Limitations" section in the Jailer docs to highlight the above change in behaviour introduced in PR#4259.
*  [#4442](https://github.com/firecracker-microvm/firecracker/pull/4442): As a solution to the change in behaviour introduced in PR#4259, provided a mechanism to reliably fetch Firecracker PID. With this change, Firecracker process's PID will always be available in the Jailer's root directory regardless of whether new_pid_ns was set.
* [#4468](https://github.com/firecracker-microvm/firecracker/pull/4468): Fixed a bug where a client would hang or timeout when querying for an MMDS path whose content is empty, because the 'Content-Length' header field was missing in a response.

### Added

  - [#4145](https://github.com/firecracker-microvm/firecracker/pull/4145): Added support for per net device metrics. In addition to aggregate metrics `net`, each individual net device will emit metrics under the label `"net_{iface_id}"`. E.g. the associated metrics for the endpoint `"/network-interfaces/eth0"` will be available under `"net_eth0"` in the metrics json object.
  - [#4202](https://github.com/firecracker-microvm/firecracker/pull/4202): Added support for per block device metrics. In addition to aggregate metrics `block`, each individual block device will emit metrics under the label `"block_{drive_id}"`. E.g. the associated metrics for the endpoint `"/drives/{drive_id}"` will be available under `"block_drive_id"` in the metrics json object.
  - [#4205](https://github.com/firecracker-microvm/firecracker/pull/4205): Added a new `vm-state` subcommand to `info-vmstate` command in the `snapshot-editor` tool to print MicrovmState of vmstate snapshot file in a readable format. Also made the `vcpu-states` subcommand available on x86\_64.
  - [#4063](https://github.com/firecracker-microvm/firecracker/pull/4063): Added source-level instrumentation based tracing. See [tracing](./docs/tracing.md) for more details.
  - [#4138](https://github.com/firecracker-microvm/firecracker/pull/4138), [#4170](https://github.com/firecracker-microvm/firecracker/pull/4170), [#4223](https://github.com/firecracker-microvm/firecracker/pull/4223), [#4247](https://github.com/firecracker-microvm/firecracker/pull/4247), [#4226](https://github.com/firecracker-microvm/firecracker/pull/4226): Added **developer preview only** (NOT for production use) support for vhost-user block devices. Firecracker implements a vhost-user frontend. Users are free to choose from existing open source backend solutions or their own implementation. Known limitation: snapshotting is not currently supported for microVMs containing vhost-user block devices. See the [related doc page](./docs/api_requests/block-vhost-user.md) for details. The device emits metrics under the label `"vhost_user_{device}_{drive_id}"`.

### Changed

  - [#4309](https://github.com/firecracker-microvm/firecracker/pull/4309): The jailer’s option `--parent-cgroup` will move the process to that cgroup if no `cgroup` options are provided.
  - Simplified and clarified the removal policy of deprecated API elements to follow semantic versioning 2.0.0. For more information, please refer to [this GitHub discussion](https://github.com/firecracker-microvm/firecracker/discussions/4135).
  - [#4180](https://github.com/firecracker-microvm/firecracker/pull/4180): Refactored error propagation to avoid logging and printing an error on exits with a zero exit code. Now, on successful exit “Firecracker exited successfully” is logged.
  - [#4194](https://github.com/firecracker-microvm/firecracker/pull/4194): Removed support for creating Firecracker snapshots targeting older versions of Firecracker. With this change, running ‘firecracker –version’ will not print the supported snapshot versions.
  - [#4301](https://github.com/firecracker-microvm/firecracker/pull/4301): Allow merging of diff snapshots into base snapshots by directly writing the diff snapshot on top of the base snapshot’s memory file. This can be done by setting the `mem_file_path` to the path of the pre-existing full snapshot.

### Deprecated

  - [#4209](https://github.com/firecracker-microvm/firecracker/pull/4209): `rebase-snap` tool is now deprecated. Users should use `snapshot-editor` for rebasing diff snapshots.

### Fixed

  - [#4171](https://github.com/firecracker-microvm/firecracker/pull/4171): Fixed a bug that ignored the `--show-log-origin` option, preventing it from printing the source code file of the log messages.
  - [#4178](https://github.com/firecracker-microvm/firecracker/pull/4178): Fixed a bug reporting a non-zero exit code on successful shutdown when starting Firecracker with `--no-api`.
  - [#4261](https://github.com/firecracker-microvm/firecracker/pull/4261): Fixed a bug where Firecracker would log “RunWithApiError error: MicroVMStopped without an error: GenericError” when exiting after encountering an emulation error. It now correctly prints “RunWithApiError error: MicroVMStopped *with* an error: GenericError”.
  - [#4242](https://github.com/firecracker-microvm/firecracker/pull/4242): Fixed a bug introduced in #4047 that limited the `--level` option of logger to Pascal-cased values (e.g. accepting “Info”, but not “info”). It now ignores case again.
  - [#4286](https://github.com/firecracker-microvm/firecracker/pull/4286): Fixed a bug in the asynchronous virtio-block engine that rendered the device non-functional after a PATCH request was issued to Firecracker for updating the path to the host-side backing file of the device.
  - [#4301](https://github.com/firecracker-microvm/firecracker/pull/4301): Fixed a bug where if Firecracker was instructed to take a snapshot of a microvm which itself was restored from a snapshot, specifying `mem_file_path` to be the path of the memory file from which the microvm was restored would result in both the microvm and the snapshot being corrupted. It now instead performs a “write-back” of all memory that was updated since the snapshot was originally loaded.

### Added

* [#4287](https://github.com/firecracker-microvm/firecracker/issues/4287): Document a caveat to the jailer docs when using the `--parent-cgroup` option, which results in it being ignored by the jailer. Refer to the [jailer documentation](./docs/jailer.md#caveats) for a workaround.

### Changed

* [#4191](https://github.com/firecracker-microvm/firecracker/pull/4191): Refactored error propagation to avoid logging and printing an error on exits with a zero exit code. Now, on successful exit "Firecracker exited successfully" is logged.

### Fixed

* [#4277](https://github.com/firecracker-microvm/firecracker/pull/4277): Fixed a bug that ignored the `--show-log-origin` option, preventing it from printing the source code file of the log messages.
* [#4179](https://github.com/firecracker-microvm/firecracker/pull/4179): Fixed a bug reporting a non-zero exit code on successful shutdown when starting Firecracker with `--no-api`.
* [#4271](https://github.com/firecracker-microvm/firecracker/pull/4271): Fixed a bug where Firecracker would log "RunWithApiError error: MicroVMStopped without an error: GenericError" when exiting after encountering an emulation error. It now correctly prints "RunWithApiError error: MicroVMStopped *with* an error: GenericError".
* [#4270](https://github.com/firecracker-microvm/firecracker/pull/4270): Fixed a bug introduced in #4047 that limited the `--level` option of logger to Pascal-cased values (e.g. accepting "Info", but not "info"). It now ignores case again.
* [#4295](https://github.com/firecracker-microvm/firecracker/pull/4295): Fixed a bug in the asynchronous virtio-block engine that rendered the device non-functional after a PATCH request was issued to Firecracker for updating the path to the host-side backing file of the device.

### Added

- [#3837](https://github.com/firecracker-microvm/firecracker/issues/3837): Added official support for Linux 6.1. See [prod-host-setup](./docs/prod-host-setup.md) for some security and performance considerations.
- [#4045](https://github.com/firecracker-microvm/firecracker/pull/4045) and [#4075](https://github.com/firecracker-microvm/firecracker/pull/4075): Added `snapshot-editor` tool for modifications of snapshot files. It allows for rebasing of memory snapshot files, printing and removing aarch64 registers from the vmstate and obtaining snapshot version.
- [#3967](https://github.com/firecracker-microvm/firecracker/pull/3967/): Added new fields to the custom CPU templates. (aarch64 only) `vcpu_features` field allows modifications of vCPU features enabled during vCPU initialization. `kvm_capabilities` field allows modifications of KVM capability checks that Firecracker performs during boot. If any of these fields are in use, minimal target snapshot version is restricted to 1.5.

### Changed

- Updated deserialization of `bitmap` for custom CPU templates to allow usage of '_' as a separator.
- Changed the strip feature of `cpu-template-helper` tool to operate bitwise.
- Better logs during validation of CPU ID in snapshot restoration path. Also Firecracker now does not fail if it can't get CPU ID from the host or can't find CPU ID in the snapshot.
- Changed the serial device to only try to initialize itself if stdin is a terminal or a FIFO pipe. This fixes logged warnings about the serial device failing to initialize if the process is daemonized (in which case stdin is /dev/null instead of a terminal).
- Changed to show a warning message when launching a microVM with C3 template on a processor prior to Intel Cascade Lake, because the guest kernel does not apply the mitigation against MMIO stale data vulnerability when it is running on a processor that does not enumerate FBSDP_NO, PSDP_NO and SBDR_SSDP_NO on IA32_ARCH_CAPABILITIES MSR.
- Made Firecracker resize its file descriptor table on process start. It now preallocates the in-kernel fdtable to hold `RLIMIT_NOFILE` many fds (or 2048 if no limit is set). This avoids the kernel reallocating the fdtable during Firecracker operations, resulting in a 30ms to 70ms reduction of snapshot restore times for medium to large microVMs with many devices attached.
- Changed the dump feature of `cpu-template-helper` tool not to enumerate program counter (PC) on ARM because it is determined by the given kernel image and it is useless in the custom CPU template context.
- The ability to create snapshots for an older version of Firecracker is now deprecated. As a result, the `version` body field in `PUT` on `/snapshot/create` request in deprecated.
- Added support for the /dev/userfaultfd device available on linux kernels >= 6.1. This is the default for creating UFFD handlers on these kernel versions. If it is unavailable, Firecracker falls back to the userfaultfd syscall.
- Deprecated `cpu_template` field in `PUT` and `PATCH` requests on `/machine-config` API, which is used to set a static CPU template. Custom CPU templates added in v1.4.0 are available as an improved iteration of the static CPU templates. For more information about the transition from static CPU templates to custom CPU templates, please refer to [this GitHub discussion](https://github.com/firecracker-microvm/firecracker/discussions/4135).
- Changed default log level from [`Warn`](https://docs.rs/log/latest/log/enum.Level.html#variant.Warn) to [`Info`](https://docs.rs/log/latest/log/enum.Level.html#variant.Info). This results in more logs being output by default.
### Fixed

- Fixed a change in behavior of normalize host brand string that breaks Firecracker on external instances.
- Fixed the T2A CPU template not to unset the MMX bit (CPUID.80000001h:EDX[23]) and the FXSR bit (CPUID.80000001h:EDX[24]).
- Fixed the T2A CPU template to set the RstrFpErrPtrs bit (CPUID.80000008h:EBX[2]).
- Fixed a bug where Firecracker would crash during boot if a guest set up a virtio queue that partially overlapped with the MMIO gap. Now Firecracker instead correctly refuses to activate the corresponding virtio device.
- Fixed the T2CL CPU template to pass through security mitigation bits that are listed by KVM as bits able to be passed through. By making the most use of the available hardware security mitigations on a processor that a guest is running on, the guest might be able to benefit from performance improvements.
- Fixed the T2S CPU template to set the GDS_NO bit of the IA32_ARCH_CAPABILITIES MSR to 1 in accordance with an Intel microcode update. To use the template securely, users should apply the latest microcode update on the host.
- Fixed the spelling of the `nomodule` param passed in the default kernel command line parameters. This is a **breaking change** for setups that use the default kernel command line which also depend on being able to load kernel modules at runtime. This may also break setups which use the default kernel command line and which use an init binary that inadvertently depends on the misspelled param ("nomodules") being present at the command line, since this param will no longer be passed.

Fixed

* Fixed a change in behavior of normalize host brand string that breaks
  Firecracker on external instances.
* Fixed the T2A CPU template not to unset the MMX bit (CPUID.80000001h:EDX[23])
  and the FXSR bit (CPUID.80000001h:EDX[24]).
* Fixed the T2A CPU template to set the RstrFpErrPtrs bit
  (CPUID.80000008h:EBX[2]).

Added

* Added support for custom CPU templates allowing users to adjust vCPU features
  exposed to the guest via CPUID, MSRs and ARM registers.
* Introduced V1N1 static CPU template for ARM to represent Neoverse V1 CPU
  as Neoverse N1.
* Added support for the `virtio-rng` entropy device. The device is optional. A
  single device can be enabled per VM using the `/entropy` endpoint.
* Added a `cpu-template-helper` tool for assisting with creating and managing
  custom CPU templates.

Changed

* Set FDP_EXCPTN_ONLY bit (CPUID.7h.0:EBX[6]) and ZERO_FCS_FDS bit
  (CPUID.7h.0:EBX[13]) in Intel's CPUID normalization process.

Fixed

* Fixed feature flags in T2S CPU template on Intel Ice Lake.
* Fixed CPUID leaf 0xb to be exposed to guests running on AMD host.
* Fixed a performance regression in the jailer logic for closing open file
  descriptors. Related to:
  [#3542](https://github.com/firecracker-microvm/firecracker/issues/3542).
* A race condition that has been identified between the API thread and the VMM
  thread due to a misconfiguration of the `api_event_fd`.
* Fixed CPUID leaf 0x1 to disable perfmon and debug feature on x86 host.
* Fixed passing through cache information from host in CPUID leaf 0x80000006.
* Fixed the T2S CPU template to set the RRSBA bit of the IA32_ARCH_CAPABILITIES
  MSR to 1 in accordance with an Intel microcode update.
* Fixed the T2CL CPU template to pass through the RSBA and RRSBA bits of the
  IA32_ARCH_CAPABILITIES MSR from the host in accordance with an Intel microcode
  update.
* Fixed passing through cache information from host in CPUID leaf 0x80000005.
* Fixed the T2A CPU template to disable SVM (nested virtualization).
* Fixed the T2A CPU template to set EferLmsleUnsupported bit
  (CPUID.80000008h:EBX[20]), which indicates that EFER[LMSLE] is not supported.

Fixed

* Fixed passing through cache information from host in CPUID leaf 0x80000006.

Changed

* Upgraded Rust toolchain from 1.64.0 to 1.66.1.

Fixed

* A race condition that has been identified between the API thread and the VMM
  thread due to a misconfiguration of the `api_event_fd`.

Fixed

* A race condition that has been identified between the API thread and the VMM
  thread due to a misconfiguration of the `api_event_fd`.

Fixed

* Fixed taking a snapshot with `target_version = 1.3.0` failing with the
  message "Cannot translate microVM version to snapshot data version".

**Update 03-03**: This release contains a bug when trying to snapshot with `target_version=1.3.0`, which results in an error and no snapshot taken. To mitigate this bug, leave `target_version` empty or use `target_version=1.2.0` We are currently working on a fix.

---

Added

* Introduced T2CL (Intel) and T2A (AMD) CPU templates to provide
  instruction set feature parity between Intel and AMD CPUs when using
  these templates.
* Added Graviton3 support (c7g instance type).

Changed

* Improved error message when invalid network backend provided.
* Improved TCP throughput by between 5% and 15% (depending on CPU) by using
  scatter-gather I/O in the net device's TX path.
* Upgraded Rust toolchain from 1.64.0 to 1.66.0.
* Made seccompiler output bit-reproducible.

Fixed

* Fixed feature flags in T2 CPU template on Intel Ice Lake.

### Fixed

* Fixed a bug on ARM64 hosts where the upper 64bits of the V0-V31 FL/SIMD
  registers were not saved correctly when taking a snapshot, potentially leading
  to data loss. This change invalidates all ARM64 snapshots taken with versions
  of Firecracker <= 1.1.3.

### Added

* Added a new CPU template called `T2S`. This exposes the same CPUID as `T2` to
  the Guest and also overwrites the `ARCH_CAPABILITIES` MSR to expose a reduced
  set of capabilities. With regards to hardware vulnerabilities and mitigations,
  the Guest vCPU will apear to look like a Skylake CPU, making it safe to
  snapshot uVMs running on a newer host CPU (Cascade Lake) and restore on a host
  that has a Skylake CPU.
* Added a new CLI option `--metrics-path PATH`. It accepts a file parameter
  where metrics will be sent to.
* Added baselines for m6i.metal and m6a.metal for all long running performance
  tests.
* Releases now include debuginfo files.

### Changed

* Changed the jailer option `--exec-file` to fail if the filename does not
  contain the string `firecracker` to prevent from running non-firecracker
  binaries.
* Upgraded Rust toolchain from 1.52.1 to 1.64.0.
* Switched to specifying our dependencies using caret requirements instead
  of comparison requirements.
* Updated all dependencies to their respective newest versions.

### Fixed

* Made the `T2` template more robust by explicitly disabling additional
  CPUID flags that should be off but were missed initially or that were
  not available in the spec when the template was created.
* Now MAC address is correctly displayed when queried with GET `/vm/config`
  if left unspecified in both pre and post snapshot states.
* Fixed a self-DoS scenario in the virtio-queue code by reporting and
  terminating execution when the number of available descriptors reported
  by the driver is higher than the queue size.
* Fixed the bad handling of kernel cmdline parameters when init arguments were
  provided in the `boot_args` field of the JSON body of the PUT `/boot-source`
  request.
* Fixed a bug on ARM64 hosts where the upper 64bits of the V0-V31 FL/SIMD
  registers were not saved correctly when taking a snapshot, potentially
  leading to data loss. This change invalidates all ARM64 snapshots taken
  with versions of Firecracker <= 1.1.3.
* Improved stability and security when saving CPU MSRs in snapshots.

### Fixed

* Fixed the bad handling of kernel cmdline parameters when init arguments were
  provided in the `boot_args` field of the JSON body of the PUT `/boot-source`
  request.

### Changed

* Upgraded Rust version to 1.64. This enables us to keep supporting v1.1. [not reflected in CHANGELOG.md]

### Fixed

* Fixed the bad handling of kernel cmdline parameters when init arguments were
  provided in the `boot_args` field of the JSON body of the PUT `/boot-source`
  request.

### Fixed

* Fixed a self-DoS scenario in the virtio-queue code by reporting and
  terminating execution when the number of available descriptors reported
  by the driver is higher than the queue size.

### Fixed

* Fixed a self-DoS scenario in the virtio-queue code by reporting and
  terminating execution when the number of available descriptors reported
  by the driver is higher than the queue size.

### Added

* Add a new CPU template called `T2S`. This exposes the same CPUID as `T2`
  to the Guest and also overwrites the `ARCH_CAPABILITIES` MSR to expose a
  reduced set of capabilities. With regards to hardware vulnerabilities
  and mitigations, the Guest vCPU will apear to look like a Skylake CPU,
  making it safe to snapshot uVMs running on a newer host CPU (Cascade Lake)
  and restore on a host that has a Skylake CPU.

### Fixed

* Make the `T2` template more robust by explicitly disabling additional
  CPUID flags that should be off but were missed initially or that were
  not available in the spec when the template was created.

### Added

* The API `PATCH` methods for `machine-config` can now be used to reset
  the `cpu_template` to `"None"`. Until this change there was no way to
  reset the `cpu_template` once it was set.
* Added a `rebase-snap` tool for rebasing a diff snapshot over a base
  snapshot.
* Mmds version is persisted across snapshot-restore. Snapshot compatibility is
  preserved bidirectionally, to and from a Firecracker version that does not
  support persisting the Mmds version. In such cases, the default V1 option is
  used.
* Added `--mmds-size-limit` for limiting the mmds data store size instead of
  piggy-backing on `--http-api-max-payload-size`. If left unconfigured it
  defaults to the value of `--http-api-max-payload-size`, to provide backwards
  compatibility.
* Added optional `mem_backend` body field in `PUT` requests on `/snapshot/load`.
  This new parameter is an object that defines the configuration of the backend
  responsible for handling memory loading during snapshot restore. The
  `mem_backend` parameter contains `backend_type` and `backend_path` required
  fields. `backend_type` is an enum that can take either `File` or `Uffd` as
  value. Interpretation of `backend_path` field depends on the value of
  `backend_type`. If `File`, then the user must provide the path to file that
  contains the guest memory to be loaded. Otherwise, if `backend_type` is `Uffd`,
  then `backend_path` is the path to a unix domain socket where a custom page
  fault handler process is listening and expecting a UFFD to be sent by
  Firecracker. The UFFD is used to handle the guest memory page faults in the
  separate process.
* Added logging for the snapshot/restore and async block device IO engine
  features to indicate they are in development preview.

### Changed

* The API `PATCH` method for `/machine-config` can be now used to change
  `track_dirty_pages` on aarch64.
* MmdsV2 is now Generally Available.
* MmdsV1 is now deprecated and will be removed in Firecracker v2.0.0.
  Use MmdsV2 instead.
* Deprecated `mem_file_path` body field in `PUT` on `/snapshot/load` request.

### Fixed

* Fixed inconsistency that allowed the start of a microVM from a JSON file
  without specifying the `vcpu_count` and `mem_size_mib` parameters for
  `machine-config` although they are mandatory when configuring via the API.
  Now these fields are mandatory when specifying `machine-config` in the JSON
  file and when using the `PUT` request on `/machine-config`.
* Fixed inconsistency that allowed a user to specify the `cpu_template`
  parameter and set `smt` to `True` in `machine-config` when starting from a
  JSON file on aarch64 even though they are not permitted when using `PUT` or
  `PATCH` in the API. Now Firecracker will return an error on aarch64 if `smt`
  is set to `True` or if `cpu_template` is specified.
* Fixed inconsistent behaviour of the `PUT` method for `/machine-config` that
  would reset the `track_dirty_pages` parameter to `false` if it was not
  specified in the JSON body of the request, but left the `cpu_template`
  parameter intact if it was not present in the request. Now a `PUT` request
  for `/machine-config` will reset all optional parameters (`smt`,
  `cpu_template`, `track_dirty_pages`) to their default values if they are
  not specified in the `PUT` request.
* Fixed incosistency in the swagger definition with the current state of the
  `/vm/config` endpoint.

### Added

* Added jailer option `--parent-cgroup ` to allow the placement
  of microvm cgroups in custom cgroup nested hierarchies. The default value is
  `` which is backwards compatible to the behavior before this
  change.
* Added jailer option `--cgroup-version <1|2>` to support running the jailer
  on systems that have cgroup-v2. Default value is `1` which means that if
  `--cgroup-version` is not specified, the jailer will try to create cgroups
  on cgroup-v1 hierarchies only.
* Added `--http-api-max-payload-size` parameter to configure the maximum payload
  size for PUT and PATCH requests.
* Limit MMDS data store size to `--http-api-max-payload-size`.
* Cleanup all environment variables in Jailer.
* Added metrics for accesses to deprecated HTTP and command line API endpoints.
* Added permanent HTTP endpoint for `GET` on `/version` for getting the
  Firecracker version.
* Added `--metadata` parameter to enable MMDS content to be supplied from a file
  allowing the MMDS to be used when using `--no-api` to disable the API server.
* Checksum file for the release assets.
* Added support for custom headers to MMDS requests. Accepted headers are:
  `X-metadata-token`, which accepts a string value that provides a session
  token for MMDS requests; and `X-metadata-token-ttl-seconds`, which
  specifies the lifetime of the session token in seconds.
* Support and validation for host and guest kernel 5.10.
* A [kernel support policy](docs/kernel-policy.md).
* Added `io_engine` to the pre-boot block device configuration.
  Possible values: `Sync` (the default option) or `Async` (only available for
  kernels newer than 5.10.51). The `Async` variant introduces a block device
  engine that uses io_uring for executing requests asynchronously, which is in
  **developer preview** (NOT for production use).
  See `docs/api_requests/block-io-engine.md`.
* Added `block.io_engine_throttled_events` metric for measuring the number of
  virtio events throttled because of the IO engine.
* New optional `version` field to PUT requests towards `/mmds/config` to
  configure MMDS version. Accepted values are `V1` and `V2` and default is
  `V1`. MMDS `V2` is **developer preview only** (NOT for production use) and
  it does not currently work after snapshot load.
* Mandatory `network_interfaces` field to PUT requests towards
  `/mmds/config` which contains a list of network interface IDs capable of
  forwarding packets to MMDS.

### Changed

* Removed the `--node` jailer parameter.
* Deprecated `vsock_id` body field in `PUT`s on `/vsock`.
* Removed the deprecated the `--seccomp-level parameter`.
* `GET` requests to MMDS require a session token to be provided through
  `X-metadata-token` header when using V2.
* Allow `PUT` requests to MMDS in order to generate a session token
  to be used for future `GET` requests when version 2 is used.
* Remove `allow_mmds_requests` field from the request body that attaches network
  interfaces. Specifying interfaces that allow forwarding requests to MMDS is done
  by adding the network interface's ID to the `network_interfaces` field of PUT
  `/mmds/config` request's body.
* Renamed `/machine-config` `ht_enabled` to `smt`.
* `smt` field is now optional on PUT `/machine-config`, defaulting to
  `false`.
* Configuring `smt: true` on aarch64 via the API is forbidden.

### Fixed

* Fixed incorrect propagation of init parameters in kernel commandline.
  Related to:
  [#2709](https://github.com/firecracker-microvm/firecracker/issues/2709).
* Adapt T2 and C3 CPU templates for kernel 5.10. Firecracker was not previously
  masking some CPU features of the host or emulated by KVM, introduced in more
  recent kernels: `umip`, `vmx`, `avx512_vnni`.
* Fix jailer's cgroup implementation to accept properties that contain multiple
  dots.

### Fixed

* Adapt T2 and C3 CPU templates for kernel 5.10. Firecracker was not previously
  masking some CPU features of the host or emulated by KVM, introduced in more
  recent kernels: `umip`, `vmx`, `avx512_vnni`.
* Fixed incorrect propagation of init parameters in kernel commandline.
  Related to:
  [#2709](https://github.com/firecracker-microvm/firecracker/issues/2709).

### Added

* Added `--http-api-max-payload-size` parameter to configure the maximum payload
  size for PUT and PATCH requests.
* Limit MMDS data store size to `--http-api-max-payload-size`.
* Cleanup all environment variables in Jailer.

### Fixed

- Updated version of yanked third party dependency.

### Added

- Added devtool build `--ssh-keys` flag to support fetching from private
  git repositories.
- Added option to configure block device flush.
- Added `--new-pid-ns` flag to the Jailer in order to spawn the Firecracker
  process in a new PID namespace.
- Added API metrics for `GET`, `PUT` and `PATCH` requests on `/mmds` endpoint.
- Added `--describe-snapshot` flag to Firecracker to fetch the data format
  version of a snapshot state file provided as argument.
- Added `--no-seccomp` parameter for disabling the default seccomp filters.
- Added `--seccomp-filter` parameter for supplying user-provided, custom filters.
- Added the `seccompiler-bin` binary that is used to compile JSON seccomp filters
  into serialized BPF for Firecracker consumption.
- Snapshotting support for GICv2 enabled guests.
- Added `devtool install` to deploy built binaries in `/usr/local/bin` or a
  given path.
- Added code logic to send `VIRTIO_VSOCK_EVENT_TRANSPORT_RESET` on snapshot
  creation, when the Vsock device is active. The event will close active
  connections on the guest.
- Added `GET` request on `/vm/config` that provides full microVM configuration
  as a JSON HTTP response.
- Added `--resource-limit` flag to jailer to limit resources such as: number of
  file descriptors allowed at a time (with a default value of 2048) and maximum
  size of files created by the process.

### Changed

- Changed Docker images repository from DockerHub to Amazon ECR.
- Fixed off-by-one error in virtio-block descriptor address validation.
- Changed the `PATCH` request on `/balloon/statistics` to schedule the first
  statistics update immediately after processing the request.
- Deprecated the `--seccomp-level parameter`. It will be removed  in a future
  release. Using it logs a runtime warning.
- Experimental gnu libc builds use empty default seccomp filters, allowing all
  system calls.

### Fixed

- Fixed non-compliant check for the RTC device ensuring a fixed
  4-sized data buffer.
- Unnecessary interrupt assertion was removed from the RTC.
  However, a dummy interrupt is still allocated for snapshot
  compatibility reasons.
- Fixed the SIGPIPE signal handler so Firecracker no longer exits. The signal
  is still recorded in metrics and logs.
- Fixed ballooning API definitions by renaming all fields which mentioned "MB"
  to use "MiB" instead.
- Snapshot related host files (vm-state, memory, block backing files) are now
  flushed to their backing mediums as part of the CreateSnapshot operation.
- Fixed the SSBD mitigation not being enabled on `aarch64` with the provided
  `prod-host-setup.md`.
- Fixed the balloon statistics not working after a snapshot restore event.
- The `utc_timestamp_ms` now reports the timestamp in ms from the UTC UNIX
  Epoch, as the name suggests. It was previously using a monotonic clock with
  an undefined starting point.