Image Customizer Live OS Support

What is a Live OS?

A Live OS refers to a system that is booted from removable media (for example, an ISO or a USB) or from the network (for example, using PXE). In such flows, the OS file system is not installed to persistent storage on the host prior to fully booting the system.

While the Live OS itself is not a concrete image format, there is a typical composition and a flow that are used when producing a Live OS ISO image or Live OS PXE artifacts (folder or tar.gz).

The Live OS is composed of the following core artifacts:

  • the boot loader (the shim and grub (or something similar))
  • the boot loader configuration
  • the kernel image
  • the initrd image
  • additional files (optional)

The flow is as follows:

  • The firmware starts the bootloader.
  • The bootloader reads its configuration and determines where the kernel/initrd files are.
  • The bootloader loads the kernel and initrd in memory.
  • The bootloader transfers control to the kernel and the initrd is attached.
  • The kernel initializes itself and then runs the initrd initialization entry point.
  • The initrd initialization entry point starts systemd (or other processes depending on the configuration).
  • Optionally, an initrd processes can locate another root file system and pivots to it if desired.

Support in Image Customizer

The Image Customizer tool can take an input image (qcow, vhd(x), previously generated Live OS ISO), and produces new ISO images and PXE artifacts. For details, see:

Limitations

The current implementation for the Live OS does not support the following:

  • Persistent storage: The machine loses all its state on reboot or shutdown.

  • dm-verity: The image cannot run dm-verity for the Live OS partitions.

  • Disk layout: There is always one disk generated when an Live OS output format is specified.

  • Non-Azure Linux official images: Image generated by tools other than the Azure Linux toolkit may fail to be converted to a Live OS. This is due to certain assumptions on the input image layout that may not be satisfied. See the section titled Input Image Requirements for more details.

Input Image Requirements

The input image can come in many formats:

  • an image with multiple partitions.
  • an ISO image.

The Image Customizer tool takes all supported input formats and normalize them into a full OS layout (mount partitions, extract squashfs contents, etc.). Then, it will proceed to customize it and then look for following:

  • /boot/grub2/grub.cfg must exist and is the ‘main’ grub configuration (not a redirection grub configuration file for example).
  • The bootloader and the shim must exist under the /boot folder or one of its subdirectories.
    • For x64, bootx64.efi and grubx64.efi (or grubx64-noprefix.efi).
    • For ARM64, bootaa64.efi and grubaa64.efi (or grubaa64-noprefix.efi).
  • All grub configurations and related files must be stored under the /boot folder. For example, grub.cfg cannot reference files outside that folder. If it does, those referenced files will not be copied to the iso and may cause grub to fail booting the desired operating system.

OS Modifications

The user can customize the full OS using the Image Customizer the as usual (using the os section in the configuration). All those customizations will carry over into the Live OS naturally. This includes customizations for kernel modules, dracut, and other early boot-time actions.

While converting the input full disk image into a Live OS involves copying almost all the artifacts unchanged - some artifacts are changed as follows:

  • grub.cfg is modified by updating the kernel command-line arguments:
    • If the root file system is bootstrapped (initramfsType=boostrap), the root value is updated to the location of the bootstrap root file system image (for example: root=live:/liveos/image.img).
    • If the root file system is the same as the initramfs (initramfsType=full-os), the root parameter is removed from the kernel parameters.
    • The user-specified iso kernel parameters are appended.

  • /etc/fstab is removed from the root file system as it typically conflicts with the overlay setup required by the LiveOS.

  • initrd.img is regenerated to serve the Live OS boot flow.
    • If the initramfs type is set to bootstrap, Dracut will regenerate the initrd.img using the input root file system and the Dracut configurations already there. This allows the user to customize the initramfs by dropping Dracut configurations on the root file system during the OS customization phase, and then when the Image Customizer tool runs Dracut to regenerate the initrd image, the newly placed/updated configuration will take effect (this is the typical way of customizing the initramfs when using Dracut to generate it).
    • If the initramfs type is full-os, the Image Customizer tool copies all the mounted partitions (minus the /boot folder) to the initramfs image. This allows the user to customize the initramfs by using the os sections to install/uninstall packages, copy files, etc as usual.
  • SELinux must be disabled if the full OS is initrd. This is because the kernel expects the initrd image to be a CPIO archive, and that format does not support extended attributes - which are necessary to hold the SELinux labels.

Initramfs Contents

The user can decide whether the full OS file system will be embedded in the initrd image itself or stored in a separate image (in which case, a process on the initrd image will need to find it, and pivot to it).

The main differences between the two configurations are:

  • When the initrd image is the full OS:
    • The initrd image size is larger and may not meet the memory restrictions on certain hardware SKUs (leading to an “out of memory” grub error). However, this is much simpler to deploy to a PXE server since the full OS content will be served using the PXE protocol without any extra setup.
    • initrd is customized using the Image Customizer tool configuration constructs directly (install packages, copy files, etc).
    • SELinux is not supported (mainly because CPIO does not support extended file system attributes).
  • When the initrd image is a bootstrap image:
    • The initrd image size will be much smaller (~30MB) - which avoids the memory restriction issue on affected hardware models.
    • The downside is that extra PXE environment setup is necessary (like setting up a server to host and serve the bootstrapped extra image file).
    • initrd is customized using Dracut configuration files. The user will need to craft those files, use the Image Customizer tool to place them in the right locations on the root file system so that when Dracut is run to generate a new initrd, it will take them into consideration.
    • SELinux can be enabled.

The user can specify either configuration using the initramfsType property defined under both the iso and the pxe sections.

Bootstrap Implementation

  • ISO support:
    • The bootstrap support is implemented using Dracut’s dmsquash module. The module expects the full OS image to be a squashfs image placed under /liveos/rootfs.img. Once it is found, it will be mounted, an overlay file system will be created, and then the kernel will pivot to it.
  • PXE Support:
    • To support the Live OS bootstrap PXE scenario, Dracut’s livenet module downloads the bootstrapped image (expects an ISO image with an embedded /liveos/rootfs.img), and then passes it to the dmsquash module as in the ISO scenario describe above.