Squash FS Howto

SquashFS HOWTO
Artemiy I. Pavlov

Artemio.net



2004-06-07 Revision History Revision 1.5 2004-06-07 Changes according to SquashFS release 2.0 alpha. Lots of description improvements and clarifications. Split instructions for Linux kernels of 2.6.x (new) and 2.4.x series. Revision 1.1 2004-05-22 Changes according to SquashFS release 1.3r3. Revision 1.0 2004-02-19 Initial Release, reviewed by LDP. Revision 0.2 2003-12-08 Text corrections, license added. Revision 0.1 2003-11-24 Initial version. Instructions for SquashFS release 1.3r2.

Abstract

This  HOWTO   describes   the   usage   of   SquashFS   -   a highly-compressed  read-only  file  system for Linux, which is intended  for  use  in  tiny-sized  and  embedded systems, and anywhere else you'd want to use a compressed file system. With this document,  you'll  learn how to prepare a SquashFS-ready Linux  kernel,  create  a sqaushed file system and happily use it.

Home of this HOWTO

The SquashFS HOWTO lives at http://artemio.net/projects/linuxdoc/squashfs. There you will always find  the  latest version of the document, and will be able to send your feedback.

Introduction
When creating  tiny-sized  and  embedded Linux systems, every byte  of the storage device (floppy, flash disk, etc.) is very important,  so  compression is used everywhere possible. Also, compressed file  systems  are frequently needed for archiving purposes. For huge  public archives, as well as for personal media archives, this is essential.

SquashFS brings  all  this  to a new level. It is a read-only file system  that  lets  you  compress  whole file systems or single  directories, write them to other devices/partitions or to  ordinary files, and then mount them directly (if a device) or  using  a  loopback  device (if it is a file). The modular, compact system  design  of  SquashFS  is bliss. For archiving purposes, SquashFS  gives  you  a  lot  more  flexibility and performance speed than a .tar.gz archive.

SquashFS is distributed as a Linux kernel source patch (which enables  SquashFS  read  support  in  your  kernel),  and  the mksquashfs  tool,  which  creates  squashed file systems (in a file or on a block device).

The latest  SquashFS  release tree is 2.x, the former one was 1.x.  This  document describes both these releases with proper notes  given. For example,  if  some feature or parameter is different  in  these  release  trees,  it  will  be written as follows: new value (2.x) or old value (1.x)

Overview of SquashFS

 * Data, inodes and directories are compressed
 * SquashFS stores full uid/gids (32 bits), and file creation time
 * Files up to 2^32 bytes are supported; file systems can be up to 2^32 bytes
 * Inode and directory data are highly compacted, and packed on  byte boundaries; each compressed inode is on average 8 bytes  in  length  (the  exact length varies on file type, i.e.   regular   file,   directory,   symbolic  link,  and block/character device inodes have different sizes)
 * SquashFS can  use  block sizes up to 32 Kb (1.x) and 64Kb (2.x),  which achieves greater compression ratios than the normal 4K block size
 * SquashFS 2.x inroduced the concept of fragment blocks: an ability  to  join  multiple  files smaller than block size into a single block, achieving greater compression ratios
 * File duplicates are detected and removed
 * Both big  and  little endian architectures are supported; SquashFS  can  mount  file  systems  created  on different byte-order machines

Making it clear
Now let's  make  sure any further discussions will be clearer for  you  to  understand. The procedure  of getting SquashFS working, basically, consists of the following steps:
 * 1) Patching and recompiling the target Linux kernel to enable SquashFS support
 * 2) Compiling the mksquashfs tool
 * 3) Creating a compressed file system with mksquashfs
 * 4) Testing: mounting  a  squashed file system to a temporary location
 * 5) Modifying the /etc/fstab or startup scripts of your target Linux system  to  mount the new squashed file system when needed

Acquiring SquashFS
The SquashFS home site is located at http://squashfs.sourceforge.net/ -  it  contains news for the latest   release  and  it's  changelog,  as  well  as  general information about SquashFS. You can grab the latest version at the SqaushFS project page at SourceForge.

Preparing a SquashFS-capable kernel
In order  to  read  SquashFS,  you  need it supported in your kernel - just as if it was a reiserfs or ext3 file system. You have to  make  sure  there  is  an appropriate patch for your kernel   version   -  it  should  be  located  in  linux-2.x.y subdirectory  of the SquashFS source tree. Also, remember that in most  cases  you will need a clean (original) Linux kernel source from kernel.org. If your kernel source is from a distro vendor, it  may  be  already  pre-patched  with custom vendor patches, and patching with a SquashFS patch will almost surely not  work, as SquashFS patches are made against original Linux kernels.

Patching the kernel source
With a kernel source and a proper SquashFS patch present, all you  have  to  do  is  (we'll  assume that you have your Linux kernel source in /usr/src/linux and that you have the SquashFS source in /usr/src/squashfs):

Change to  the  SquashFS source directory and copy the kernel patch    (we'll   assume   it's   named   squashfs-patch) to /usr/src/linux.

Go to the linux kernel source directory /usr/src/linux:

Note: please  remember  that  we  will  not  be  leaving this directory  during  all  further kernel-related procedures, and all paths will be given relative to /usr/src/linux.

Now patch the source with the SquashFS patch:

Compiling a 2.6.x kernel
Cleanup and prepare the kernel source:

Configure   the    kernel   using   your   favourite method (config/menuconfig/xconfig/gconfig):


 * 1) In  the   "File  systems"  section,  "Miscellaneous  file systems"  subsection,  enable  the  "Squashed  filesystem" option,  whether  as module or bundled with the kernel. It is  only  obligatory to compile SquashFS inside the kernel if you plan using squashed initial RAM disks (initrd).
 * 2) If you  would  like  to  use a squashed initial RAM disk, enable  the  "Initial  RAM  disk  support"  in the "Device drivets" section, "Block devices" subsection. If  you  want to be able to mount the squashed file system via  a  loopback  device  in  future,  you  should  enable
 * 3) "Loopback device support" in the "Device drivers" section,
 * 4) "Block devices" subsection.

Now you may compile the kernel and modules:

Compiling a 2.4.x kernel
Configure the kernel:


 * 1) In the  "File  systems"  section,  enable  the  "Squashed filesystem"  option, whether as module or bundled with the kernel.  It  is only obligatory to compile SquashFS inside the  kernel  if  you plan using squashed initial RAM disks (initrd).
 * 2) If you  would  like  to  use a squashed initial RAM disk, enable  the  "Initial  RAM  disk  support"  in  the "Block devices" section.
 * 3) If you  want to be able to mount the squashed file system via  a  loopback  device  in  future,  you  should  enable "Loopback device support" in the "Block devices" section.

Now you may compile the kernel and modules:

Installing and testing the kernel
It's time  to  install  your new SquashFS-enabled kernel. The instructions below  are for installing and booting the kernel on  the  host  machine. You may want to install and test it on the target system.

We assume that the kernel was compiled for a x86 architecture, and  the   compressed   kernel   image   is  located  in  the arch/i386/boot/  subdirectory of the kernel tree. Now copy the kernel to  the  /boot directory (and name it bzImage-sqsh for convenience, if you like):

Don't forget to install the kernel modules if you have any:

Modify your  boot loader's configuration file to include your new  kernel  and install (update) the boot loader. Now you may reboot with  your  new  kernel. When it  boots,  check that everything went fine:

Or, if you built SquashFS support as a kernel module:

If you  see  the squashfs line among other file systems, this means you have successfully enabled SquashFS in your kernel.

Compiling the mksquashfs tool
Now you  need  to  compile mksquashfs - the tool for creating squashed file systems.

Compile and install mksquashfs:

If everything went fine, typing mksquashfs at the shell prompt should print it's "usage" message.

Using mksquashfs
mksquashfs is the tool for creating new squashed file systems, and for appending new data to existing squashed file systems. The general command-line format for mksquashfs is:


 * source1, source2, etc.: files and directories to be added to  the resulting filke system, given with relative and/or absolute paths
 * destination: a regular file (filesystem image file), or a   block  device  (such  as  /dev/fd0 or /dev/hda3) where you   want to have your squashed file system

Notes for default mksquashfs behavior:
 * When the  new  files  are added to the new file system or   appended to an existing one, mksquashfs will automatically   rename  files  with  duplicate names: if two or more files named  text  will  appear in the same resulting directory, the  second  file  will be renamed to text_1, third one to text_2 and so on.
 * Duplicate files will be removed, so there will be only one physical instance (with SquashFS 2.x, you can disable the  detection/rtemoval    of    the    duplicates   with   the -no-duplicates option).
 * If destination has a pre-existing SquashFS file system on it,  by  default, the new source items will be appended to the  existing  root  directory.  Examine the options table below to   force   mksquashfs  to  overwrite  the  whole destination  and/or  change  the  way new source items are added.  Please note that it is not possible to append to a file  system  created with mksquashfs 1.x using mksquashfs 2.x.  You  will need to mount the SquashFS-1.x file system and  copy  the  files to some location, and then join them with  other  needed  files  to  create a SquashFS-2.x file system.
 * If a single source file or directory is given, it becomes the  root  in  a newly created file system. If two or more source  files  and/or directories are given, they will all become sub-items in the root of the new file system.
 * The resulting filesystem will be padded to a multiple of 4Kb: this  is required for filesystems to be used on block devices.  If you are very sure you don't ned this, use the -nopad option to disable this operation.

See the  next  section  for  more  details about all possible options.

Command-line options
All possible  options  for  mksquashfs are shown in the table below.

In most cases, you should leave all compression/block options by  default,  as  they  allow  mksquashfs  to achieve the best possible compression ratios.

Basic steps
In order  to  create  a  squashed file system out of a single directory  (say,  /some/dir),  and output it to a regular file (thus,  producing  a  file system image), you need to say only one magic phrase:

mksquashfs will perform the squashing and print the resulting number  of  inodes  and  size  of data written, as well as the average   compression  ratio. Now you  have  your  /some/dir directory image  in  the  dir.sqsh  file. You can now use the mount command to mount it using a loopback device:

To check if you have what's expected:

If you  want to output the file system directly into a device (say, your floppy at /dev/fd0):

Then just mount the device:

And check if it's okay:

Squashing file systems
Operations described  here  correspond  to most cases where a read-only compressed file system can be used, whether you want it  to  be  on  a  block  device  or  in a file. This could be anything from large FTP/HTTP-served archives that don't change often, to having a squashed /usr partition and anything alike with these.

Example 1
Let's suppose  you  have  a  /var/arch directory with lots of files and that you want to turn it into a squashed file system and  keep  it  on  your root partition as a file (it will be a file  system image that you will mount via a loopback device). The operations needed to perform are as follows.

Squash the directory, then mount it via loopback to test it:

If everything  is  as  expected,  make this file system mount automatically  at  boot  time  by  adding  this  line  to your /etc/fstab:

Unmount the  file  system from the temporary mount point, and mount using it's fstab entry:

Now just ensure that everything works fine:

Example 2
Say you  have  two  hard disk partitions, /dev/hda6 (which is empty) and /dev/hda7 (which is bigger than /dev/hda6, mounted at  /var/arch,  contains  some data and is full). Now, say you want to  squash  the  /dev/hda7  file  system  and move it to /dev/hda6, then use /dev/hda7 for some other purposes. We will suppose you have the following line in /etc/fstab (reiserfs is just an example file system used on /dev/hda7):

In the same fashion as with the previous example:

If everything went fine, unmount /dev/hda7 and use dd to copy /var/arch.sqsh to /dev/hda6:

Now change the line in /etc/fstab for /dev/hda7 to:

Mount the new file system and check to see if all went fine:

Don't forget to erase the unneeded file system image:

Creating tiny/embedded systems
By saying "tiny/embedded", I mean Linux systems that are being built for  booting  from  floppy  disks, IDE/USB flash disks, iso9660 CD-ROMs, small-sized hard drives and the like. Whether you want to have your whole root file system on a single media (a single  partition,  a  single  floppy),  or have a modular system (several floppies or disk partitions), the procedure is almost  identical. Creating such Linux systems themselves is out  of  scope  of this HOWTO - there are dedicated HOWTOs and guides  for  this  (like  the  Bootdisk  HOWTO  and Linux From Scratch - visit www.tldp.org to retrieve these documents).

Squashed file systems on floppy/flash/hard disks
In order  to use SquashFS for creating Linux systems on small disks,  you just have to follow the usual steps for creating a minimal   system,   performing  the  following  operations  at respective points:
 * 1) When developing  a  kernel for your system, make sure you enable  SquashFS  support  so  it  can mount squashed file systems
 * 2) Use mksquashfs  for  creating read-only initial ram disks and/or root and/or other file systems
 * 3) Don't forget  to  set  file  system  types to squashfs in /etc/fstab  and/or  the startup scripts of your system for mounting squashed file systems

Floppy example. Let's say you have your floppy system tree at /home/user/floppylinux and  you  want  to place the root file system  on  one floppy and /usr on another. What you should do is:

Note 1:  you can see here how we use the -e option to exclude the /usr directory for root file system's image.

Note 2:  don't forget to specify squashfs in your root disk's /etc/fstab  or  startup  scripts  when  mounting the /usr file system.

Insert a  root  disk  in your 3.5" floppy drive (I assume you have  a lilo or grub on it, and, thus, a file system exists on this  floppy,  and  the root file system will reside under the /boot directory of this file system):

When done,  unmount  the  root floppy, change the floppy to a /usr disk and use dd to transfer the usr file system:

Squashed file systems on CD-ROMs
With SquashFS,  you can compress large file systems that will be used in live CDs (just as an example).
 * 1) Enable SquashFS in the linux kernel of the target system
 * 2) Create a squashed root file system
 * 3) Modify the  /etc/fstab  or  startup scripts of the target system to mount the squashd file system when you need it

If you  create  a  root  file  system  out of a running Linux system,  use  the  -e  option  for  mksquashfs  to exclude all pseudo-filesystems such as /proc, /sys (on linux kernels after 2.5.x)  and /dev (when using DevFS). Also, don't forget to add the file  system  image  itself  that  is  being created with mksquashfs   (I   think   you   know  the  reasons  for  these exclusions).

Acknowledgements
I would  like  to  express my sincere thanks and immeasurable respect to:
 * Phillip Lougher  - for his brilliant work under squashfs, for  creating an exculsive patch for linux-2.4.18, for his help with polishing this howto and answers to my mails
 * Tabatha Marshall at TLDP for helping me with bringing this HOWTO to the final 1.0 release
 * Everybody at  The  Linux  Documentation Project for their great  work under all the HOWTOs and guides that helped me a lot with exploring and hacking Linux
 * All those  at  the  TLDP mailing lists who helped me with getting started
 * Endless thanks  and  respect  to  everybody  who develops open-source software

License
This document may be used and distributed under the terms and conditions  set  forth  in the Open Content licence. In short, this means  that  you can freely modify and re-distribute the HOWTO under  the  main condition that you keep the author and copyright  the  article along. The full text of the licence is available at http://www.opencontent.org/opl.shtml