TFTP Boot and NFS Root Filesystems

Revision as of 11:50, 12 September 2014 by Greytery (talk | contribs) (minor text updates and typo correction)
Jump to: navigation, search

How To Set Up TFTP Boot And NFS Root Filesystems On Parallella


Linux PC, Parallella board(s), USB to TTL/UART adapter, DHCP server


This is an approach to booting the Parallella using TFTP, and mounting the root filesystem via NFS. It should be useful to the owner of a Parallella cluster, or even the owner of a single Parallella.
Each Parallella is modified to fetch a bootfile script via TFTP, which points at a bootmode TFTP directory containing the loadable images.
The bootfile also sets up the NFS boot parameters.
By editing the bootfile on the TFTP server, the Parallella can quickly and easily be re-configured to a different bootmode, and/or change the NFS Filesystem - all without having to 'burn' another micro-SD card (ever..??).
Although it requires a bit of preparation, the approach involves minimal changes to each Parallella U-Boot environment using the USB to UART cable, and then all the loading and re-configuration is controlled on the TFTP/NFS host - in this case, a Debian PC.

This Guide describes:

  • An addressing and naming scheme to support the choice of bootmode (headless, HDMI) and NFS root filesystems.
  • A directory layout for TFTP bootfiles and bootmode image files
  • Changing the way the boot command arguments are passed to the kernel, i.e. in a U-Boot script rather than devicetree
  • A directory layout for NFS root filesystems, allowing a choice of filesystems for each Parallella
  • The minimum changes to the U-Boot variables on each board to trigger a network TFTP boot instead of micro-SD card boot
  • An example script for TFTP boot and NFS root filesystems.
  • Use of mkimage to convert the bootfile text file into a U-Boot script image file.

There are many examples and reference material to illustrate the flexibility of U-Boot, and TFTP booting for ARM-based and embedded systems - you only need to search for them. TFTP booting appears to be normal for small embedded systems. See the Manual for Das U-Boot
The following is originally based on the blog post by Scott Johnson describing how to set up a Parallella cluster to use TFTP/NFS, but along the line it has been tweaked - mainly to minimise the need to edit/update the U-Boot variables via the UART cable, and make more use of the PC-based editors, etc.
Tweak the example below to suit your local needs ...


  • There appears to be a problem in Parallella U-Boot w.r.t. the Ethernet interface after any network activity in U-Boot (e.g. TFTP). (See the post on the Parallella forum post by tajama). In effect, the network interface is left in an unuseable state. A workround is included in the script example below; but before attempting to apply it - you MUST check the U-Boot version. (To be revisited)
  • As of early Sept 2014, the Parallella U-Boot is to being updated, which may affect the script below. (To be revisited).

IP Addressing and Naming

The approach relies on consistent IP address allocation and naming:

  • On the Parallella, the U-Boot tftpboot command uses the IP address of the TFTP server <serverip>, its own IP address <ipaddr>.
  • The tftpbooot command will request a file based on a unique hostname.
  • On booting, U-Boot will tell Linux to use DHCP.
  • The NFS service uses the IP address of the Parallella to control access to the exported NFS directories.

The IP addresses for the Parallella(s) and the TFTP/NFS servers are pre-allocated in the DHCP server - based on MAC address.
The MAC address of each Parallella is printed on a label on the board, but is also held as a U-Boot variable (see later).
Refer to the documentation of your DHCP server, or discuss with your network admin. In my case, the DHCP server runs in the ISP-supplied home router.

Note: the TFTP and NFS services do not need to be on the same physical server, or even the same operating system. Here, a Debian (Wheezy) PC provides both services.

The Parallella naming convention and IP allocation (for this example) is as follows:	para1	para2	para3	para4
...   para8     	// ... I wish	TFTP server	// IP address of PC TFTP server	NFS server	// in this case, both services are on the same Debian PC

Set up the TFTP and NFS services

On Debian, get the required packages, if not already set up.

apt-get install tftpd-hpa nfs-kernel-server

By default, the NFS service is configured to start automatically, but the tftpd_hpa service is not. (There are a few ways to start the tftpd-hpa service automatically - not covered here) To control the tftp and NFS services use:

sudo service tftpd-hpa start   	        // or 'stop', or 'restart' as needed.
sudo service nfs-kernel-server restart	// or 'stop', or 'start' as needed.

The Bootfile Directories

By default, the tftpd-hpa service looks for requested files in the /srv/tftp/ directory. The directory is owned by root, so use sudo for file commands here (or run as root).

The Parallella boot files are prepared under /srv/tftp/ as follows.
Each bootmode (e.g. headless_7010 or HDMI_7010) has a separate directory under /srv/tftp. Note: that is one directory per bootmode - not one per Parallella. Different types or generations of image files can be held in different bootmode directories, e.g. hdmi_7010_new_kernel/, test_usb_fix/.
Each directory contains the equivalent of a BOOT partition on a micro-SD card, containing the kernel, parallella.bit.bin and devicetree images. To create the contents, follow the instructions for creating a micro-SD card, that is, download, extract and rename the kernel uImage, parallella.bit.bin and devicetree.dtb files. Instead of 'burning' to a micro-SD card, simply copy the image files to the bootmode directory (as root or using sudo).

Each Parallella has its own bootfile image, e.g. para1. This file is a U-Boot script, which is created using mkimage from a text file, e.g. para1 (see below). The bootfile points U-Boot to the bootmode directory. The bootfile also tells U-Boot to set up the Linux boot parameters to point to the NFS root filesystem, and finally to boot the kernel (see below).

The directory layout (for this example) looks like:

/svr/tftp/				// TFTP service directory
	hdmi_7010/			// bootmode: Zynq 7010 with HDMI support
	headless_7010/			// bootmode: Zynq 7010 Parallella configured as headless
	hdmi_7010_new_kernel/		// bootmode: example testing variation
	para1			        // initial bootfile for para1
	para2			        // initial bootfile for para2
	para3			        // initial bootfile for para3
	para4			        // initial bootfile for para4
	para8			        // etc, etc

Edit the Devicetree

In this approach, the kernel boot command parameters are created in the bootfile script and passed to the kernel by U-Boot rather than in the devicetree.dtb file. To change the devicetree, install the devicetree compiler if not already installed, and edit the supplied devicetree.dtb in each bootmode directory to remove the parameters which boot from the micro-SD card.

sudo apt-get install device-tree-compiler

Produce a .dts text file from the existing devicetree.dtb file.

cd /srv/tftp/hdmi_7010      // for example

sudo dtc -I dtb -o dev.dts -O dts devicetree.dtb

Edit the devicetree text file to remove the mmc boot parameters from the 'chosen' branch, leaving the UART console line (if required).

sudo nano dev.dts

(Edit and save.)

	chosen {
		linux,stdout-path = "/amba@0/uart@E0001000";

Recompile the devicetree.dtb in the bootmode directory, and delete the dev.dts file.

sudo dtc -I dts -O dtb -o devicetree.dtb dev.dts
sudo rm dev.dts

Prepare the NFS Root Filesystems

Each Parallella is provided with a separate NFS directory containing its root filesystem. In this example, the NFS directories are placed under /srv/nfs.
Just as with preparing and swapping different micro-SD cards, each Parallella could have different versions of root filesystem, (e.g. standard Ubuntu, development and experimentation, alternative builds). A Parallella can be re-purposed by rebooting after editing its bootfile. Here is an example layout of NFS Filesystem directories showing separate filesystems for each Parallella, and variations.

	para1/		// root filesystem for para1, based on, say, standard Parallella Ubuntu
		...	// etc, etc
	para1_deb/	// root filesystem for para1, with 9600's Debian and ESDK
		...	// etc, etc
	para1_nano/	// root filesystem for para1 based on linaro_nano
		 ...	// etc, etc	
	para2/		// root filesystem for para2
		...	// etc, etc
	para8/	       // root filesystem for para8
	...	       // etc, etc

It is possible to share the same root filesystem between two or more Parallellas, but different temporary filestore, etc. needs to be set up, (not covered here). Since there's loads of space(?) on the server disk(s), duplication is not costly in terms of storage. Care needs to be taken to manage version control (but it's much quicker and easier to manage/fix than re-burning a stack of micro-SD cards).

Place the payload root filesystem in the target directory e.g. /srv/nfs/para1/
This can be the default Parallella Ubuntu build, or, say, 9600's Debian images, or follow shodrucky's nano build from here [[1]], with some mods as below.
For example, download the linaro-saucy-nano image.


Extract the filesystem image to the NFS directory for the target Parallella.

sudo tar --strip-components=1 -C /srv/nfs/para1_nano -xzpf ~/Downloads/linaro-saucy-nano-20140410-652.tar.gz

Edit some configuration files as per shodrucky's notes. Important note: To avoid the mistake of editing the host PC's configuration, make sure you are in the correct NFS subdirectory!!!.

cd /srv/nfs/para1_nano		// for example; Note the lack of '/' before etc/ !!

Prevent installation of unnecessary packages

sudo nano etc/apt/apt.conf.d/00InstallRecommends

(Edit and save.)

    APT::Install-Recommends "false";

Network configuration

The assumption is that DHCP will be used. Edit the interfaces file. I DONT THINK YOU NEED TO DO THIS BECAUSE WE'VE ALREADY TOLD LINUX TO USE DHCP - TO BE REVISITED

sudo nano /srv/nfs/para1/etc/network/interfaces

(Edit and save.)

    source-directory /etc/network/interfaces.d
    auto lo
    iface lo inet loopback
    auto eth0
    # If you prefer DHCP, comment out the above 5 lines, uncomment the below.
    iface eth0 inet dhcp
       up sleep 3; mii-tool -F 1000baseT-FD
sudo nano etc/resolv.conf   //// ???? is this needed
(Edit and save.)

Set Up The NFS Filesystem Exports

The NFS server will require /etc/exports to be configured correctly, and export each NFS Filessystem directory to specific hosts only.

sudo nano /etc/exports

For each Paralella and for root each filesystem add the following line to /etc/exports, adjusting (as you should for all examples herein) for your local IP addressing scheme. Note that the bootfile ensures that only one of the optional filesystems can be accessed by a Parallella on a re-boot. This is based on the example layout shown above.


Ensure that these directories are exported either by using the following command, or by restarting the NFS service, as follows:

sudo exportfs -a


sudo service nfs-kernel-server restart

Change The Parallella to Boot Using TFTP

This is the only change required on the Parallella itself.
Remove any Ethernet, USB and HDMI cables and micro-SD card.
Connect a USB serial cable to the 3-pin header on the Parallella. (Example device from Adafruit).
Use a terminal program (e.g. screen,minicom) to connect to the USB device. On minicom, set hardware flow control to off, and point the logfile to, say, ~/minicomlog for diagnostics.
Power up the Parallella.
After the boot failure messages, you should get the zynq-uboot> prompt in the terminal window.
This command displays a sorted list of the current state of the U-Boot environment variables.


Note the value of the 'ethaddr=xx:xx:xx:xx:xx:xx' variable and check that this is the MAC address you are using for this Parallella in your DHCP server. It should be the same as the label on the board. It cannot be changed at the prompt.
Enter the following commands at the zynq-uboot> prompt, adjusting the hostname and IP addresses. These commands will:

  • Set the hostname and IP address <ipaddr> of this Parallella, and the IP Address of the TFTP server <serverip>, based on the example above.
  • Set up a variable containing the TFTP boot sequence <t-boot>. This fetches the bootfile (based on <hostname>) for this Parallella into a memory address and executes it as a script. The bootfile is described below.
  • Change the U-Boot boot command so that it will run the <t-boot> TFTP boot sequence.
  • Alternatively, change the U-Boot command to try the SD card first, before trying TFTP boot
setenv hostname para1
setenv ipaddr
setenv serverip
set t-boot 'tftpboot 0x100000 ${hostname} ; source 0x100000'

EITHER (much preferred) replace the current bootcmd to run the t-boot sequence:

setenv bootcmd 'run t-boot'

OR (as a fall-back) replace the U-Boot bootcmd to try to boot from a micro-SD card first; if no micro-SD card is present, then run the TFTP boot sequence.

setenv bootcmd 'run modeboot ; run t-boot'

Check (!!) that the variables have been set correctly and, if OK, save to flash memory.

printenv    	// Check. Check.


  • It's worthwhile checking (again) after a power recycle that the variables have been set correctly
  • To correct/change a variable, use the setenv command with new/correct value.
  • To remove a variable e.g. 'badvarible', use the setenv command with no value, e.g. setenv badvarible
  • U-Boot autocompletes commands so 'print', 'set', 'save' will also work.

Reconnect Ethernet cable and any peripherals you intend to use on the Parallella. Ready to TFTP boot.

Example Bootfiles

On the host, the bootfiles are created and edited as text files, and compiled into images using mkimage for loading/executing by U-Boot.

Create a working directory to hold the bootfile text files.

mkdir ~/Documents/bootscr

In summary, the directory structure on the Debian PC looks like this example:


Example Bootfile

The following shows an example bootfile. (A breakdown and explanation of the components follows).

Create a text file and copy/paste the text below. Edit/copy the file to create a bootfile script for each Parallella, adjusting the values of the <tftp_dir> and <nfs_rootfs> variables for each bootfile as necessary.

cd ~/Documents/bootscr
nano para1    // or para2, etc., etc.

(edit and save)

setenv tftp_dir hdmi_7010
setenv nfs_rootfs /srv/tftp/para1

setenv nfs_ip
setenv bootargs root=/dev/nfs rw nfsroot=${nfs_ip}:${nfs_rootfs}
setenv bootargs ${bootargs} ip=dhcp console=ttyPS0,115200

setenv fpga_image parallella.bit.bin
setenv fpga_addr 0x4000000
setenv fpga_size 0x3dbafc
setenv fpga_tftp 'tftpboot ${fpga_addr} ${tftp_dir}/${fpga_image}'
setenv fpga_load 'fpga load 0 ${fpga_addr} ${fpga_size}'

setenv fdt_image devicetree.dtb
setenv fdt_addr 0x2A00000
setenv fdt_tftp 'tftpboot ${fdt_addr} ${tftp_dir}/${fdt_image}'

setenv kernel_image uImage
setenv kernel_addr 0x3000000
setenv kernel_tftp 'tftpboot ${kernel_addr} ${tftp_dir}/${kernel_image}'

setenv phy_rst 'mw.w f8000008 df0d ; mw.w f8000140 00100801 ; mw.w f8000004 767b'
setenv rset_phy 'run phy_rst'

setenv boot_now 'bootm ${kernel_addr} - ${fdt_addr}'

run fpga_tftp fpga_load kernel_tftp fdt_tftp rset_phy boot_now

Breakdown of the Bootfile

This section takes the bootfile example above and breaks it down to illustrate what may be changed where needed to support your local environment.

  • Set the bootfile to point to the required bootmode (i.e. TFTP sub-directory) and the NFS filesystem directory chosen for this Paralella, for this boot. By editing only these first two lines, a Parallella can be re-booted using a different FPGA, kernel and/or a different NFS root filesystem.
setenv tftp_dir hdmi_7010	   // or, headless_7010, etc
setenv nfs_rootfs /srv/nfs/para1   // or, para1_deb, etc
  • Set the NFS boot parameters to be passed to the kernel by U-Boot in the <bootargs> variable. The NFS server IP is not the same variable as the <serverip> variable, the latter is used by U-Boot as the TFTP server. Note that the kernel is told to use DHCP when loaded.
setenv nfs_ip     // your NFS server IP, usually the same for all your Parallellas
setenv bootargs root=/dev/nfs rw nfsroot=${nfs_ip}:${nfs_rootfs}
setenv bootargs ${bootargs} ip=dhcp console=ttyPS0,115200
    • To add diagnostic output to the console, add (or remove) the following line to the end of the bootargs block; this will append additional parameters to the existing string.
setenv bootargs ${bootargs} nfsrootdebug earlyprintk
  • Set the tftpboot variables to request the fpga, devicetree and kernel images. This is exactly equivalent to the sequence of commands that load from the micro SD card.
setenv fpga_image parallella.bit.bin
setenv fpga_addr 0x4000000
setenv fpga_size 0x3dbafc
setenv fpga_tftp 'tftpboot ${fpga_addr} ${tftp_dir}/${fpga_image}'
setenv fpga_load 'fpga load 0 ${fpga_addr} ${fpga_size}'

setenv fdt_image devicetree.dtb
setenv fdt_addr 0x2A00000
setenv t_fdt 'tftpboot ${fdt_addr} ${tftp_dir}/${fdt_image}'

setenv kernel_image uImage
setenv kernel_addr 0x3000000
setenv kernel_tftp 'tftpboot ${kernel_addr} ${tftp_dir}/${kernel_image}'
  • NB: The following is a workround for the problem identified by tajama, and referred to above.
   IMPORTANT: this is a patch to U-Boot space and applies to the following version:
U-Boot 2012.10-00003-g792c31c (Jan 03 2014 - 12:24:08)

It MAY apply to your version but YOU MUST CHECK!! To be revisited if/when a new version of Parallella U-Boot build is provided.

setenv phy_rst 'mw.w f8000008 df0d ; mw.w f8000140 00100801 ; mw.w f8000004 767b'
setenv rset_phy 'run phy_rst'
  • Set up the command to boot into the kernel, passing the kernel and devicetree image addresses.
setenv boot_now 'bootm ${kernel_addr} - ${fdt_addr}'
  • Run the sequence of tftp requests, and boot. If any of the commands passed to 'run' fails, then the 'run' command will halt (note - there are no ';' separators).
run fpga_tftp fpga_load kernel_tftp fdt_tftp rset_phy boot_now

Make the boot script image

Install the mkimage package if not already installed. On Debian, this is as follows:

sudo apt-get install u-boot-tools

After each bootfile script has been created/edited, it must be made into a loadable image by mkimage, and the image placed in the TFTP server directory. Based on the example above, the source text file is in a user working directory, and the TFTP service will expect to find the bootfile image in the TFTP server directory.

sudo mkimage -A arm -O u-boot -T script -C none -a 0 -e 0 -n "t-Boot Script" -d ~/Documents/bootscr/para1 /srv/tftp/para1

Reboot the Parallella

Either recycle power or press the reset button. Given the reported problems with USB and HDMI initialisation, most success seems to be after full power recycle.

The output to the USB/UART terminal screen should show initialisation of the board and network, and then show the boot sequence looking for the TFTP server. If the TFTP 'Loading:' line puts out a few 'T T T's, then it's retrying. Give it a few moments but if there's a line of 'T's then check that the TFTP service is actually running, and that the IP addresses and filename of the bootfile are valid.
Once the small (~1kb) bootfile has been loaded and is executed by U-Boot, further TFTP requests will be seen as the image files are downloaded before U-Boot hands over to the kernel.