TFTP Boot and NFS Root Filesystems

= How To Set Up TFTP Boot And NFS Root Filesystems On Parallella =

Requirements
Linux PC, Parallella board(s), USB to TTL/UART adapter, DHCP server

Summary
This is an approach to booting the Parallella using TFTP, and mounting the root filesystem via NFS. It should be useful to the owner of a Parallella cluster, or even the owner of a single Parallella.

Each Parallella is modified to fetch a bootfile script via TFTP, which points at a bootmode TFTP directory containing the loadable images.

The bootfile also sets up the NFS boot parameters.

By editing the bootfile on the TFTP server, the Parallella can quickly and easily be re-configured to a different bootmode, and/or change the NFS Filesystem - all without having to 'burn' another micro-SD card (ever..??).

Although it requires a bit of preparation, the approach involves minimal changes to each Parallella U-Boot environment using the USB to UART cable, and then all the loading and re-configuration is controlled on the TFTP/NFS host - in this case, a Debian PC.

This Guide describes:
 * An addressing and naming scheme to support the choice of bootmode (headless, HDMI) and NFS root filesystems.
 * A directory layout for TFTP bootfiles and bootmode image files
 * Changing the way the boot command arguments are passed to the kernel, i.e. in a U-Boot script rather than devicetree
 * A directory layout for NFS root filesystems, allowing a choice of filesystems for each Parallella
 * The minimum changes to the U-Boot variables on each board to trigger a network TFTP boot instead of micro-SD card boot
 * An example script for TFTP boot and NFS root filesystems.
 * Use of mkimage to convert the bootfile text file into a U-Boot script image file.

There are many examples and reference material to illustrate the flexibility of U-Boot, and TFTP booting for ARM-based and embedded systems - you only need to search for them. TFTP booting appears to be normal for small embedded systems. See the Manual for Das U-Boot

The following is originally based on the blog post by Scott Johnson describing how to set up a Parallella cluster to use TFTP/NFS, but along the line it has been tweaked - mainly to minimise the need to edit/update the U-Boot variables via the UART cable, and make more use of the PC-based editors, etc.

Tweak the example below to suit your local needs ...

IMPORTANT CAVEATS:
 * There appears to be a problem in Parallella U-Boot w.r.t. the Ethernet interface after any network activity in U-Boot (e.g. TFTP). (See the post on the Parallella forum post by tajama). In effect, the network interface is left in an unuseable state. A workround is included in the script example below; but before attempting to apply it - you MUST check the U-Boot version. (To be revisited)
 * As of early Sept 2014, the Parallella U-Boot is to being updated, which may affect the script below. (To be revisited).

IP Addressing and Naming
The approach relies on consistent IP address allocation and naming:
 * On the Parallella, the U-Boot TFTP boot command uses the IP address of the TFTP server, its own IP address , and a unique hostname.
 * On booting, U-Boot will tell Linux to use DHCP.
 * The NFS service uses the IP address of the Parallella to control access to the exported NFS directories.

The IP addresses for the Parallella(s) and the TFTP/NFS servers are pre-allocated in the DHCP server - based on MAC address.

The MAC address of each Parallella is printed on a label on the board, but is also held as a U-Boot variable (see later).

Refer to the documentation of your DHCP server, or discuss with your network admin. In my case, the DHCP server runs in the ISP-supplied home router.

Note: the TFTP and NFS services do not need to be on the same physical server, or even the same operating system. Here, a Debian (Wheezy) PC provides both services.

The Parallella naming convention and IP allocation (for this example) is as follows: 192.168.1.201	para1 192.168.1.202	para2 ... ... 192.168.1.208  para8     	// ... I wish

192.168.1.50	TFTP server	// IP address of PC TFTP server 192.168.1.50	NFS server	// in this case, both services are on the same Debian PC

Set up the TFTP and NFS services
On Debian, get the required packages, if not already set up. apt-get install tftpd-hpa nfs-kernel-server By default, the NFS service is configured to start automatically, but the tftpd_hpa service is not. (There are a few ways to start the tftpd-hpa service automatically - not covered here) To control the tftp and NFS services use: sudo service tftpd-hpa start  	        // or 'stop', or 'restart' as needed. sudo service nfs-kernel-server restart	// or 'stop', or 'start' as needed.

The Bootfile Directories
By default, the tftpd-hpa service looks for requested files in the /srv/tftp/ directory. The directory is owned by root, so use sudo for file commands here (or run as root).

The Parallella boot files are prepared under /srv/tftp/ as follows.

Each bootmode (e.g. headless_7010 or HDMI_7010) has a separate directory under /srv/tftp. Note: that is one directory per bootmode - not one per Parallella. Different types or generations of image files can be held in different bootmode directories, e.g. hdmi_7010_new_kernel/, test_usb_fix/.

Each directory contains the equivalent of a BOOT partition on a micro-SD card, containing the kernel, parallella.bit.bin and devicetree images. To create the contents, follow the instructions for creating a micro-SD card, that is, download, extract and rename the kernel uImage, parallella.bit.bin and devicetree.dtb files. Instead of 'burning' to a micro-SD card, simply copy the image files to the bootmode directory (as root or using sudo).

Each Parallella has its own bootfile image, e.g. para1_boot.scr. This file is a U-Boot script, which is created using mkimage from a text file, e.g. para1_boot.script (see below). The bootfile points U-Boot to the bootmode directory. The bootfile also tells U-Boot to set up the Linux boot parameters to point to the NFS root filesystem, and finally to boot the kernel. (see below).

The directory layout (for this example) looks like: /svr/tftp/				// TFTP service directory hdmi_7010/			// bootmode: Zynq 7010 with HDMI support paralella.bit.bin devicetree.dtb uImage headless_7010/			// bootmode: Zynq 7010 Parallella configured as headless paralella.bit.bin devicetree.dtb uImage hdmi_7010_new_kernel/		// bootmode: example testing variation paralella.bit.bin devicetree.dtb uImage para1_boot.scr			// initial bootfile for para1 para2_boot.scr			// initial bootfile for para2 ...	para8_boot.scr			// etc, etc

Edit the Devicetree
In this approach, the kernel boot command parameters are created in the bootfile script and passed to the kernel by U-Boot rather than in the devicetree.dtb file. To change the devicetree, install the devicetree compiler if not already installed, and edit the supplied devicetree.dtb in each bootmode directory to remove the parameters which boot from the micro-SD card. sudo apt-get install device-tree-compiler Produce a .dts text file from the existing devicetree.dtb file. cd /srv/tftp/hdmi_7010     // for example

sudo dtc -I dtb -o dev.dts -O dts devicetree.dtb Edit the devicetree text file to remove the micro-SD card mmc boot parameters from the 'chosen' branch, leaving the UART console line (if required). sudo nano dev.dts (Edit and save.) chosen { linux,stdout-path = "/amba@0/uart@E0001000"; } Recompile the devicetree.dtb in the bootmode directory, and delete the dev.dts file. sudo dtc -I dts -O dtb -o devicetree.dtb dev.dts sudo rm dev.dts

Prepare the NFS Root Filesystems
Each Parallella is provided with a separate NFS directory containing its root filesystem. The NFS directories are placed under /srv/nfs (in this example).

Just as with preparing and swapping different micro-SD cards each Parallella could have different versions of root filesyatem, (e.g. standard Ubuntu, development and experimentation, alternative builds). A Parallella can be re-purposed by rebooting after editing its bootfile. Here is an example layout of NFS Filesystem directories showing separate filesystems for each Parallella, and variations. /srv/nfs/ para1/		// root filesystem for para1, based on, say, standard Parallella Ubuntu bin/ boot/ etc/ ...	// etc, etc para1_deb/	// root filesystem for para1, with 9600's Debian and ESDK bin/ boot/ etc/ ...	// etc, etc para1_nano/	// root filesystem for para1 based on linaro_nano bin/ boot/ etc/ ...	// etc, etc para2/		// root filesystem for para2 bin/ boot/ etc/ ...	// etc, etc para8/	// root filesystem for para8 ...	// etc, etc

It is possible to share the same root Filesystem between two or more Parallellas, but different temporary filestore, etc. need to be set up, (not covered here). Since there's loads of space(?) on the server disk(s), duplication is not costly in terms of storage. Care needs to be taken to manage version control (but it's quicker and easier to manage/fix than re-burning a stack of micro-SD cards).

Place the payload root filesystem in the target directory e.g. /srv/nfs/para1/

This can be the default Parallella Ubuntu build, or, say, 9600's Debian images, or follow shodrucky's nano build from here [|Linaro-nano], with some mods as below.

For example, download the linaro-saucy-nano image. wget http://releases.linaro.org/14.04/ubuntu/saucy-images/nano/linaro-saucy-nano-20140410-652.tar.gz Extract the filesystem image to the NFS directory for the target Parallella. sudo tar --strip-components=1 -C /srv/nfs/para1_nano -xzpf ~/Downloads/linaro-saucy-nano-20140410-652.tar.gz Edit some configuration files as per shodrucky's notes. Important note: To avoid the mistake of editing the host PC's configuration, make sure you are in the correct NFS subdirectory. cd /srv/nfs/para1_nano		// for example; Note the lack of '/' before etc/ !!

Prevent installation of unnecessary packages
sudo nano etc/apt/apt.conf.d/00InstallRecommends (Edit and save.) APT::Install-Recommends "false";

Network configuration
The assumption is that DHCP will be used. Edit the interfaces file. I DONT THINK YOU NEED TO DO THIS BECAUSE WE'VE ALREADY TOLD LINUX TO USE DHCP - TO BE REVISITED sudo nano /srv/nfs/para1/etc/network/interfaces (Edit and save.) source-directory /etc/network/interfaces.d   auto lo    iface lo inet loopback auto eth0 # If you prefer DHCP, comment out the above 5 lines, uncomment the below. iface eth0 inet dhcp up sleep 3; mii-tool -F 1000baseT-FD sudo nano etc/resolv.conf  //// ???? is this needed (Edit and save.) nameserver 192.168.1.1

Set Up The NFS Filesystem Exports
The NFS server will require /etc/exports to be configured correctly, and export each NFS Filessystem directory to specific hosts only. sudo nano /etc/exports For each Paralella and for root each filesystem add the following line to /etc/exports, adjusting (as you should for all examples herein) for your local IP addressing scheme. Note that the bootfile ensures that only one of the optional filesystems can be accessed by a Parallella on a re-boot. This is based on the example layout shown above. /srv/nfs/para1 192.168.1.201(rw,sync,no_root_squash,no_subtree_check) /srv/nfs/para1_dev 192.168.1.201(rw,sync,no_root_squash,no_subtree_check) /srv/nfs/para1_nano 192.168.1.201(rw,sync,no_root_squash,no_subtree_check) /srv/nfs/para2 192.168.1.202(rw,sync,no_root_squash,no_subtree_check) ... ... /srv/nfs/para8 192.168.1.208(rw,sync,no_root_squash,no_subtree_check)

Ensure that these directories are exported either by using the following command, or by restarting the NFS service, as follows: sudo exportfs -a or sudo service nfs-kernel-server restart

Change The Parallella to Boot Using TFTP
This is the only change required on the Parallella itself.

Remove any Ethernet, USB and HDMI cables and micro-SD card.

Connect a USB serial cable to the 3-pin header on the Parallella. (Example device http://www.adafruit.com/products/954 from Adafruit).

Use a terminal program (e.g. screen,minicom) to connect to the USB device. On minicom, set hardware flow control to off, and point the logfile to, say, ~/minicomlog for diagnostics.

Power up the Parallella.

After the boot failure messages, you should get the zynq-uboot> prompt in the terminal window.

This command displays a sorted list of the current state of the U-Boot environment variables. printenv Note the value of the 'ethaddr=xx:xx:xx:xx:xx:xx' variable and check that this is the MAC address you are using for this Parallella in your DHCP server. It should be the same as the label on the board. It cannot be changed at the prompt.

Enter the following commands at the zynq-uboot> prompt, adjusting the hostname and IP addresses. These commands will: setenv hostname para1 setenv ipaddr 192.168.1.201 setenv serverip 192.168.1.50
 * Set the hostname and IP address of this Parallella, and the IP Address of the TFTP server, based on the example above.
 * Set up a variable containing the TFTP boot sequence . This fetches the bootfile (based on ) for this Parallella into memory at  and executes it as a script. The bootfile is described below.
 * Change the U-Boot boot command so that it will run the  TFTP boot sequence.
 * Alternatively, change the U-Boot command to try the SD card first, before trying TFTP boot

setenv t-addr 0x100000 set t-boot 'tftpboot ${t-addr} ${hostname}_boot.scr ; source ${t-addr}' EITHER (preferred) replace the current bootcmd to run the t-boot sequence setenv bootcmd 'run t-boot' OR (as a fall-back) replace the U-Boot bootcmd to try to boot from an micro-SD card first; if no micro-SD card is present, then run the TFTP boot sequence. setenv bootcmd 'run modeboot ; run t-boot' Check (!!) that the variables have been set correctly and, if OK, save to flash memory. printenv   	// Check. Check. saveenv Notes:
 * It's worthwhile checking (again) after a power recycle that the variables have been set correctly
 * To correct/change a variable, use the setenv command with new/correct value.
 * To remove a variable e.g. 'badvarible', use the setenv command with no value, e.g. setenv badvarible
 * U-Boot autocompletes commands so 'print', 'set', 'save' will also work.

Reconnect Ethernet cable and any peripherals you intend to use on the Parallella. Ready to TFTP boot.

Example Bootfiles
On the host, the bootfiles are created and edited as text files, and compiled into images by mkimage for loading/executing by U-Boot.

Create a directory to hold the working text files. mkdir ~/Documents/bootscr In summary, the directory structure on the Debian PC looks like this example: /home/ /Documents/bootscr/ para1_boot.script para2_boot.script ...	...	para8_boot.script

Example Bootfile
The following shows an example bootfile. (A breakdown and explanation of the components follows).

Create a file and copy/paste the text below. Edit/copy the file to create a bootfile script for each Parallella, adjusting the values of the  and  variables for each bootfile as necessary. nano para1_boot.script   // para2_boot.script, etc., etc. (edit and save) setenv tftp_dir hdmi_7010 setenv nfs_dir para1

setenv nfs_ip 192.168.1.52 setenv nfs_root '/srv/nfs' setenv bootargs root=/dev/nfs rw nfsroot=${nfs_ip}:${nfs_root}/${nfs_dir} setenv bootargs ${bootargs} ip=dhcp console=ttyPS0,115200

setenv fpga_image parallella.bit.bin setenv fpga_addr 0x4000000 setenv fpga_size 0x3dbafc setenv fpga_tftp 'tftpboot ${fpga_addr} ${tftp_dir}/${fpga_image}' setenv fpga_load 'fpga load 0 ${fpga_addr} ${fpga_size}'

setenv fdt_image devicetree.dtb setenv fdt_addr 0x2A00000 setenv fdt_tftp 'tftpboot ${fdt_addr} ${tftp_dir}/${fdt_image}'

setenv kernel_image uImage setenv kernel_addr 0x3000000 setenv kernel_tftp 'tftpboot ${kernel_addr} ${tftp_dir}/${kernel_image}'

setenv phy_rst 'mw.w f8000008 df0d ; mw.w f8000140 00100801 ; mw.w f8000004 767b' setenv rset_phy 'run phy_rst'

setenv boot_now 'bootm ${kernel_addr} - ${fdt_addr}'

run fpga_tftp fpga_load kernel_tftp fdt_tftp rset_phy boot_now

Breakdown of the Bootfile
This section takes the bootfile example above and breaks it down to illustrate what may be changed where needed to support your local environment.

setenv tftp_dir hdmi_7010	// or, headless_7010, etc setenv nfs_dir para1		// or, para1_deb, etc setenv nfs_ip 192.168.1.52    // your NFS server IP, usually the same for all your Parallellas setenv nfs_root '/srv/nfs'    // change to apply to your NFS directory structure setenv bootargs root=/dev/nfs rw nfsroot=${nfs_ip}:${nfs_root}/${nfs_dir} setenv bootargs ${bootargs} ip=dhcp console=ttyPS0,115200 setenv bootargs ${bootargs} nfsrootdebug earlyprintk setenv fpga_image parallella.bit.bin setenv fpga_addr 0x4000000 setenv fpga_size 0x3dbafc setenv fpga_tftp 'tftpboot ${fpga_addr} ${tftp_dir}/${fpga_image}' setenv fpga_load 'fpga load 0 ${fpga_addr} ${fpga_size}'
 * Set the bootfile to point to the required bootmode (i.e. TFTP directory) and the NFS directory chosen for this Paralella, for this boot. By editing only these first two lines, a Parallella can boot using a different FPGA, kernel and/or a different NFS root filesystem.
 * Set the NFS boot parameters to be passed to the kernel by U-Boot in the variable. The NFS server IP is not the same variable as the variable, the latter is used by U-Boot as the TFTP server. Note that the kernel is told to use DHCP when loaded.
 * To add diagnostic output to the console, add/remove the following line to the end of the bootargs block; this will append the additional parameters to the existing string.
 * Set the TFTP boot variables to request the fpga, devicetree and kernel images. This is equivalent to the sequence of commands that load from the micro SD card.

setenv fdt_image devicetree.dtb setenv fdt_addr 0x2A00000 setenv t_fdt 'tftpboot ${fdt_addr} ${tftp_dir}/${fdt_image}'

setenv kernel_image uImage setenv kernel_addr 0x3000000 setenv kernel_tftp 'tftpboot ${kernel_addr} ${tftp_dir}/${kernel_image}' IMPORTANT: this is a patch to U-Boot space and applies to the following version: U-Boot 2012.10-00003-g792c31c (Jan 03 2014 - 12:24:08) It MAY apply to your version but YOU MUST CHECK!! To be revisited if/when a new version of Parallella U-Boot build is provided. setenv phy_rst 'mw.w f8000008 df0d ; mw.w f8000140 00100801 ; mw.w f8000004 767b' setenv rset_phy 'run phy_rst' setenv boot_now 'bootm ${kernel_addr} - ${fdt_addr}' run fpga_tftp fpga_load kernel_tftp fdt_tftp rset_phy boot_now
 * NB: The following is a workround for the problem identified by tajama, and referred to above.
 * Set up the command to boot into the kernel, passing the kernel and devicetree image addresses.
 * Run the sequence of tftp requests, and boot. If any of the commands passed to 'run' fails, then the 'run' command will halt (note - there are no ';' separators).

Make the boot script image
Install the mkimage package if not already installed. On Debian, this is as follows: sudo apt-get install u-boot-tools

After each bootfile scripts has been created/edited, it must be made into a loadable image by mkimage, and the image loaded into the TFTP server directory. Based on the example above, the source text file is in the user directory, and the TFTP server will expect to find the bootfile image in the TFTP server directory. sudo mkimage -A arm -O u-boot -T script -C none -a 0 -e 0 -n "t-Boot Script" -d ~/Documents/bootscr/para1_boot.script /srv/tftp/para1_boot.scr

Reboot the Parallella
Either recycle power or press the reset button. Given the reported problems with USB and HDMI initialisation, most success seems to be after full power recycle.

The output to the USB/UART terminal screen should show initialisation of the board and network, and then show the boot sequence looking for the TFTP server. If the TFTP 'Loading:' line puts out a few 'T T T's, then it's retrying. Give it a few moments but if there's a line of 'T's then check that the TFTP service is actually running, and that the IP addresses and filename of the bootfile are valid.

Once the small (~1kb) bootfile has been loaded and is executed by U-Boot, further TFTP requests will be seen as the image files are downloaded before U-Boot hands over to the kernel.