TFTP Boot and NFS Root Filesystems

= How To Set Up TFTP Boot And NFS Root Filesystems On Parallella =

Requirements
Linux PC, Parallella board(s), USB to TTL/UART adapter, DHCP server

Summary
This is an approach to booting the Parallella using TFTP, and mounting the root filesystem via NFS.

It should be useful to the owner of a Parallella cluster, or even the owner of a single Parallella who needs to quickly and easily re-purpose the Parallella, or experiment with different profiles - all without having to 'burn' another micro-SD card, or keep a small stack of easilly lost thumbnail-sized micro-SD cards to hand.

Although it requires a bit of preparation, the approach involves minimal changes to each Parallella U-Boot environment using the USB to UART cable, and then all the loading and re-configuration is controlled on the TFTP/NFS host - in this case, a Debian PC.

This Guide describes:
 * An addressing and naming scheme to support the choice of bootmode (headless, HDMI) and NFS root filesystems.
 * A directory layout for TFTP bootfiles and bootmode image files
 * Changing the way the boot command arguments are passed to the kernel, i.e. in a U-Boot script rather than devicetree
 * A directory layout for NFS root filesystems, allowing a choice of filesystems for each Parallella
 * The minimum changes to the U-Boot variables on each board to trigger a network TFTP boot instead of micro-SD card boot
 * An example script for TFTP boot and NFS root filesystems.
 * Use of mkimage to convert the bootfile text file into a U-Boot script image file.

There are many examples and reference material to illustrate the flexibility of U-Boot, and TFTP booting for ARM-based and embedded systems - you only need to search for them. TFTP booting appears to be normal for small embedded systems.

The following is based on the original blog post by Scott Johnson on setting up a Parallella cluster to use TFTP/NFS, but along the line it has been tweaked - mainly to minimise the need to edit/update the U-Boot variables via the UART cable, and make more use of the PC-based editors, etc.

Tweak the example below to suit your local needs ...

IMPORTANT CAVEATS:
 * There appears to be a problem in Parallella U-Boot w.r.t. the Ethernet interface after any network activity in U-Boot (e.g. TFTP). (See the post on the Parallella forum post by tajama). In effect, the network interface is left in an unuseable state. A workround is included in the script example below; but before attempting to apply it - you MUST check the U-Boot version. (To be revisited)
 * As of early Sept 2014, the Parallella U-Boot is to being updated, which may affect the script below. (To be revisited).

IP Addressing and Naming
The approach relies on consistent IP address allocation and naming:
 * On the Parallella, the U-Boot TFTP boot command uses the IP address of the TFTP server, its own IP address , and a unique hostname.
 * On booting, U-Boot will tell Linux to use DHCP.
 * The NFS service uses the IP address of the Parallella to control access to the exported NFS directories.

The IP addresses for the Parallella(s) and the TFTP/NFS servers are pre-allocated in the DHCP server - based on MAC address.

The MAC address of each Parallella is printed on a label on the board, but is also held as a U-Boot variable (see later).

Refer to the documentation of your DHCP server, or discuss with your network admin. In my case, the DHCP server runs in the ISP-supplied home router.

Note: the TFTP and NFS services do not need to be on the same physical server, or even the same operating system. Here, a Debian (Wheezy) PC provides both services.

The Parallella naming convention and IP allocation (for this example) is as follows: 192.168.1.201	para1 192.168.1.202	para2 ... ... 192.168.1.208  para8     	// .....I wish.

192.168.1.50	TFTP server	// IP address of PC TFTP server 192.168.1.50	NFS server	// in this case, both services are on the same Debian PC

Set up the TFTP and NFS services
On Debian, get the required packages, if not already set up. apt-get install tftpd-hpa nfs-kernel-server By default, the NFS service is configured to start automatically. To start the tftpd-hpa service, add a command to /etc/rc.local (just before the exit 0 line): sudo nano /etc/rc.local (edit and save) service tftpd-hpa start To control the tftp and NFS services use:

sudo service tftpd-hpa restart  	// or 'stop', or 'start' as needed. sudo service nfs-kernel-server restart	// or 'stop', or 'start' as needed. sudo service tftpd-hpa start  // or 'stop', or 'restart' as needed.

The Bootfile Directories
By default, the tftpd_hpa service looks for requested files in /srv/tftp/. The directory is owned by root, so use sudo for file commands (or run as root!). All the files are loadable images of some kind.

The directory layout (for this example) looks like: /svr/tftp/				// TFTP service directory hdmi_7010/			// bootmode: 7010 with HDMI paralella.bit.bin devicetree.dtb uImage headless_7010/			// bootmode: 7010 headless paralella.bit.bin devicetree.dtb uImage hdmi_7010_new_kernel/		// example testing variation paralella.bit.bin devicetree.dtb uImage para1_boot.scr			// initial bootfile for para1 para2_boot.scr			// initial bootfile for para2 ...	para8_boot.scr			// etc, etc

Each Parallella has its own bootfile image, e.g. 'para1_boot.scr'. This file is a U-Boot script, which is created using mkimage from a text file, e.g 'para1_boot.script'. The bootfile points U-Boot to the bootmode directory holding the kernel, bit.bin and devicetree images. The bootfile also tells U-Boot to set up the Linux boot parameters to use a particular NFS root filesystem, and finally to boot the kernel. (see below).

Each bootmode (e.g. headless_7010 or HDMI_7010) has a separate directory under /srv/tftp. Follow the instructions for creating a micro-SD card, that is, download, extract and rename the kernel uImage, parallella.bit.bin and devicetree.dtb files. Instead of 'burning' to a micro-SD card, simply copy the files to the bootmode directory (as root or using sudo).

Different types or generations of image files can be held in different bootmode directories, e.g. '/hdmi_7010_new_kernel', '/test_usb_fix'. A Parallella can be re-booted to call up a new boot profile, by editing its bootfile script to point at a different bootmode directory (see below).

Edit the Devicetree
In this approach, the boot command parameters are created in the bootfile script and passed to the kernel by U-Boot rather than in the devicetree. No need for a specific version for each Parallella. If not already installed: sudo apt-get install device-tree-compiler

Edit the supplied devicetree.dtb in each bootmode directory to remove the boot parameters. Produce a .dts text file from your existing devicetree.dtb file. cd /srv/tftp/hdmi_7010     // for example

sudo dtc -I dtb -o dev.dts -O dts devicetree.dtb

Edit the devicetree text file to remove the micro-SD card mmc boot parameters from the 'chosen' branch. sudo nano dev.dts

(Edit and save.) chosen { linux,stdout-path = "/amba@0/uart@E0001000"; } Recompile the devicetree.dtb in the bootmode directory, and delete the dev.dts file. sudo dtc -I dts -O dtb -o devicetree.dtb dev.dts sudo rm dev.dts

Prepare the NFS Root Filesystems
Each Parallella is provided with a separate NFS directory containing its root filesystem. The NFS directories are placed under '/srv/nfs' in this example. Just as with preparing and swapping different micro-SD cards there can be many versions of root filestore, (e.g. standard Ubuntu, development and experimentation, alternative builds). A Parallella can be re-purposed by rebooting after editing its bootfile. Here is an example layout of the NFS directories showing separate filesystems for each Parallella, and variations. /srv/nfs/ para1/		// root filesystem for para1, based on, say, standard Parallella Ubuntu bin boot etc ...	// etc, etc para1_deb/	// root filesystem for para1, with 9600's Debian and ESDK ...	para1_nano/	// root filesystem for para1 based on linaro_nano para2/		// root filesystem for para2 ...	...	para8/	// root filesystem for para8 ...

Sharing the same root filestore between two or more Parallellas is possible, but there is a need to set up different temp filestore, etc., (not covered here). Since there's loads of space(?) on the server disk(s), duplication is not necessarilly that important. Care needs to be taken to manage version control (but it's quicker and easier to manage/fix than re-burning a stack of micro-SD cards).

Place the payload root filesystem in the target directory e.g. '/srv/nfs/para1/ This can be the default Parallella Ubuntu build, or, say, 9600's Debian images, or follow shodrucky's nano build from here (http://elinux.org/Parallella_Linaro_Nano), with some mods as below. Download the linaro-saucy-nano image. wget http://releases.linaro.org/14.04/ubuntu/saucy-images/nano/linaro-saucy-nano-20140410-652.tar.gz Extract the filesystem image to the NFS directory for the target Parallella. sudo tar --strip-components=1 -C /srv/nfs/para1_nano -xzpf ~/Downloads/linaro-saucy-nano-20140410-652.tar.gz Edit some configuration files as per shodrucky's notes. Important note: To avoid the mistake of editing the host PC's configuration, you are in the correct NFS subdirectory. Note the lack of '/' before etc/ !! cd /srv/nfs/para1_nano		// for example

Prevent installation of unnecessary packages
sudo nano etc/apt/apt.conf.d/00InstallRecommends

(Edit and save.) APT::Install-Recommends "false";

Network configuration
The assumption is that DHCP will be used. Edit the interfaces file. I DONT THINK YOU NEED TO DO THIS BECAUSE WE'VE ALREADY TOLD LINUX TO USE DHCP !! REVISIT THIS. sudo nano /srv/nfs/para1/etc/network/interfaces

(Edit and save.) source-directory /etc/network/interfaces.d

auto lo   iface lo inet loopback

auto eth0

# If you prefer DHCP, comment out the above 5 lines, uncomment the below. iface eth0 inet dhcp up sleep 3; mii-tool -F 1000baseT-FD sudo nano etc/resolv.conf  //// ???? needed (Edit and save.) nameserver 192.168.1.1 Change directory out of the NFS subdirectory cd /		// or anywhere else !!

Set Up The NFS Filesystem Exports
The NFS server will require /etc/exports to be configured correctly, and export each filessystem directory to certain hosts only. On the Debian PC host, edit /etc/exports. sudo nano /etc/exports For each Paralella and for root each filesystem add the following line to /etc/exports, adjusting (as you should for all examples herein) for your local IP addressing scheme. Note that the bootfile ensures that only one of the optional filesystems can be accessed by a Parallella on a re-boot. This is based on the example layout shown above. /srv/nfs/para1 192.168.1.201(rw,sync,no_root_squash,no_subtree_check) /srv/nfs/para1_dev 192.168.1.201(rw,sync,no_root_squash,no_subtree_check) /srv/nfs/para1_nano 192.168.1.201(rw,sync,no_root_squash,no_subtree_check) /srv/nfs/para2 192.168.1.202(rw,sync,no_root_squash,no_subtree_check) ... ... /srv/nfs/para8 192.168.1.208(rw,sync,no_root_squash,no_subtree_check)

Ensure that these directories are exported either by using the following command, or by restarting the NFS service. sudo exportfs -a or sudo service nfs-kernel-server restart

Change The Parallella to Boot Using TFTP
Remove any Ethernet, USB and HDMI cables and micro-SD card. Connect a USB serial cable to the 3-pin header on the Parallella. (Example device http://www.adafruit.com/products/954 from Adafruit). Use a terminal program (e.g. screen,minicom) to connect to the USB device. On minicom, set hardware flow control to off, and point the logfile to, say, ~/minicomlog for diagnostics. Power up the Parallella. After the boot failure messages, you should get the zynq-uboot> prompt in the terminal window. This command displays a sorted list of the current state of the U-Boot environment variables. printenv Note the value of the 'ethaddr=xx:xx:xx:xx:xx:xx' variable and check that this is the MAC address you are using for this Parallella in your DHCP server. It should be the same as the label on the board. It cannot be changed at the prompt.

Enter the following commands at the zynq-uboot> prompt. These will: Set the hostname and IP address of this Parallella, and the TFTP server, based on the example above. Set up a variable containing the TFTP boot sequence . This loads the bootfile for this Parallella to memory at  and executes it as a script. The bootfile is described below. Change the U-Boot boot command so that it will run the TFTP boot sequence. setenv hostname para1 setenv ipaddr 192.168.1.201 setenv serverip 192.168.1.50 setenv t-addr 0x100000 set t-boot 'tftpboot ${t-addr} ${hostname}_boot.scr ; source ${t-addr}' EITHER replace the current bootcmd to run the t-boot sequence setenv bootcmd 'run t-boot' OR, as a fall-back, you could replace the U-Boot bootcmd to try to boot from an micro-SD card first; if no micro-SD card is present, then run the TFTP boot sequence. setenv bootcmd 'run modeboot ; run t-boot' Check (!!) that the variables have been set correctly and, if OK, save to flash. printenv   	// Check. Check. saveenv Notes:
 * It's worthwhile checking (again) after a power recycle that the variables have been set correctly
 * To correct/change a variable, use the setenv command with new/correct value.
 * To remove a variable e.g. 'badvarible', use the setenv command with no value, e.g. setenv badvarible
 * U-Boot autocompletes commands so print, set, save will also work.

Reconnect Ethernet cable and any peripherals you intend to use on the Parallella. Ready to TFTP boot.

Example Bootfiles
On the host, bootfiles are created and edited as text files, and compiled into images by mkimage for loading/executing by U-Boot.

Create a directory to hold the working text files. mkdir ~/Documents/bootscr In summary, the directory structure on the Debian PC looks like this example: /home/ /Documents/bootscr/ para1_boot.script para2_boot.script ...	...	para8_boot.script

Breakdown of Bootfile
The following shows an example bootfile, and then a breakdown of the components for explanation.

Create a file and copy/paste the text below. Edit/copy the file to create a bootfile script for each Parallella, adjusting the values of the  and  variables for each bootfile if necessary. nano para1_boot.script   // para2_boot.script, etc

(edit and save) setenv tftp_dir hdmi_7010 setenv nfs_dir para1

setenv nfs_ip 192.168.1.52 setenv nfs_root '/srv/nfs' setenv bootargs root=/dev/nfs rw nfsroot=${nfs_ip}:${nfs_root}/${nfs_dir} setenv bootargs ${bootargs} ip=dhcp console=ttyPS0,115200

setenv fpga_image parallella.bit.bin setenv fpga_addr 0x4000000 setenv fpga_size 0x3dbafc setenv fpga_tftp 'tftpboot ${fpga_addr} ${tftp_dir}/${fpga_image}' setenv fpga_load 'fpga load 0 ${fpga_addr} ${fpga_size}'

setenv fdt_image devicetree.dtb setenv fdt_addr 0x2A00000 setenv fdt_tftp 'tftpboot ${fdt_addr} ${tftp_dir}/${fdt_image}'

setenv kernel_image uImage setenv kernel_addr 0x3000000 setenv kernel_tftp 'tftpboot ${kernel_addr} ${tftp_dir}/${kernel_image}'

setenv phy_rst 'mw.w f8000008 df0d ; mw.w f8000140 00100801 ; mw.w f8000004 767b' setenv rset_phy 'run phy_rst'

setenv boot_now 'bootm ${kernel_addr} - ${fdt_addr}'

run fpga_tftp fpga_load kernel_tftp fdt_tftp rset_phy boot_now

Breakdown of the Bootfile Script
This section takes the bootfile example above and breaks it down to illustrate what may be changed where needed to support your local environment.

Set the bootfile to point to the required bootmode and the NFS directory for this Paralella, for this boot. By editing only these first two lines, a Parallella can boot into a different kernel and/or a different NFS root filesystem. setenv tftp_dir hdmi_7010	// or, headless_7010, etc setenv nfs_dir para1		// or, para1_deb, etc Set the NFS boot parameters to be passed to the kernel by U-Boot in the variable. The NFS server IP is not the same variable as the variable, the latter is used by U-Boot as the TFTP server. Note that the kernel is told to use DHCP when loaded. setenv nfs_ip 192.168.1.52    // your NFS server IP, usually the same for all your Parallellas setenv nfs_root '/srv/nfs'    // change to apply to your NFS directory structure setenv bootargs root=/dev/nfs rw nfsroot=${nfs_ip}:${nfs_root}/${nfs_dir} setenv bootargs ${bootargs} ip=dhcp console=ttyPS0,115200 To add diagnostic output to the console, add the following line to the end of the bootargs block. setenv bootargs ${bootargs} nfsrootdebug earlyprintk Set the TFTP boot variables to request the fpga, devicetree and kernel images. This is equivalent to the sequence of commands that load from the micro SD card. setenv fpga_image parallella.bit.bin setenv fpga_addr 0x4000000 setenv fpga_size 0x3dbafc setenv fpga_tftp 'tftpboot ${fpga_addr} ${tftp_dir}/${fpga_image}' setenv fpga_load 'fpga load 0 ${fpga_addr} ${fpga_size}'

setenv fdt_image devicetree.dtb setenv fdt_addr 0x2A00000 setenv t_fdt 'tftpboot ${fdt_addr} ${tftp_dir}/${fdt_image}'

setenv kernel_image uImage setenv kernel_addr 0x3000000 setenv kernel_tftp 'tftpboot ${kernel_addr} ${tftp_dir}/${kernel_image}' The following is a workround for the problem identified by ....url to forum topic here. The Ethernet interface clock setting is incorrect after any U-Boot network activity, and needs to be set properly. IMPORTANT: this is a patch to U-Boot and applies to the following version: U-Boot 2012.10-00003-g792c31c (Jan 03 2014 - 12:24:08) It MAY apply to your version but YOU MUST CHECK!! To be revisited if/when a new version of Parallella U-Boot build is provided. setenv phy_rst 'mw.w f8000008 df0d ; mw.w f8000140 00100801 ; mw.w f8000004 767b' setenv rset_phy 'run phy_rst' Set up the command to boot into the kernel, passing the kernel and devicetree image addresses. setenv boot_now 'bootm ${kernel_addr} - ${fdt_addr}' Run the sequence of tftp requests, and boot. If any of the commands passed to 'run' fails, then the 'run' command will halt (note - there are no ';' separators). run fpga_tftp fpga_load kernel_tftp fdt_tftp rset_phy boot_now

Make the boot script image
Install the mkimage package if not already installed. On Debian, this is as follows: sudo apt-get install u-boot-tools

After a bootfile scripts has been created/edited, it must be made into a loadable image by mkimage, and the image loaded into the TFTP server directory. Based on the example above, the source text file is in a user directory, and the TFTP server will expect to find the bootfile image in the TFTP server directory. sudo mkimage -A arm -O u-boot -T script -C none -a 0 -e 0 -n "t-Boot Script" -d ~/Documents/bootscr/para1_boot.script /srv/tftp/para1_boot.scr

Reboot the Parallella
Either recycle power or press the reset button. Given the reported problems with USB and HDMI initialisation, most success seems to be after full power recycle.

The output to the USB/UART terminal screen should show initialisation of the board and network, and then show the boot sequence looking for the TFTP server. If the TFTP 'Loading:' line puts out a few 'T T T's, then it's retrying. Give it a few moments but if there's a line of 'T's then check that the TFTP service is actually running, and that the IP addresses and filename of the bootfile are valid. Once the bootfile has been loaded and excuted by U-Boot, further TFTP requests will be seen, before U-Boot hands over to the kernel.