TFTP Boot and NFS Root Filesystems

= How To Set Up TFTP Boot And NFS Root Filesystems On Parallella = This Guide describes how to set up the Parallella U-Boot to network boot using TFTP.

Also, it describes how to set up mounting a root filesystem (rootfs) via NFS.

See the Manual for Das U-Boot

The following is based on the original blog post by Scott Johnson describing how to set up a Parallella cluster to use TFTP/NFS, but has been tweaked - mainly to minimise the need to edit/update the U-Boot variables via the Serial/UART cable, and provide more flexibility by using PC-based editors and storage.

By editing a bootscript on the TFTP server (and re-booting), a Parallella can be re-configured to a different boot mode (headless,HDMI) and/or different root filesystem (Ubuntu, Debian, nano) - all without having to 'burn' another Parallella SD card (perhaps ever..?).

Tweak the example below to suit your needs.

Requirements
Linux PC, Parallella board(s), USB to Serial (TTL/UART) adapter, DHCP server.

Summary
This Guide describes: IMPORTANT NOTE:
 * An IP addressing and naming scheme to support different boot modes (headless, HDMI) and NFS root filesystems.
 * Setting up TFTP and a directory layout for TFTP bootscripts and boot mode image files.
 * Changing the way the boot arguments are passed to the kernel, i.e. in the U-Boot 'bootargs' rather than devicetree.
 * Setting up NFS and a directory layout for NFS root filesystems, allowing a choice of root filesystems for each Parallella.
 * Changing the U-Boot boot command to fetch a small script file using TFTP.
 * An example bootscript for TFTP boot and NFS root filesystem.
 * Use of mkimage to convert the bootscript text file into a U-Boot script image file.
 * A breakdown of the bootscript to show how it might be altered to suit different environments.
 * {Sept 2014} There is a problem in Parallella U-Boot w.r.t. the Ethernet interface after any network activity in U-Boot (e.g. TFTP). (See the post on the Parallella forum post by tajama). The ethernet interface is left in an unuseable state. A workaround is included in the script example below; but before attempting to apply it - you MUST check the U-Boot version. (Revisit)
 * {Sept 2014} the Parallella U-Boot is being updated which just might affect the script below. (Revisit).

= Prepare File Service Environments =

Set up IP Addresses
Allocate IP addresses for the Parallella(s) and the TFTP/NFS servers in your DHCP server based on MAC address. The Parallella MAC address is printed on the board, and is also a U-Boot variable (see later).

Refer to the documentation of your DHCP server, or discuss with your network admin.

The IP addresses are used as follows: This is an example IP address allocation and naming convention (your network may vary): 192.168.1.201	para1		// IP address of Parallella "para" 192.168.1.202	para2 ... 192.168.1.208  para8     	// ... I wish 192.168.1.50	TFTP server	// IP address of TFTP server "serverip" 192.168.1.50	NFS server	// IP address of NFS server "nfs_ip" Note: The TFTP and NFS services do not need to be on the same physical server, or even the same operating system. Here, a Debian (Wheezy) PC provides both services.
 * U-Boot tftpboot command requests a bootscript file, and needs the TFTP server IP address, and its own IP address.
 * U-Boot tells Linux to use DHCP.
 * NFS uses IP addresses to control access to the rootfs directories.

Prepare the TFTP Service
Get the required software if not already set up. On Debian: apt-get install tftpd-hpa The TFTP service does not start automatically. Start the tftpd-hpa service automatically by adding a command to /etc/rc.local (just before the exit 0 line). sudo nano /etc/rc.local (Edit and save) service tftpd-hpa start To control the TFTP service use: sudo service tftpd-hpa restart	// or 'stop', or 'start' as needed.

Set up the TFTP Directories
By default, the tftpd-hpa service looks for requested files under /srv/tftp/. Each Parallella has its own bootscript, e.g. para1. This is a U-Boot script image file, created using mkimage from a text file (see below). The bootscript points U-Boot to a boot mode directory. The bootscript also provides U-Boot with the Linux boot parameters to mount the NFS root filesystem, and finally to boot the kernel.

Each boot mode (headless, HDMI, etc) has a separate directory containing the equivalent of the BOOT partition on a SD card, containing the kernel, parallella.bit.bin and devicetree image files.

Different sets of image files can be held in different directories, e.g. hdmi_7010_new_kernel/, test_usb_fix/.

Note: that is one directory per boot mode - not one per Parallella.

Create a boot mode directory: sudo mkdir -p /srv/tftp/ 		// replace with boot mode, e.g. hdmi_7010

Prepare the Image Files
Follow the instructions for creating the SD card (that is, download, extract and rename the kernel uImage, parallella.bit.bin and devicetree.dtb files), but instead of 'burning' to a SD card simply copy the image files to the boot mode directory (as root).

The TFTP service directory layout (for this example) will look something like: /svr/tftp/				// TFTP service directory hdmi_7010/			// boot mode = Zynq 7010 with HDMI support paralella.bit.bin devicetree.dtb uImage headless_7010/			// boot mode = Zynq 7010 Parallella configured as headless paralella.bit.bin devicetree.dtb uImage hdmi_7010_new_kernel/		// boot mode = temp testing variation paralella.bit.bin devicetree.dtb uImage para1			       // bootscript for para1 para2			       // bootscript for para2 ...	para8			       // bootscript for para8, etc Note: The image files follow the same naming convention as for SD card booting.

Remove Bootargs from Devicetree
The devicetree as supplied by Adapteva passes the boot arguments to support booting from the SD card. Instead, the boot arguments are passed to the kernel by U-Boot, as defined in the bootscript.

Install the devicetree compiler if not already installed. On Debian: sudo apt-get install device-tree-compiler Extract/edit/rebuild the devicetree.dtb file to remove the parameters from the chosen {} branch. Repeat for all supplied devicetree.dtb files in each boot mode directory.

{Check.. do we need the linux console line in chosen ..? Revisit} cd /srv/tftp/hdmi_7010 sudo dtc -I dtb -o dev.dts -O dts devicetree.dtb sudo nano dev.dts Look for chosen { branch, remove the arguments, and save. chosen { } Recompile the devicetree.dtb and delete the dev.dts text file. sudo dtc -I dts -O dtb -o devicetree.dtb dev.dts sudo rm dev.dts

Prepare the NFS Service
Get the required software if not already set up. On Debian: apt-get install nfs-kernel-server The NFS service starts automatically. To control NFS services use: sudo service nfs-kernel-server restart		// or 'stop', or 'start' as needed.

Set up the NFS Directories
Each Parallella has one or more NFS directory containing a root filesystem. Each Parallella may have alternative root filesystems (e.g. Ubuntu, Linaro Nano) - just like preparing and swapping alternative SD cards.

Create a top directory /srv/nfs, and create each Parallella NFS directory under that, for example:

sudo mkdir -p /srv/nfs sudo mkdir -p /srv/nfs/para1_ubuntu sudo mkdir -p /srv/nfs/para1_nano Here is an example layout of NFS filesystem directories showing separate branches for each Parallella, after adding root filesystem contents. /srv/nfs/ para1_ubuntu/		// rootfs for para1, based on, say, Parallella Ubuntu bin/ boot/ etc/ ...	para1_deb/		// rootfs for para1, with 9600's Debian and ESDK bin/ boot/ etc/ ...	para1_nano/		// rootfs for para1 based on linaro_nano bin/ boot/ etc/ ...		para2_ubuntu/		// rootfs for para2, based on, say, Parallella Ubuntu bin/ boot/ etc/ ...       ...	para8_ubuntu/	       	// rootfs for para8, based on, say, Parallella Ubuntu ...	      		// etc, etc

Update NFS Exports File
The NFS server requires /etc/exports to be configured correctly, to control access to each NFS filesystem directory to specific hosts.

For each Paralella root filesystem that is created, add the export control line to /etc/exports, adjusting (as you should for all examples herein) for your local IP address and directory naming scheme. The following is an example based on the layout described above. sudo nano /etc/exports (Edit and save) /srv/nfs/para1_ubuntu 192.168.1.201(rw,sync,no_root_squash,no_subtree_check) /srv/nfs/para1_deb 192.168.1.201(rw,sync,no_root_squash,no_subtree_check) /srv/nfs/para1_nano 192.168.1.201(rw,sync,no_root_squash,no_subtree_check) /srv/nfs/para2_ubuntu 192.168.1.202(rw,sync,no_root_squash,no_subtree_check) /srv/nfs/para8_ubuntu 192.168.1.208(rw,sync,no_root_squash,no_subtree_check) Note: The bootscript ensures that only one of the optional filesystems will be accessed/mounted by a Parallella on a re-boot.

Export the NFS Directories
Ensure that these directories are made visible by the NFS service. Either: sudo exportfs -a or sudo service nfs-kernel-server restart

Set up Parallella NFS Root Filesystems
Instead of 'burning' a filesystem image to a partition on the SD card, place the payload root filesystem in the NFS directory, e.g. /srv/nfs/para1_ubuntu/.

Each source of root filesystem may involve different preparation to enable the NFS access to work smoothly.

See the examples in the How-to for Extracting Parallella NFS rootfs

= Set Up the Parallella to TFTP Boot = Power off the Parallella and remove the ethernet, USB peripheral and HDMI cables and SD card.

Connect a USB serial cable to the 3-pin UART header on the Parallella. (Example device http://www.adafruit.com/products/954 from Adafruit).

On the PC, use a terminal program (e.g. screen, minicom) to connect to the UART Serial device. On minicom, set hardware flow control to off, and point the logfile to, say, ''~/minicomlog' for diagnostics.

Power up the Parallella.

After the boot failure messages, you should get the zynq-uboot> prompt in the terminal window.

Display a list of the current state of the U-Boot environment variables using the printenv command.

Check the value of the ethaddr=xx:xx:xx:xx:xx:xx variable; this should be the same as the label on the board, and the MAC address you are using for this Parallella in your DHCP server.

Enter the following commands at the zynq-uboot> prompt in order to: setenv hostname para1 setenv ipaddr 192.168.1.201 setenv serverip 192.168.1.50 setenv t-addr 0x100000 setenv t-boot 'tftpboot ${t-addr} ${hostname} ; source ${t-addr}' setenv bootcmd 'run t-boot' Alternatively, if preferred as a fall-back, replace the U-Boot bootcmd in the above sequence so that the boot sequence tries to boot from the SD card first. If no SD card is present, then run the TFTP boot sequence. setenv bootcmd 'run modeboot ; run tboot' Check that the variables have been set correctly and, if OK, save to flash memory. printenv saveenv Power off.
 * set the hostname variable which will be used as the bootscript name. Change for each Parallella.
 * set the IP address of the Parallella and the TFTP server. Change for each Parallella.
 * create a variable t-boot which contains the commands to fetch and then execute the bootscript.
 * change the U-Boot bootcmd so that it will run the t-boot TFTP boot sequence.

Reconnect the ethernet cable and any USB peripherals or HDMI you intend to use on the Parallella.

Ready to TFTP boot.

Notes:
 * It's worthwhile checking (again) after a power recycle that the U-Boot variables have been set correctly
 * To correct/change a variable, use the setenv command with new/correct value.
 * To remove a variable e.g. 'badvarible', use the setenv command with no value, e.g. setenv badvarible
 * U-Boot autocompletes commands so print, set, save will also work.

= Write a Parallella Bootscript = On the PC host, the bootscript image file is created and edited as a text file, and compiled into a U-Boot executable image using mkimage.

Create a working directory to hold the bootscript text file(s). mkdir ~/Documents/bootscr Create a text file and copy/paste the text below as a template. The following is an example bootscript. This loads the headless image files.

cd ~/Documents/bootscr nano para1.txt (Edit and save) setenv tftp_dir headless_7010 setenv nfs_rfs /srv/tftp/para1_ubuntu

setenv bootargs ip=dhcp setenv bootargs ${bootargs} console=ttyPS0,115200

setenv nfs_ip 192.168.1.50 setenv bootargs ${bootargs} root=/dev/nfs rw nfsroot=${nfs_ip}:${nfs_rfs} setenv bootargs ${bootargs} nfsrootdebug earlyprintk

setenv fpga_image parallella.bit.bin setenv fpga_addr 0x4000000 setenv fpga_size 0x3dbafc setenv fpga_tftp 'tftpboot ${fpga_addr} ${tftp_dir}/${fpga_image}' setenv fpga_load 'fpga load 0 ${fpga_addr} ${fpga_size}'

setenv fdt_image devicetree.dtb setenv fdt_addr 0x2A00000 setenv fdt_tftp 'tftpboot ${fdt_addr} ${tftp_dir}/${fdt_image}'

setenv kernel_image uImage setenv kernel_addr 0x3000000 setenv kernel_tftp 'tftpboot ${kernel_addr} ${tftp_dir}/${kernel_image}'

setenv phy_rst 'mw.w f8000008 df0d ; mw.w f8000140 00100801 ; mw.w f8000004 767b' setenv rset_phy 'run phy_rst'

setenv boot_now 'bootm ${kernel_addr} - ${fdt_addr}'

run fpga_tftp fpga_load kernel_tftp fdt_tftp rset_phy boot_now Copy the file to create a bootscript for each Parallella, changing the filename.

Edit the values of the tftp_dir and nfs_rfs variables in each file as required.

The directory structure on the Debian PC will then look something like this: /home/{username}/Documents/bootscr/ para1.txt para2.txt ...	para8.txt

Make the Bootscript Image
Install the mkimage package if not already installed. On Debian: sudo apt-get install u-boot-tools Convert each bootscript text file into a loadable image by mkimage, and place the image in the TFTP server directory. For example: cd ~/Documents/bootscr sudo mkimage -A arm -O u-boot -T script -C none -a 0 -e 0 -n "t-Boot Script" -d para1.txt /srv/tftp/para1

Reboot the Parallella
Reboot by power off, wait a few seconds, power on.

The output to the UART Serial terminal screen shows initialisation of the board and network, and then the boot sequence looking for the TFTP server. If the TFTP 'Loading:' line puts out a few 'T T T's, then it's retrying. Give it a few moments but if there's a line of 'T's then check that the TFTP service is actually running, and that the IP addresses and filename of the bootscript are valid.

Once the small (~1kb) bootscript has been loaded and is executed by U-Boot, further TFTP requests will be seen as the image files are downloaded before U-Boot hands over to the kernel.

Breakdown of the Bootscript
This section takes the bootscript example and breaks it down to illustrate what may be changed to support your local environment.

setenv tftp_dir headless_7010	  	// or, hdmi_7010, etc setenv nfs_rootfs /srv/nfs/para1_ubuntu	// or, para1_deb, etc setenv bootargs ip=dhcp setenv bootargs ${bootargs} console=ttyPS0,115200 setenv nfs_ip 192.168.1.50 setenv bootargs ${bootargs} root=/dev/nfs rw nfsroot=${nfs_ip}:${nfs_rfs} setenv bootargs ${bootargs} nfsrootdebug earlyprintk setenv fpga_image parallella.bit.bin setenv fpga_addr 0x4000000 setenv fpga_size 0x3dbafc setenv fpga_tftp 'tftpboot ${fpga_addr} ${tftp_dir}/${fpga_image}' setenv fpga_load 'fpga load 0 ${fpga_addr} ${fpga_size}' setenv fdt_image devicetree.dtb setenv fdt_addr 0x2A00000 setenv fdt_tftp 'tftpboot ${fdt_addr} ${tftp_dir}/${fdt_image}' setenv kernel_image uImage setenv kernel_addr 0x3000000 setenv kernel_tftp 'tftpboot ${kernel_addr} ${tftp_dir}/${kernel_image}' U-Boot 2012.10-00003-g792c31c (Jan 03 2014 - 12:24:08) setenv phy_rst 'mw.w f8000008 df0d ; mw.w f8000140 00100801 ; mw.w f8000004 767b' setenv rset_phy 'run phy_rst' setenv boot_now 'bootm ${kernel_addr} - ${fdt_addr}' run fpga_tftp fpga_load kernel_tftp fdt_tftp rset_phy boot_now
 * Set the bootscript to point to the required boot mode (i.e. TFTP sub-directory) and the NFS filesystem directory chosen for this Paralella. By editing these first two lines, a Parallella can be re-booted using a different FPGA, kernel and/or a different NFS root filesystem. Note that the the NFS filepath is the same as the line in NFS exports.
 * Set the initial bootargs string to tell the kernel to use DHCP when loaded.
 * Add the console argument to the bootargs string. If the console is not required, delete this line.
 * Add the NFS boot parameters to the bootargs string. Note the NFS server IP variable is not the same as the TFTP serverip variable, although they may have the same value.
 * Add additional kernel boot diagnostic flags to the bootargs string. If not required, delete this line.
 * Set up the variables to download the fpga image, and to load the image into the fpga.
 * Set up the variables to download the devicetree image.
 * Set up the variables to dowlnload the kernel image.
 * Patch U-Boot network handling. This is a workround for the problem identified by tajama, and referred to above. It MAY apply to your version but YOU MUST CHECK!! This U-Boot patch applies to the following version, which is output initially by U-Boot:
 * Set up the command to boot into the kernel, passing the kernel and devicetree image addresses.
 * Finally, run the sequence of tftp requests, and boot. If any of the command sequence fails, then the 'run' command will halt. Note there are no ';' separators in the command parameters.