RPi iSCSI Initiator

Back to RPi Guides.

Raspberry Pi iSCSI Initiator Setup

=Introduction=

This guide is split into two parts, part 1 deals with rebuilding the standard Raspbian/Debian image to include iscsi support so that the pi can access an iscsi target (and function as a target itself if needed, but this guide deals with setting up as an initiator), part 2 then extends on this to enable booting from an iscsi root. For the sake of simplicity we will be performing on-device kernel recompiles, these take 4-6 hours, you can simply perform a cross-compile elsewhere to speed this up if you prefer.

=Assumptions/Pre-requisites=

This guide assumes a number of things:
 * At least a basic knowledge of iscsi, and that you already have a suitable target setup on your SAN/NAS/wherever.
 * You are working as the root user - If you're using a non-root user, most commands will need to be prefixed with a sudo.
 * You have expanded your SD card to increase the amount of space in / over the default 1.6GB/1.9GB, to at least 2GB - More recent images can do this automatically using raspi-config, for older images or a manual process see Here.

=Part 1 - iSCSI Support=

First up, we need to get the iscsi initiator service installed, along with git, gcc, make, ncurses-devel and python, which are needed for an on-device kernel recompile:

apt-get install open-iscsi git gcc make python libncurses5-dev bc

Lets create a folder in which to keep the various content we'll need for the kernel rebuild:

mkdir /root/raspberrypi cd /root/raspberrypi

We now need to fetch the kernel sources to build from, this will take a while:

git clone https://github.com/raspberrypi/linux.git --depth 1

Now take a copy of your current running kernel configuration, which we can then alter:

cd /root/raspberrypi/linux zcat /proc/config.gz > .config

Now let's load the system's UI-based kernel configuration editor (these changes can be adjusted manually, but the UI is simple):

make menuconfig

Once the UI has loaded, navigate to the "Device Drivers" section (enter to navigate between pages, space to change an option), then "SCSI device support", then "SCSI Transports". Change the "iSCSI Transport Attributes" option twice (so that it shows a *) to build it into the kernel. Go back a page, hit space for "SCSI low-level drivers" to enable this option, then enter to descend into it, then space twice for a * on "iSCSI Initiator over TCP/IP".

Once these options are enabled, exit the UI and save changes. To ensure these are set, run:

grep ISCSI .config

This should return the following:

CONFIG_SCSI_ISCSI_ATTRS=y CONFIG_ISCSI_TCP=y
 * 1) CONFIG_ISCSI_BOOT_SYSFS is not set

With this in place, compile your new kernel:

- for mono core CPU : make

- for 4 core CPU (Pi 2 B) : make -j5

This process will take in the region of 4-6 hours to complete.

Once done, we need to generate an image of this kernel for the Pi to boot from, then replace your existing kernel:

First up, we need the mkimage python tool:

cd /root/raspberrypi git clone https://github.com/raspberrypi/tools

Now build the image:

cd /root/raspberrypi/tools/mkimage python imagetool-uncompressed.py /root/raspberrypi/linux/arch/arm/boot/Image

Now create a backup of your existing kernel, then replace the running kernel with the newly-generated one:

cp /boot/kernel.img /boot/kernel.img-backup cp kernel.img /boot/kernel.img

With this done, reboot your pi into your new iscsi-enabled kernel:

reboot

If you have any problems with booting at this point, you can always revert to your old kernel by mounting your SD card into a card reader on another system, then revert kernel.img-backup to kernel.img.

Assuming all has gone well, you can now use iscsiadm to discover then login to your iscsi target (though hopefully youll already be familiar with configuring an initiator to access a target), for example:

iscsiadm --mode discovery --type sendtargets --portal XXX.XXX.XXX.XXX

where the IP address in the above should be the address of your iscsi target, then once it has been discovered, login to the target with:

iscsiadm --mode node --loginall=all

=Part 2 - iSCSI root=

This section will cover moving your Debian install's root to an iscsi volume, then generating a new kernel with included initramfs that will load open-iscsi to be able to mount your iscsi volume as root, essentially performing a network boot of your pi using iSCSI (by the end of this the pi's SD card will still be needed for the initial kernel/initramfs load, but after that the card will be removable). Part 2 assumes you have already followed Part 1, both to end up with an iscsi-capable kernel and have the relevant tools installed to be able to rebuild your kernel.

First up, we need to prepare the iscsi volume with a working install of Debian.

I will assume you already have an iscsi session connected to a target (and that the target volume is empty and unpartitioned), this volume will most likely be /dev/sda unless you have other disks in the system. *NOTE* we will be making destructive changes to your target volume (partitioning and formatting it), all of the below assumes the use of /dev/sda, do check this on your pi first if there's any risk of you having other devices mounted.

First off, partition your iSCSI target:

fdisk -uc /dev/sda

n to add a new partition, p to create a primary partition, 1 to create partition 1, accept first and last sector defaults to use the entire volume. w to write the new partition.

Now write an ext4 filesystem to the new partition created:

mkfs.ext4 /dev/sda1

Now create a mountpoint for the volume, we'll use /mnt/iscsi:

mkdir /mnt/iscsi

Mount the volume:

mount -t ext4 /dev/sda1 /mnt/iscsi

Once the partition is mounted, we need to copy our existing debian install to this. There are a number of fancy ways of doing so but a simple recursive copy of the relevant folders works fine, which we'll use for the sake of simplicity:

cd / cp -a bin/ Desktop/ dev/ etc/ home/ lib/ media/ opt/ root/ sbin/ sd/ selinux/ srv/ tmp/ usr/ var/ /mnt/iscsi

If you have any top-level folders you've written to your pi which youll want on your iscsi-booted system these should also be copied in the above.

mkdir /mnt/iscsi/proc mkdir /mnt/iscsi/sys mkdir -p /mnt/iscsi/run/lock mkdir /mnt/iscsi/run/shm mkdir /mnt/iscsi/boot

From this point, we should now have a working install in place on our iscsi volume - In order to test this we can swap our root to /mnt/iscsi (the iscsi volume's mount point):

mkdir /mnt/iscsi/oldroot pivot_root /mnt/iscsi/ /mnt/iscsi/oldroot/ mount -t proc none /proc

This has now swapped your root to the iscsi volume, assuming you still appear to have a working system (basic system commands etc still work) you're all good so far. If you're feeling brave, you can test this further by pulling the SD card from your pi, and it should continue functioning as normal.

We now need to disable a couple of services while running on the iscsi root (as these will cause boot problems with the initramfs we setup later:

update-rc.d -f open-iscsi remove update-rc.d -f ifplugd remove

While we're rooted to the iscsi volume we can also alter its fstab, so open /etc/fstab, remove any current mount point for / (most likely /dev/mmcblk0p2) and add a new one as:

/dev/sda1 /      ext4    defaults,noatime,nodiratime     0 0

We now need to get back to our standard SD install, depending on what you've done above the most straightforward way is to simply power down the device, put the SD card back in (if you removed it) and start your pi back up again.

Now that we have a working install on the iscsi target, we need to configure the kernel to be able to boot directly from this.

First up, getting a network connection. We'll assume a wired connection, wireless is perfectly possible but will depend on whether you're using an adaptor the kernel natively supports or needs a module, which can get complicated quickly. We need to ensure a valid network address is being set early during boot, to do this edit /boot/cmdline.txt with your favourite editor.

A default cmdline.txt will look like this:

dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 root=/dev/mmcblk0p2 rootfstype=ext4 rootwait

If you are assigning IPs via DHCP, change it to:

dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 ip=dhcp root=/dev/mmcblk0p2 rootfstype=ext4 rootwait

If you assign IPs manually, simply enter the IP for the device to use as:

dwc_otg.lpm_enable=0 console=ttyAMA0,115200 kgdboc=ttyAMA0,115200 console=tty1 ip=XX.XX.XX.XX root=/dev/mmcblk0p2 rootfstype=ext4 rootwait

Open/create /etc/iscsi/iscsi.initramfs - This file needs to contain initiator/target details for your iscsi volume, in the following format:

ISCSI_INITIATOR=initiatorname ISCSI_TARGET_NAME=targetname ISCSI_TARGET_IP=target_ip

targetname and target_ip need to be entered per your iscsi volume's correct target name and address, initiatorname can be any validly formatted name (eg. iqn.2012-06.org.raspberrypi:01)

We now need to install initramfs-tools:

apt-get install initramfs-tools

Now to build the initramfs:

cd /boot update-initramfs -c -k $(uname -r)

If you receive a message like "missing /lib/modules/...", check for updated firmware. We now have a compressed initramfs image (which will be named initrd.img-3.2.27+ on the current Debian image), it's easier to build this uncompressed however, so now we need to extract the image to a /boot subfolder:

mkdir initramfs cd initramfs

gunzip -c -9 /boot/initrd.img-3.2.27+ \ | cpio -i -d -H newc --no-absolute-filenames

With 3.12.26+ kernel, you will also experience symlink issues with sbin/modprobe, sbin/rmmod and /lib/ld-linux-armhf.so.3 uncompressing the produced initramfs. A simple workaround seems to unlink and copy files !

With that done it's now back to our kernel sources to modify the kernel's configuration to include the initramfs:

cd /root/raspberrypi/linux make menuconfig

In the menuconfig UI, navigate to "General Setup" then enable "Initial RAM filesystem and RAM disk (initramfs/initrd) support". Then edit "Initramfs source file(s)" and set its path to /boot/initramfs/, okay the path, then select "Built-in initramfs compression mode" and set this to gzip. Save and exit menuconfig.

Now it's time to make our new kernel with included iscsi-enabled initramfs, this will however be much quicker than the full recompile in Part 1 (this should take no more than ~10 minutes:

make

As earlier, we now need to generate a pi-usable kernel image from this and copy it to /boot, taking a backup of the current kernel in the process:

cd /root/raspberrypi/tools/mkimage/ python imagetool-uncompressed.py /root/raspberrypi/linux/arch/arm/boot/Image

cp /boot/kernel.img /boot/kernel.img-backup cp kernel.img /boot/kernel.img

Now edit /boot/cmdline.txt and alter

root=/dev/mmcblk0p2 to root=/dev/sda1

Save the file, and you should now be ready to reboot to your configured iSCSI volume! If you have a screen plugged in for the boot process and you keep your eyes open you should see your pi mount sda1 as its root device rather than mmcblk0p2. Once this has happened, you should be able to disconnect the device's SD card and it'll continue functioning from the network.

Should you have any problems with booting from the network using your initramds-enabled kernel, you can revert to local booting using your old kernel by loading your SD card up into a host machine, copying kernel.img-backup back to kernel.img, and modifying cmdline.txt to set root= back to /dev/mmcblk0p2.