Jetson/L4T BSP development tips

Preparation

First, please download BSP package from internet. It's preferred to download BSP package, instead of Jetpack.

Go to https://developer.nvidia.com/embedded/linux-tegra.

The following packages are necessary:

L4T Driver Package (BSP)

Sample Root Filesystem

Source code for kernel and other components:

L4T Driver Package (BSP) Sources

Toolchain for kernel building:

GCC 7.3.1 for 64 bit BSP and Kernel

Secure package if secure-boot is necessary:

Jetson Platform Fuse Burning and Secure Boot Documentation and Tools

Refer to https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%2520Linux%2520Driver%2520Package%2520Development%2520Guide%2Fquick_start.html

Then the device can be flashed by command line. With this method, user can have more controls for the BSP configuration, like pinmux, kernel/kernel DTB customization, etc.

BSP customization

PINMUX

How to find out the actual pinmux configuration file

Refer to https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%2520Linux%2520Driver%2520Package%2520Development%2520Guide%2Fmb1_platform_config_xavier.html%23wwpID0E0240HA

There are several types of Jetson reference boards which are supported in SDK.

Check the configuration file

For example, when the flash command is run:

 sudo ./flash.sh jetson-xavier mmcblk0p1

Check the configuration file: jetson-xavier.conf → p2822-0000+p2888-0004.conf → PINMUX_CONFIG="tegra19x-mb1-pinmux-p2888-0000-a04-p2822-0000-b01.cfg";

(Note: the value of PINMUX_CONFIG may be overwritten. So the later one should take effect.)

Or check the flash log

Run following command:

sudo ./flash.sh -r --no-flash jetson-xavier mmcblk0p1

And check the log:

...

copying pinmux_config(/home/Work/jetson_customer/32.4.3/Linux_for_Tegra/bootloader/t186ref/BCT/tegra19x-mb1-pinmux-p2888-0000-a04-p2822-0000-b01.cfg)... done.

...

How to update pinmux

There are several ways to customize the device PINMUX. The simple way is to generate PINMUX configuration through pre-defined excel. Another way is to edit the PINMUX configuration file directly, but that may need more knowledge about PINMUX setting for the chip.

Edit the excel and generate the Configuration

Refer to https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%2520Linux%2520Driver%2520Package%2520Development%2520Guide%2Fadaptation_and_bringup_xavier_nx.html%23wwpID0E0WL0HA

Search 'Pinmux Changes'

Also, another good reference: https://elinux.org/Jetson/AGX_Xavier_Update_Pinmux

Still, please make sure the PINMUX configuration file name is correct.

Edit the pinmux configuration file

That's the direct and simple way, assumed the developer is familiar with PINMUX setting.

Download technical reference manual from https://developer.nvidia.com/embedded/downloads#?search=TRM for different platforms. And those documents contain detailed information.

Edit the prod configuration

L4T document shows another way to override the PINMUX:

https://docs.nvidia.com/jetson/l4t/index.html#page/Tegra%2520Linux%2520Driver%2520Package%2520Development%2520Guide%2Fmb1_platform_config_xavier.html%23wwpID0E0A40HA

Same command as 2.1.1.2, and check the prod configuration in following log:

...

copying prod_config(/home/Work/jetson_customer/32.4.3/Linux_for_Tegra/bootloader/t186ref/BCT/tegra19x-mb1-prod-p2888-0000-p2822-0000.cfg)... done.

...

Follow the guide in above link.

How to update device PINMUX

After the pinmux configuration files are changed, re-flash the device:

 sudo ./flash.sh jetson-xavier mmcblk0p1        #flash the whole device

or

 sudo ./flash.sh -k MB1_BCT jetson-xavier mmcblk0p1      #only flash the MB1_BCT

How to verify the new PINUX configuration works

User can read the PINMUX registers to confirm the new configuration works.

For example,

In PINMUX configuration file, there's an entry:

pinmux.0x0c302030 = 0x00000540; # gen2_i2c_scl_pcc7: i2c2, tristate-disable, input-enable, io_high_voltage-disable, lpdr-enable

And in prod configuration file, another entry:

prod.0x0c302030.0x0000100 = 0x00000000; #gen2_i2c_scl_pcc7: LPDR disable

Run a physical memory access tool, like devmem2 in Jetson device:

root@nvidia-desktop:/home/nvidia# devmem2 0x0c302030
/dev/mem opened.
Memory mapped at address 0x7f84359000.
Value at address 0xC302030 (0x7f84359030): 0x440

Linux kernel

Kernel building

There are several resources in internet introducing how to build Jetson Linux kernel from source. But it seems that the NV-provided nvbuild.sh never works in my side. Here's the script I'm using for kernel building:

#!/bin/bash
set -e
export KERNEL_SRC_DIR=${HOME}/Work/jetson_sdk/32.4.3/source/Linux_for_Tegra/source/public/kernel-source/kernel/kernel-4.9
export CROSS_COMPILE=${HOME}/Tools/kernel-toolchain/gcc-linaro-7.3.1-2018.05-x86_64_aarch64-linux-gnu/bin/aarch64-linux-gnu-
export TEGRA_KERNEL_OUT=${HOME}/Work/jetson_sdk/32.4.3/source/Linux_for_Tegra/source/public/kernel-build
export ARCH=arm64
make -C $KERNEL_SRC_DIR ARCH=arm64 LOCALVERSION="-tegra" O=$TEGRA_KERNEL_OUT tegra_defconfig
make -C $KERNEL_SRC_DIR ARCH=arm64 LOCALVERSION="-tegra" O=$TEGRA_KERNEL_OUT -j8
make -C $KERNEL_SRC_DIR ARCH=arm64 INSTALL_MOD_PATH=$TEGRA_KERNEL_OUT/modules_install INSTALL_MOD_STRIP=1 O=$TEGRA_KERNEL_OUT modules_install -j8

Note the parameter 'INSTALL_MOD_STRIP=1' should be added, otherwise the built module will be quite large. For example, nvgpu.ko size will increase from 2.6MB to 89MB.

Ramdisk customization

The original ram-disk image can be copied from device (/boot/initrd) or host SDK directory (Linux_for_Tegra/bootloader/l4t_initrd.img).

   1. Extract the initrd by following command: 

zcat xxx/initrd | cpio -idmv

   2. Change all files in this directory owner as root: 

sudo chown root.root * -R

   3. Make some private changes. (For example, to add some special echo message in init script.)
   4. Repack the initrd: 

find . | cpio -o -H newc | gzip > ../initrd.debug

   5. Replace the file /boot/initrd with generated file initrd.debug in above step.
   6. Reboot the device, and check the kernel log to confirm the new initrd works.

In addition, the initrd can also be built-in to kernel image.

   1. Extract the initrd by following command: 

zcat xxx/initrd | cpio -idmv

   2. Change all files in this directory owner as root: 

sudo chown root.root * -R

   3. Make some private changes. (For example, to add some special echo message in init script.)
   4. Repack the initrd: 

find . | cpio -o -H newc > ../initrd.debug.cpio

   5. Edit the kernel config, and add the following line:

    CONFIG_BLK_DEV_INITRD=y
    CONFIG_INITRAMFS_SOURCE="/home/temp/initr.debug.cpio"

   6. Re-build the kernel, replace the kernel image in device.
   7. Comment out the default INITRD in /boot/extlinux/extlinux.conf

    ...
    LINUX /boot/Image
    #INITRD /boot/initrd
    ...

   8. Reboot the device, and the kernel should use the built-in initrd.

OVERLAY FS support in Jetson L4T

L4T BSP is using EXT4 file-system as root-FS in eMMC or SDCARD. For EXT4 file-system, there's high risk of file-system corruption if the device does not shut down correctly, like sudden power loss. 'overlayfs' may help to avoid such file-system corruption, especially in embedded products.

Introduction of overlayfs

(Abstracted from https://en.wikipedia.org/wiki/OverlayFS)

In computing, OverlayFS is a union mount filesystem implementation for Linux. It combines multiple different underlying mount points into one, resulting in single directory structure that contains underlying files and sub-directories from all sources. Common applications overlay a read/write partition over a read-only partition, such as with LiveCDs and IoT devices with limited flash memory write cycles.

(Abstracted from kernel/kernel-4.9/Documentation/filesystems/overlayfs.txt)

An overlay filesystem combines two filesystems - an 'upper' filesystem

and a 'lower' filesystem. When a name exists in both filesystems, the

object in the 'upper' filesystem is visible while the object in the

'lower' filesystem is either hidden or, in the case of directories,

merged with the 'upper' object.

...

The lower filesystem can be any filesystem supported by Linux and does

not need to be writable. The lower filesystem can even be another

overlayfs. The upper filesystem will normally be writable and if it

is it must support the creation of trusted.* extended attributes, and

must provide valid d_type in readdir responses, so NFS is not suitable.

General usage:

 mount -t overlay overlay -olowerdir=/lower,upperdir=/upper,\

workdir=/work /merged

(Abstracted from https://unix.stackexchange.com/questions/324515/linux-filesystem-overlay-what-is-workdir-used-for-overlayfs)

The workdir option is required, and used to prepare files before they are switched to the overlay destination in an atomic action (the workdir needs to be on the same filesystem as the upperdir).

Add overlayfs support in L4T (verified in Jetson AGX devkit, SDK 32.5)

Kernel update

By default, overlayfs is built by module in kernel configuration. And it should be changed as built-in in kernel.

--- ./kernel/kernel-4.9/arch/arm64/configs/tegra_defconfig.orig    2021-07-22 14:55:08.229066055 +0800
+++ ./kernel/kernel-4.9/arch/arm64/configs/tegra_defconfig    2021-07-21 15:24:10.965243034 +0800
@@ -1132,7 +1132,7 @@
 CONFIG_AUTOFS4_FS=y
 CONFIG_FUSE_FS=m
 CONFIG_CUSE=m
-CONFIG_OVERLAY_FS=m
+CONFIG_OVERLAY_FS=y
 CONFIG_VFAT_FS=y
 CONFIG_NTFS_FS=y
 CONFIG_NTFS_RW=y

Rebuild the kernel, and boot the device with new kernel.

initrd update

(A good reference: http://wiki.psuter.ch/doku.php?id=solve_raspbian_sd_card_corruption_issues_with_read-only_mounted_root_partition) Edit the script 'init' in initrd. This is the patch for ram-FS overlayfs:

--- init.orig    2021-07-21 14:35:50.109250634 +0800
+++ init    2021-07-22 16:36:52.948849123 +0800
@@ -142,7 +142,8 @@
         echo "ERROR: ${rootdev} not found" > /dev/kmsg;
         exec /bin/bash;
     fi
-    mount /dev/${rootdev} /mnt/;
+    #do not mount rootdev now
+    #mount /dev/${rootdev} /mnt/;
     if [ $? -ne 0 ]; then
         echo "ERROR: ${rootdev} mount fail..." > /dev/kmsg;
         exec /bin/bash;
@@ -214,13 +215,27 @@
 # Disable luks-srv TA
 nvluks-srv-app -n > /dev/null 2>&1;
 
+#create /mnt as mount point
+mount -t tmpfs inittemp /mnt;
+mkdir /mnt/lower;
+mkdir /mnt/rw;
+mount -t tmpfs root-rw /mnt/rw;
+mkdir /mnt/rw/upper;
+mkdir /mnt/rw/work;
+mkdir /mnt/newroot;
+mount  -o ro /dev/mmcblk0p1 /mnt/lower;
+mount -t overlay -o lowerdir=/mnt/lower,upperdir=/mnt/rw/upper,workdir=/mnt/rw/work overlayfs-root /mnt/newroot;
+mkdir /mnt/proc;
+mkdir /mnt/sys;
+mkdir /mnt/dev;
+
 echo "Rootfs mounted over ${rootdev}" > /dev/kmsg;
 mount -o bind /proc /mnt/proc;
 mount -o bind /sys /mnt/sys;
 mount -o bind /dev/ /mnt/dev;
-cd /mnt;
-cp /etc/resolv.conf etc/resolv.conf
 
-echo "Switching from initrd to actual rootfs" > /dev/kmsg;
-mount --move . /
+cd /mnt/newroot;
+cp /etc/resolv.conf etc/resolv.conf
+echo "Switching from initrd to actual rootfs (ro-root-fs debug)" > /dev/kmsg;
 exec chroot . /sbin/init 2;

Update the device to boot with new initrd. After the kernel's up, the mount information will look like:

overlayfs-root on / type overlay (rw,relatime,lowerdir=/mnt/lower,upperdir=/mnt/rw/upper,workdir=/mnt/rw/work)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
...

(Refer to https://elinux.org/Jetson/L4T_BSP_development_tips#Ramdisk_customization for how to customize initrd in L4T SDK.) Now, every change in root-FS will be in ram-FS, and after the device reboots, all changes will be lost. In addition, a lot of memory will also be consumed if big files added/changed.

Also, non-volatile media, like USB-Disk can also be used as overlayfs, which can reserve the changes in root-FS. Here's the init script patch, which use sda1 (USB-Disk) as overlayfs:

--- init.orig    2021-07-21 14:35:50.109250634 +0800
+++ init.sda1.ok    2021-07-22 18:02:48.424700032 +0800
@@ -142,7 +142,8 @@
         echo "ERROR: ${rootdev} not found" > /dev/kmsg;
         exec /bin/bash;
     fi
-    mount /dev/${rootdev} /mnt/;
+    #do not mount rootdev now
+    #mount /dev/${rootdev} /mnt/;
     if [ $? -ne 0 ]; then
         echo "ERROR: ${rootdev} mount fail..." > /dev/kmsg;
         exec /bin/bash;
@@ -214,13 +215,46 @@
 # Disable luks-srv TA
 nvluks-srv-app -n > /dev/null 2>&1;
 
+# Wait till sda1 ready
+count=0;
+if [ ! -e "/dev/sda1" ]; then
+    while [ ${count} -lt 50 ]
+    do
+        sleep 0.2;
+        count=`expr $count + 1`;
+        if [ -e "/dev/sda1" ]; then
+            break;
+        fi
+    done
+fi
+if [ -e "/dev/sda1" ]; then
+    echo "sda1 found" > /dev/kmsg;
+else
+    echo "ERROR: sda1 not found" > /dev/kmsg;
+    exec /bin/bash;
+fi
+
+#create /mnt as mount point
+mount -t tmpfs inittemp /mnt;
+mkdir /mnt/lower;
+mkdir /mnt/rw;
+mount /dev/sda1 /mnt/rw;
+mkdir /mnt/rw/upper;
+mkdir /mnt/rw/work;
+mkdir /mnt/newroot;
+mount  -o ro /dev/mmcblk0p1 /mnt/lower;
+mount -t overlay -o lowerdir=/mnt/lower,upperdir=/mnt/rw/upper,workdir=/mnt/rw/work overlayfs-root /mnt/newroot;
+mkdir /mnt/proc;
+mkdir /mnt/sys;
+mkdir /mnt/dev;
+
 echo "Rootfs mounted over ${rootdev}" > /dev/kmsg;
 mount -o bind /proc /mnt/proc;
 mount -o bind /sys /mnt/sys;
 mount -o bind /dev/ /mnt/dev;
-cd /mnt;
+
+cd /mnt/newroot;
 cp /etc/resolv.conf etc/resolv.conf
 
-echo "Switching from initrd to actual rootfs" > /dev/kmsg;
-mount --move . /
+echo "Switching from initrd to actual rootfs (ro-root-fs debug)" > /dev/kmsg;
 exec chroot . /sbin/init 2;

With this script, after the kernel's up, the mount information is similar. But the root-FS changes will lie in mounted USB-Disk. So it will not be lost after reboot. After sda1 is mounted:

sudo mount /dev/sda1 /media/sda

The content in /media/sda/upper will look like:

├── upper
│   ├── boot
│   │   └── grub
│   │       └── grubenv
│   ├── dev
│   ├── etc
│   │   ├── asound.conf -> /etc/asound.conf.tegrahdat194ref
│   │   ├── fstab
│   │   ├── machine-id
│   │   ├── rc.local
│   │   └── X11
│   │       └── xorg.conf -> /etc/X11/xorg.conf.t194_ref
│   ├── home
│   │   └── nvidia
│   │       ├── Desktop
│   │       │   └── nv_l4t_readme.desktop
...

All changes in root-FS will still be there. Note the USB-Disk read/write throughput may have impact on system performance. In addition, if the device suddenly shutdown/power-off, the file-system in USB-Disk may, with some probabilities, be corrupted. Anyway, the content in EMMC root-FS should be good, and that makes it possible to scan/fix the file-system error or even format the USB-disk.

Secured device

Signing and Encrypting Kernel, Kernel-DTB, Initrd, and extlinux.conf Files

Basic instructions are in (https://docs.nvidia.com/jetson/archives/l4t-archived/l4t-3261/index.html#page/Tegra%20Linux%20Driver%20Package%20Development%20Guide/bootloader_secure_boot.html#wwpID0ESHA)

Device preparation

Fuse the device with PKC+SBK+KEK0/1/2. Sample

$ cat odmfuse_pkc.xml
<genericfuse MagicId="0x45535546" version="1.0.0">
<fuse name="SecureBootKey" size="16" value="0x123456789abcdef0fedcba9876543210" />
<fuse name="Kek0" size="16" value="0x00112233445566778899aabbccddeeff" />
<fuse name="Kek1" size="16" value="0x112233445566778899aabbccddeeff00" />
<fuse name="Kek2" size="16" value="0x2233445566778899aabbccddeeff0011" />
<fuse name="PublicKeyHash" size="32" value="0xf3c1b8aae1a056e3cb2226121c0b3cb77f01fe6c8131ca274cce7939b45559ca" />
<fuse name="BootSecurityInfo" size="4" value="0x6" />
<fuse name="SecurityMode" size="4" value="0x1" />
</genericfuse>

Prepare the EKB

Download source package from 32.7.1 BSP

wget https://developer.nvidia.com/embedded/l4t/r32_release_v7.1/sources/t186/public_sources.tbz2

Extract the trusty_src.tbz2 in source package, and enter directory

.../trusty/app/nvidia-sample/hwkey-agent/CA_sample/tool/gen_ekb

Run following command, with correct paramters

NOTE: the fixed vector (FV) should match the one in trusty/app/nvidia-sample/hwkey-agent/key_mgnt.c

/*
* Random fixed vector for EKB.
*
* Note: This vector MUST match the 'fv' vector used for EKB binary
* generation process.
* ba d6 6e b4 48 49 83 68 4b 99 2f e5 4a 64 8b b8
*/
static uint8_t fv_for_ekb[] = {
0xba, 0xd6, 0x6e, 0xb4, 0x48, 0x49, 0x83, 0x68,
0x4b, 0x99, 0x2f, 0xe5, 0x4a, 0x64, 0x8b, 0xb8,
};

With following command:

$ python3 gen_ekb.py -kek2_key KEK2_.txt \
-fv fv_ekb \
-in_sym_key user_key_.txt \
-in_sym_key2 sym2_key_file.txt \
-out eks_image.bin

Keys format and content:

building-pc:~/Work/jetson_sdk/32.7.1/sources/Linux_for_Tegra/source/public/trusty/trusty/app/nvidia-sample/hwkey-agent/CA_sample/tool/gen_ekb$ cat KEK2_.txt
2233445566778899aabbccddeeff0011
building-pc:~/Work/jetson_sdk/32.7.1/sources/Linux_for_Tegra/source/public/trusty/trusty/app/nvidia-sample/hwkey-agent/CA_sample/tool/gen_ekb$ cat fv_ekb
bad66eb4484983684b992fe54a648bb8
building-pc:~/Work/jetson_sdk/32.7.1/sources/Linux_for_Tegra/source/public/trusty/trusty/app/nvidia-sample/hwkey-agent/CA_sample/tool/gen_ekb$ cat user_key_.txt
123456789abcdef0ffeeddccbbaa9988
building-pc:~/Work/jetson_sdk/32.7.1/sources/Linux_for_Tegra/source/public/trusty/trusty/app/nvidia-sample/hwkey-agent/CA_sample/tool/gen_ekb$ cat sym2_key_file.txt
445566778899aabbccddeeff00112233

Prepare the flash environment

Create symbol link for new EKS generated in above step

building-pc:~/Work/jetson_sdk/32.7.1/Linux_for_Tegra$ ll bootloader/eks.img
lrwxrwxrwx 1 124 10月 31 13:59 bootloader/eks.img -> ../../sources/Linux_for_Tegra/source/public/trusty/trusty/app/nvidia-sample/hwkey-agent/CA_sample/tool/gen_ekb/eks_image.bin

Flash the device

sudo BOARDID=3668 BOARDSKU=0001 FAB=100 ./flash.sh -r -u rsa_priv_3072.pem -v sbk.key --user_key user_key.txt jetson-xavier-nx-devkit-emmc mmcblk0p1

user_key.txt content:

building-pc:~/Work/jetson_sdk/32.7.1/Linux_for_Tegra$ cat user_key.txt
0x12345678 0x9abcdef0 0xffeeddcc 0xbbaa9988

Verify the log

Trusty OS should print following log through debug UART:

...
hwkey-agent: 41: hwkey-agent is running!!
hwkey-agent: 347: key_mgnt_processing .......
hwkey-agent: 255: Setting EKB key 0 to slot 14
hwkey-agent: 178: Init hweky-agent services!!
...

CBoot should authenticate and decrypt the payload by following logs:

...
[0015.245] I> Loading extlinux.conf ...
[0015.246] I> Loading extlinux.conf binary from rootfs ...
[0015.246] I> rootfs path: /sdmmc_user/boot/extlinux/extlinux.conf
[0015.289] I> Loading extlinux.conf sig file from rootfs ...
[0015.289] I> rootfs path: /sdmmc_user/boot/extlinux/extlinux.conf.sig
[0015.314] I> Validate extlinux.conf ...
[0015.314] I> T19x: Authenticate extlinux.conf (bin_type: 54), max size 0x2000
[0015.316] I> RSA PSS signature check: OK
[0015.316] I> authenticate_oem_payload: Decrypt the binary
[0015.317] I> L4T boot options
[0015.317] I> [1]: "primary kernel"
[0015.317] I> Enter choice:
...

In root-FS, extlinux.conf is encrypted.

nvidia@xavier-8gb-secured:~$ hexdump -C /boot/extlinux/extlinux.conf
00000000 93 9a 9b b1 cd ee dd 39 e1 d3 05 4d a6 65 c1 18 |.......9...M.e..|
00000010 dc 89 85 9c 07 6b a2 db 2d f7 45 70 2f 11 80 3f |.....k..-.Ep/..?|
00000020 5e a6 de ba 3a 73 c2 dc b1 17 7a 02 34 22 b3 43 |^...:s....z.4".C|
00000030 a5 a4 79 a0 35 a6 09 d1 ea 99 2c 8c 3e 2c 6d 25 |..y.5.....,.>,m%|
00000040 c6 d4 4b 65 9e b8 24 56 ee a5 9e 18 4e f6 ca 30 |..Ke..$V....N..0|
00000050 37 08 9b e6 c9 07 45 ca 60 86 c5 28 28 08 18 66 |7.....E.`..((..f|
...

Debug tips

Following error log means the EKB content is not correct.

hwkey-agent: 41: hwkey-agent is running!!
hwkey-agent: 347: key_mgnt_processing .......
hwkey-agent: 162: ekb_verification: EKB_CMAC verification is not match.
hwkey-agent: 400: key_mgnt_processing: failed (-7)
hwkey-agent: 45: main: Failed to verify or extract EKB (-7).
exit called, thread 0xffffffffea8a4d58, name trusty_app_2_92b92883-f96a-4177

Tips:

   * KEK2 in fuse blob and EKB generation should match.
   * user_key in flash command and EKB generation should match.
   * FV in EKB generation and TOS app should match.