Jetson/FAQ/BSP/RootFS Reduction

= How to reduce disk usage on Jetson platform =

Jetson platform files system is based on ubuntu sample file system. Some Nvidia platform related binaries (BSP related) are added to support platform basic function. Jetson SDK Components including CUDA, cuDNN, TensorRT, OpenCV, VisionWorks, DeepStream and multimedia API provide AI related support.

This page breaks down Nvidia platform related binaries and Jetson SDK Component, so user can remove unused files to save space. A command to list all installed ubuntu packages by size is also provided for user to remove unused package based on needs.

Nvidia platform related binaries breakdown
Below is a breakdown of Nvidia platform related binaries.

/ ├── boot (43M)           Image(kernel binary), dtb and extlinux.conf. All are not used by Xavier, Image and extlinux.conf are used by Nano, TX1 and TX2 ├── etc (536K)          Nvidia specified config file. DO NOT touch ├── lib │   ├── firmware (4.3M) Firmwares used by platform. DO NOT touch │   ├── modules (47M)   Modules(.ko) used by platform. Can delete some based on needs │   └── systemd (32K)   Nvidia specified config file. DO NOT touch ├── opt │   └── nvidia (17M)    Jetson gpio python lib and usb device mode service. Can delete based on needs ├── usr │   ├── bin (356K)      Nvidia speficied tools. DO NOT touch │   ├── lib (130M)      Nvidia speficied libs. DO NOT touch │   ├── sbin (2M)       Nvidia speficied tools. DO NOT touch │   ├── share (2.5M)    Nvidia specified boot logo, license, icon, etc. DO NOT touch │   └── src (234M)      Linux header files and grahics demos. Can delete based on needs └── var └── nvidia (188K)   Placeholder for nvcam. Can delete based on needs

Jetson SDK Components
Based on Jetson OS, Jetson SDK Components, including CUDA, cuDNN, TensorRT, OpenCV, VisionWorks, MultiMedia API and DeepStream, are installed by sdkmanager(or jetpack tool for older version). MultiMedia API is installed by tbz2 package while others are installed by deb package.

Both Jetson SDK libs(both static and dynamic libs) and sample code are installed and sample code can be deleted to save space. And if user's application is dynamicly linked to these libs, all static libs can be deleted.

After Jetson SDK Components are installed, run below commands can remove all deb files. sudo apt clean   # clean debs in /var/cache/apt/archives sudo rm /etc/apt/sources.list.d/*       # remove /var/cudaxxxx, /var/visonworksxxxx from apt source list sudo rm /var/cuda-repo-10-0-local-10.0.326/ /var/visionworks-repo/ /var/visionworks-sfm-repo/ /var/visionworks-tracking-repo/ -rf    # remove nvidia's debs

Below table list all Jetson SDK Components brief breakdown.

Remove installed deb packages
Run below command can remove all cached .deb file. sudo apt clean Run below command can remove all non-required installed packages. sudo apt autoremove Below command can list all install deb packages by size. User can remove the undesired packages by hand to save space. dpkg-query -Wf '${Installed-Size}\t${Package}\n' | sort -rn

= Example =

Remove all deb installation packages
sudo apt clean sudo rm /etc/apt/sources.list.d/* sudo rm -rf /var/cuda-repo-10-0-local-10.0.326/ /var/visionworks-repo/ /var/visionworks-sfm-repo/ /var/visionworks-tracking-repo

Remove ubuntu-desktop and the accessories
sudo apt-get purge gnome-shell sudo apt-get purge docker* sudo apt-get purge chromium-browser thunderbird fonts-noto-cjk libreoffice-common  containerd  snapd sudo apt-get remove --purge python* libpython* sudo apt-get purge libwebkit2gtk-4.0-37 libqt5webkit5  ubuntu-wallpapers-bionic freepats  libreoffice-writer libicu60  libreoffice-calc vim-runtime libflite1  libperl5.26 libmozjs-52-0 humanity-icon-theme samba-libs kwin-data perl-modules-5.26 light-themes libjavascriptcoregtk-4.0-18 chromium-browser-l10n sudo apt-get purge libvisionworks libvisionworks-sfm-dev libvisionworks-sfm-repo sudo apt-get autoremove

Remove some source code and sample code
cd /usr/src && sudo rm -rf cudnn_samples_v7 linux-headers-4.9.140-tegra-ubuntu18.04_aarch64 linux-headers-4.9.140-tegra-linux_x86_64 tensorrt nvidia rm -rf /home/nvidia/VisionWorks-SFM-0.90-Samples

= How to fix library missing issue after reduction = Sometimes after removing ubuntu packages, application execution may fail due to depended libraries missing.

For example, after removing packages by the above example, Jetson Multimedia API sample cannot run like below. nvidia@nvidia-desktop:~/jetson_multimedia_api/samples/00_video_decode$ ./video_decode ./video_decode: error while loading shared libraries: libv4l2.so.0: cannot open shared object file: No such file or directory

"ldd" command shows both libv4l2.so.0 and libnvjpeg.so are missing like below. So we need to copy these libraries from a clean platform. nvidia@nvidia-desktop:~/jetson_multimedia_api/samples/00_video_decode$ ldd video_decode linux-vdso.so.1 (0x0000007f869e8000) libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0 (0x0000007f86916000) libv4l2.so.0 => not found libEGL.so.1 => /usr/lib/aarch64-linux-gnu/libEGL.so.1 (0x0000007f868f5000) libGLESv2.so.2 => /usr/lib/aarch64-linux-gnu/libGLESv2.so.2 (0x0000007f868bf000) libX11.so.6 => /usr/lib/aarch64-linux-gnu/libX11.so.6 (0x0000007f86796000) libnvbuf_utils.so.1.0.0 => /usr/lib/aarch64-linux-gnu/tegra/libnvbuf_utils.so.1.0.0 (0x0000007f8677c000) libnvjpeg.so => not found libdrm.so.2 => /usr/lib/aarch64-linux-gnu/libdrm.so.2 (0x0000007f8674a000) libstdc++.so.6 => /usr/lib/aarch64-linux-gnu/libstdc++.so.6 (0x0000007f865b7000) libgcc_s.so.1 => /lib/aarch64-linux-gnu/libgcc_s.so.1 (0x0000007f86593000) libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000007f8643a000) /lib/ld-linux-aarch64.so.1 (0x0000007f869bd000) libGLdispatch.so.0 => /usr/lib/aarch64-linux-gnu/libGLdispatch.so.0 (0x0000007f8630e000) libdl.so.2 => /lib/aarch64-linux-gnu/libdl.so.2 (0x0000007f862f9000) libxcb.so.1 => /usr/lib/aarch64-linux-gnu/libxcb.so.1 (0x0000007f862c9000) libnvrm.so => /usr/lib/aarch64-linux-gnu/tegra/libnvrm.so (0x0000007f86287000) libnvrm_graphics.so => /usr/lib/aarch64-linux-gnu/tegra/libnvrm_graphics.so (0x0000007f86268000) libnvddk_vic.so => /usr/lib/aarch64-linux-gnu/tegra/libnvddk_vic.so (0x0000007f8624a000) libnvbuf_fdmap.so.1.0.0 => /usr/lib/aarch64-linux-gnu/tegra/libnvbuf_fdmap.so.1.0.0 (0x0000007f86237000) libnvll.so => /usr/lib/aarch64-linux-gnu/tegra/libnvll.so (0x0000007f8621b000) libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000007f86161000) libXau.so.6 => /usr/lib/aarch64-linux-gnu/libXau.so.6 (0x0000007f8614e000) libXdmcp.so.6 => /usr/lib/aarch64-linux-gnu/libXdmcp.so.6 (0x0000007f86139000) libnvos.so => /usr/lib/aarch64-linux-gnu/tegra/libnvos.so (0x0000007f8611b000) libnvdc.so => /usr/lib/aarch64-linux-gnu/tegra/libnvdc.so (0x0000007f860fc000) libbsd.so.0 => /lib/aarch64-linux-gnu/libbsd.so.0 (0x0000007f860da000) librt.so.1 => /lib/aarch64-linux-gnu/librt.so.1 (0x0000007f860c3000) libnvimp.so => /usr/lib/aarch64-linux-gnu/tegra/libnvimp.so (0x0000007f860ae000)

On a clean platform, we can use the "find" and "ls" command to get all the necessary libraries like below. nvidia@nvidia-xavier:~$ find /usr/lib/ -name "libv4l2.so*" /usr/lib/aarch64-linux-gnu/libv4l2.so.0 /usr/lib/aarch64-linux-gnu/libv4l2.so /usr/lib/aarch64-linux-gnu/libv4l2.so.0.0.0 /usr/lib/aarch64-linux-gnu/tegra/libv4l2.so.0 /usr/lib/aarch64-linux-gnu/libv4l2.so.0.0.999999

nvidia@nvidia-xavier:/usr/lib/aarch64-linux-gnu$ ls -l /usr/lib/aarch64-linux-gnu/libv4l2.so* lrwxrwxrwx 1 root root   12 Dec 26 16:15 /usr/lib/aarch64-linux-gnu/libv4l2.so -> libv4l2.so.0 lrwxrwxrwx 1 root root   21 Dec 19 14:43 /usr/lib/aarch64-linux-gnu/libv4l2.so.0 -> libv4l2.so.0.0.999999 -rw-r--r-- 1 root root 55424 Feb 10 2018 /usr/lib/aarch64-linux-gnu/libv4l2.so.0.0.0 lrwxrwxrwx 1 root root   18 Dec 10 15:08 /usr/lib/aarch64-linux-gnu/libv4l2.so.0.0.999999 -> tegra/libnvv4l2.so

After restore libv4l2(/usr/lib/aarch64-linux-gnu/libv4l2.so.0, /usr/lib/aarch64-linux-gnu/libv4l2.so.0.0.0, /usr/lib/aarch64-linux-gnu/libv4l2.so.0.0.999999 and /usr/lib/aarch64-linux-gnu/tegra/libnvv4l2.so) and libnvjpeg(/usr/lib/aarch64-linux-gnu/tegra/libnvjpeg.so), remember to run "sudo ldconfig" to update /etc/ld.so.cache

Now, re-run video_decode, it still fails. That's because the new added libv4l2 and libnvjpeg depends on other missing libraries like below: nvidia@nvidia-desktop:~/jetson_multimedia_api/samples/00_video_decode$ ldd video_decode linux-vdso.so.1 (0x0000007f9abff000) libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0 (0x0000007f9ab2d000) libv4l2.so.0 => /usr/lib/aarch64-linux-gnu/libv4l2.so.0 (0x0000007f9aa18000) libEGL.so.1 => /usr/lib/aarch64-linux-gnu/libEGL.so.1 (0x0000007f9a9f7000) libGLESv2.so.2 => /usr/lib/aarch64-linux-gnu/libGLESv2.so.2 (0x0000007f9a9c1000) libX11.so.6 => /usr/lib/aarch64-linux-gnu/libX11.so.6 (0x0000007f9a898000) libnvbuf_utils.so.1.0.0 => /usr/lib/aarch64-linux-gnu/tegra/libnvbuf_utils.so.1.0.0 (0x0000007f9a87e000) libnvjpeg.so => /usr/lib/aarch64-linux-gnu/tegra/libnvjpeg.so (0x0000007f9a826000) libdrm.so.2 => /usr/lib/aarch64-linux-gnu/libdrm.so.2 (0x0000007f9a7f4000) libstdc++.so.6 => /usr/lib/aarch64-linux-gnu/libstdc++.so.6 (0x0000007f9a661000) libgcc_s.so.1 => /lib/aarch64-linux-gnu/libgcc_s.so.1 (0x0000007f9a63d000) libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000007f9a4e4000) /lib/ld-linux-aarch64.so.1 (0x0000007f9abd4000) libv4lconvert.so.0 => not found libdl.so.2 => /lib/aarch64-linux-gnu/libdl.so.2 (0x0000007f9a4cf000) libGLdispatch.so.0 => /usr/lib/aarch64-linux-gnu/libGLdispatch.so.0 (0x0000007f9a3a3000) libxcb.so.1 => /usr/lib/aarch64-linux-gnu/libxcb.so.1 (0x0000007f9a373000) libnvrm.so => /usr/lib/aarch64-linux-gnu/tegra/libnvrm.so (0x0000007f9a331000) libnvrm_graphics.so => /usr/lib/aarch64-linux-gnu/tegra/libnvrm_graphics.so (0x0000007f9a312000) libnvddk_vic.so => /usr/lib/aarch64-linux-gnu/tegra/libnvddk_vic.so (0x0000007f9a2f4000) libnvbuf_fdmap.so.1.0.0 => /usr/lib/aarch64-linux-gnu/tegra/libnvbuf_fdmap.so.1.0.0 (0x0000007f9a2e1000) libnvddk_2d_v2.so => /usr/lib/aarch64-linux-gnu/tegra/libnvddk_2d_v2.so (0x0000007f9a2bc000) libnvbufsurface.so.1.0.0 => not found libnvll.so => /usr/lib/aarch64-linux-gnu/tegra/libnvll.so (0x0000007f9a2a0000) libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000007f9a1e6000) libXau.so.6 => /usr/lib/aarch64-linux-gnu/libXau.so.6 (0x0000007f9a1d3000) libXdmcp.so.6 => /usr/lib/aarch64-linux-gnu/libXdmcp.so.6 (0x0000007f9a1be000) libnvos.so => /usr/lib/aarch64-linux-gnu/tegra/libnvos.so (0x0000007f9a1a0000) libnvdc.so => /usr/lib/aarch64-linux-gnu/tegra/libnvdc.so (0x0000007f9a181000) libbsd.so.0 => /lib/aarch64-linux-gnu/libbsd.so.0 (0x0000007f9a15f000) librt.so.1 => /lib/aarch64-linux-gnu/librt.so.1 (0x0000007f9a148000) libnvimp.so => /usr/lib/aarch64-linux-gnu/tegra/libnvimp.so (0x0000007f9a133000)

After restore libv4lconvert and libnvbufsurface just like libv4l2, all dependencies are available now. nvidia@nvidia-desktop:~/jetson_multimedia_api/samples/00_video_decode$ ldd video_decode linux-vdso.so.1 (0x0000007f8bb5c000) libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0 (0x0000007f8ba8a000) libv4l2.so.0 => /usr/lib/aarch64-linux-gnu/libv4l2.so.0 (0x0000007f8b975000) libEGL.so.1 => /usr/lib/aarch64-linux-gnu/libEGL.so.1 (0x0000007f8b954000) libGLESv2.so.2 => /usr/lib/aarch64-linux-gnu/libGLESv2.so.2 (0x0000007f8b91e000) libX11.so.6 => /usr/lib/aarch64-linux-gnu/libX11.so.6 (0x0000007f8b7f5000) libnvbuf_utils.so.1.0.0 => /usr/lib/aarch64-linux-gnu/tegra/libnvbuf_utils.so.1.0.0 (0x0000007f8b7db000) libnvjpeg.so => /usr/lib/aarch64-linux-gnu/tegra/libnvjpeg.so (0x0000007f8b783000) libdrm.so.2 => /usr/lib/aarch64-linux-gnu/libdrm.so.2 (0x0000007f8b751000) libstdc++.so.6 => /usr/lib/aarch64-linux-gnu/libstdc++.so.6 (0x0000007f8b5be000) libgcc_s.so.1 => /lib/aarch64-linux-gnu/libgcc_s.so.1 (0x0000007f8b59a000) libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000007f8b441000) /lib/ld-linux-aarch64.so.1 (0x0000007f8bb31000) libv4lconvert.so.0 => /usr/lib/aarch64-linux-gnu/libv4lconvert.so.0 (0x0000007f8b3bc000) libdl.so.2 => /lib/aarch64-linux-gnu/libdl.so.2 (0x0000007f8b3a7000) libGLdispatch.so.0 => /usr/lib/aarch64-linux-gnu/libGLdispatch.so.0 (0x0000007f8b27b000) libxcb.so.1 => /usr/lib/aarch64-linux-gnu/libxcb.so.1 (0x0000007f8b24b000) libnvrm.so => /usr/lib/aarch64-linux-gnu/tegra/libnvrm.so (0x0000007f8b209000) libnvrm_graphics.so => /usr/lib/aarch64-linux-gnu/tegra/libnvrm_graphics.so (0x0000007f8b1ea000) libnvddk_vic.so => /usr/lib/aarch64-linux-gnu/tegra/libnvddk_vic.so (0x0000007f8b1cc000) libnvbuf_fdmap.so.1.0.0 => /usr/lib/aarch64-linux-gnu/tegra/libnvbuf_fdmap.so.1.0.0 (0x0000007f8b1b9000) libnvddk_2d_v2.so => /usr/lib/aarch64-linux-gnu/tegra/libnvddk_2d_v2.so (0x0000007f8b194000) libnvbufsurface.so.1.0.0 => /usr/lib/aarch64-linux-gnu/tegra/libnvbufsurface.so.1.0.0 (0x0000007f8b116000) libnvll.so => /usr/lib/aarch64-linux-gnu/tegra/libnvll.so (0x0000007f8b0fa000) libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000007f8b040000) librt.so.1 => /lib/aarch64-linux-gnu/librt.so.1 (0x0000007f8b029000) libXau.so.6 => /usr/lib/aarch64-linux-gnu/libXau.so.6 (0x0000007f8b016000) libXdmcp.so.6 => /usr/lib/aarch64-linux-gnu/libXdmcp.so.6 (0x0000007f8b001000) libnvos.so => /usr/lib/aarch64-linux-gnu/tegra/libnvos.so (0x0000007f8afe3000) libnvdc.so => /usr/lib/aarch64-linux-gnu/tegra/libnvdc.so (0x0000007f8afc4000) libbsd.so.0 => /lib/aarch64-linux-gnu/libbsd.so.0 (0x0000007f8afa2000) libnvimp.so => /usr/lib/aarch64-linux-gnu/tegra/libnvimp.so (0x0000007f8af8d000)