Jetson/FAQ/BSP/RootFS Reduction

From eLinux.org
< Jetson‎ | FAQ/BSP
Revision as of 02:52, 3 February 2020 by Peter Pan (talk | contribs) (How to fix library missing issue after reduction)
Jump to: navigation, search

How to reduce disk usage on Jetson platform

Jetson platform files system is based on ubuntu sample file system. Some Nvidia platform related binaries (BSP related) are added to support platform basic function. Jetson SDK Components including CUDA, cuDNN, TensorRT, OpenCV, VisionWorks, DeepStream and multimedia API provide AI related support.

This page breaks down Nvidia platform related binaries and Jetson SDK Component, so user can remove unused files to save space. A command to list all installed ubuntu packages by size is also provided for user to remove unused package based on needs.

Nvidia platform related binaries breakdown

Below is a breakdown of Nvidia platform related binaries.

 /
 ├── boot (43M)           Image(kernel binary), dtb and extlinux.conf. All are not used by Xavier, Image and extlinux.conf are used by Nano, TX1 and TX2
 ├── etc (536K)           Nvidia specified config file. DO NOT touch
 ├── lib
 │   ├── firmware (4.3M)  Firmwares used by platform. DO NOT touch
 │   ├── modules (47M)    Modules(.ko) used by platform. Can delete some based on needs
 │   └── systemd (32K)    Nvidia specified config file. DO NOT touch
 ├── opt
 │   └── nvidia  (17M)    Jetson gpio python lib and usb device mode service. Can delete based on needs
 ├── usr
 │   ├── bin (356K)       Nvidia speficied tools. DO NOT touch
 │   ├── lib (130M)       Nvidia speficied libs. DO NOT touch
 │   ├── sbin (2M)        Nvidia speficied tools. DO NOT touch
 │   ├── share (2.5M)     Nvidia specified boot logo, license, icon, etc. DO NOT touch
 │   └── src (234M)       Linux header files and grahics demos. Can delete based on needs
 └── var
     └── nvidia (188K)    Placeholder for nvcam. Can delete based on needs

Jetson SDK Components

Based on Jetson OS, Jetson SDK Components, including CUDA, cuDNN, TensorRT, OpenCV, VisionWorks, MultiMedia API and DeepStream, are installed by sdkmanager(or jetpack tool for older version). MultiMedia API is installed by tbz2 package while others are installed by deb package.

Both Jetson SDK libs(both static and dynamic libs) and sample code are installed and sample code can be deleted to save space. And if user's application is dynamicly linked to these libs, all static libs can be deleted.

After Jetson SDK Components are installed, run below commands can remove all deb files.

 sudo apt clean    # clean debs in /var/cache/apt/archives
 sudo rm /etc/apt/sources.list.d/*        # remove /var/cudaxxxx, /var/visonworksxxxx from apt source list
 sudo rm /var/cuda-repo-10-0-local-10.0.326/ /var/visionworks-repo/ /var/visionworks-sfm-repo/ /var/visionworks-tracking-repo/ -rf     # remove nvidia's debs

Below table list all Jetson SDK Components brief breakdown.

CUDA

Elememt Path Size
sample /usr/local/cuda/samples 198M
dynamic lib /usr/local/cuda/targets/aarch64-linux/lib/*.so* 723M
static lib /usr/local/cuda/targets/aarch64-linux/lib/*.a 898M
doc /usr/local/cuda/doc 231M

cuDNN

Elememt Path Size
sample /usr/src/cudnn_samples_v7 11M
dynamic lib /usr/lib/aarch64-linux-gnu/libcudnn*.so* 367M
static lib /usr/lib/aarch64-linux-gnu/libcudnn*.a 358M

TensorRT

Elememt Path Size
sample /usr/src/tensorrt 783M
dynamic lib /usr/lib/aarch64-linux-gnu/libnvcaffe_parser*.so*
/usr/lib/aarch64-linux-gnu/libnvinfer*.so*
/usr/lib/aarch64-linux-gnu/libnvonnxparser*.so*
/usr/lib/aarch64-linux-gnu/libnvparsers*.so*
156M
static lib /usr/lib/aarch64-linux-gnu/libnvcaffe_parser*.a
/usr/lib/aarch64-linux-gnu/libnvinfer*.a
/usr/lib/aarch64-linux-gnu/libnvonnxparser*.a
/usr/lib/aarch64-linux-gnu/libnvparsers*.a
172M
doc /usr/share/doc/libnvinfer* 24K

OpenCV

Elememt Path Size
sample /usr/share/OpenCV 11M
dynamic lib /usr/lib/libopencv*.so* 19M

VisionWorks

Elememt Path Size
sample /usr/share/visionworks*
~/VisionWorks-SFM*Samples
131M
dynamic lib /usr/lib/libvisionworks*.so* 28M
doc /usr/share/doc/libvisionworks* 352K

DeepStream

Elememt Path Size
sample /opt/nvidia/deepstream/deepstream*/samples 205M
dynamic lib /opt/nvidia/deepstream/deepstream*/lib 52M
doc /opt/nvidia/deepstream/deepstream*/doc 16K

MultiMedia API

Path Size
/usr/src/tegra_multimedia_api 107M

Remove installed deb packages

Run below command can remove all cached .deb file.

 sudo apt clean

Run below command can remove all non-required installed packages.

 sudo apt autoremove

Below command can list all install deb packages by size. User can remove the undesired packages by hand to save space.

 dpkg-query -Wf '${Installed-Size}\t${Package}\n' | sort -rn

Example

Remove all deb installation packages

 sudo apt clean
 sudo rm /etc/apt/sources.list.d/*
 sudo rm -rf /var/cuda-repo-10-0-local-10.0.326/ /var/visionworks-repo/ /var/visionworks-sfm-repo/ /var/visionworks-tracking-repo

Remove ubuntu-desktop and the accessories

 sudo apt-get purge gnome-shell
 sudo apt-get purge docker*
 sudo apt-get purge chromium-browser thunderbird fonts-noto-cjk  libreoffice-common  containerd  snapd
 sudo apt-get remove --purge python* libpython*
 sudo apt-get purge libwebkit2gtk-4.0-37  libqt5webkit5  ubuntu-wallpapers-bionic freepats  libreoffice-writer libicu60  libreoffice-calc vim-runtime libflite1  libperl5.26 libmozjs-52-0 humanity-icon-theme samba-libs  iso-codes kwin-data perl-modules-5.26 light-themes libjavascriptcoregtk-4.0-18 chromium-browser-l10n
 sudo apt-get purge libvisionworks libvisionworks-sfm-dev libvisionworks-sfm-repo
 sudo apt-get autoremove

Remove some source code and sample code

 cd /usr/src && sudo rm -rf cudnn_samples_v7 linux-headers-4.9.140-tegra-ubuntu18.04_aarch64 linux-headers-4.9.140-tegra-linux_x86_64  tensorrt nvidia
 rm -rf /home/nvidia/VisionWorks-SFM-0.90-Samples

How to fix library missing issue after reduction

Sometimes after removing ubuntu packages, application execution may fail due to depended libraries missing.

For example, after removing packages by above example, Jetson Multimedia API sample cannot run like below.

 nvidia@nvidia-desktop:~/jetson_multimedia_api/samples/00_video_decode$ ./video_decode
 ./video_decode: error while loading shared libraries: libv4l2.so.0: cannot open shared object file: No such file or directory

ldd command shows both libv4l2.so.0 and libnvjpeg.so are missing like below. So we need to copy these libraries from clean platform.

 nvidia@nvidia-desktop:~/jetson_multimedia_api/samples/00_video_decode$ ldd video_decode
       linux-vdso.so.1 (0x0000007f869e8000)
       libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0 (0x0000007f86916000)
       libv4l2.so.0 => not found
       libEGL.so.1 => /usr/lib/aarch64-linux-gnu/libEGL.so.1 (0x0000007f868f5000)
       libGLESv2.so.2 => /usr/lib/aarch64-linux-gnu/libGLESv2.so.2 (0x0000007f868bf000)
       libX11.so.6 => /usr/lib/aarch64-linux-gnu/libX11.so.6 (0x0000007f86796000)
       libnvbuf_utils.so.1.0.0 => /usr/lib/aarch64-linux-gnu/tegra/libnvbuf_utils.so.1.0.0 (0x0000007f8677c000)
       libnvjpeg.so => not found
       libdrm.so.2 => /usr/lib/aarch64-linux-gnu/libdrm.so.2 (0x0000007f8674a000)
       libstdc++.so.6 => /usr/lib/aarch64-linux-gnu/libstdc++.so.6 (0x0000007f865b7000)
       libgcc_s.so.1 => /lib/aarch64-linux-gnu/libgcc_s.so.1 (0x0000007f86593000)
       libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000007f8643a000)
       /lib/ld-linux-aarch64.so.1 (0x0000007f869bd000)
       libGLdispatch.so.0 => /usr/lib/aarch64-linux-gnu/libGLdispatch.so.0 (0x0000007f8630e000)
       libdl.so.2 => /lib/aarch64-linux-gnu/libdl.so.2 (0x0000007f862f9000)
       libxcb.so.1 => /usr/lib/aarch64-linux-gnu/libxcb.so.1 (0x0000007f862c9000)
       libnvrm.so => /usr/lib/aarch64-linux-gnu/tegra/libnvrm.so (0x0000007f86287000)
       libnvrm_graphics.so => /usr/lib/aarch64-linux-gnu/tegra/libnvrm_graphics.so (0x0000007f86268000)
       libnvddk_vic.so => /usr/lib/aarch64-linux-gnu/tegra/libnvddk_vic.so (0x0000007f8624a000)
       libnvbuf_fdmap.so.1.0.0 => /usr/lib/aarch64-linux-gnu/tegra/libnvbuf_fdmap.so.1.0.0 (0x0000007f86237000)
       libnvll.so => /usr/lib/aarch64-linux-gnu/tegra/libnvll.so (0x0000007f8621b000)
       libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000007f86161000)
       libXau.so.6 => /usr/lib/aarch64-linux-gnu/libXau.so.6 (0x0000007f8614e000)
       libXdmcp.so.6 => /usr/lib/aarch64-linux-gnu/libXdmcp.so.6 (0x0000007f86139000)
       libnvos.so => /usr/lib/aarch64-linux-gnu/tegra/libnvos.so (0x0000007f8611b000)
       libnvdc.so => /usr/lib/aarch64-linux-gnu/tegra/libnvdc.so (0x0000007f860fc000)
       libbsd.so.0 => /lib/aarch64-linux-gnu/libbsd.so.0 (0x0000007f860da000)
       librt.so.1 => /lib/aarch64-linux-gnu/librt.so.1 (0x0000007f860c3000)
       libnvimp.so => /usr/lib/aarch64-linux-gnu/tegra/libnvimp.so (0x0000007f860ae000)

On clean platform, we can use "find" and "ls" command to get all the necessary libraries like below.

 nvidia@nvidia-xavier:~$ find /usr/lib/ -name "libv4l2.so*"
 /usr/lib/aarch64-linux-gnu/libv4l2.so.0
 /usr/lib/aarch64-linux-gnu/libv4l2.so
 /usr/lib/aarch64-linux-gnu/libv4l2.so.0.0.0
 /usr/lib/aarch64-linux-gnu/tegra/libv4l2.so.0
 /usr/lib/aarch64-linux-gnu/libv4l2.so.0.0.999999
 nvidia@nvidia-xavier:/usr/lib/aarch64-linux-gnu$ ls -l /usr/lib/aarch64-linux-gnu/libv4l2.so*
 lrwxrwxrwx 1 root root    12 Dec 26 16:15 /usr/lib/aarch64-linux-gnu/libv4l2.so -> libv4l2.so.0
 lrwxrwxrwx 1 root root    21 Dec 19 14:43 /usr/lib/aarch64-linux-gnu/libv4l2.so.0 -> libv4l2.so.0.0.999999
 -rw-r--r-- 1 root root 55424 Feb 10  2018 /usr/lib/aarch64-linux-gnu/libv4l2.so.0.0.0
 lrwxrwxrwx 1 root root    18 Dec 10 15:08 /usr/lib/aarch64-linux-gnu/libv4l2.so.0.0.999999 -> tegra/libnvv4l2.so

After restore libv4l2 and libnvjpeg, remember to run "sudo ldconfig" to update /etc/ld.so.cache

Now, re-run video_decode, it still fails. That's because the new added libv4l2 and libnvjpeg depends on other missing libraries like below:

 nvidia@nvidia-desktop:~/jetson_multimedia_api/samples/00_video_decode$ ldd video_decode
       linux-vdso.so.1 (0x0000007f9abff000)
       libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0 (0x0000007f9ab2d000)
       libv4l2.so.0 => /usr/lib/aarch64-linux-gnu/libv4l2.so.0 (0x0000007f9aa18000)
       libEGL.so.1 => /usr/lib/aarch64-linux-gnu/libEGL.so.1 (0x0000007f9a9f7000)
       libGLESv2.so.2 => /usr/lib/aarch64-linux-gnu/libGLESv2.so.2 (0x0000007f9a9c1000)
       libX11.so.6 => /usr/lib/aarch64-linux-gnu/libX11.so.6 (0x0000007f9a898000)
       libnvbuf_utils.so.1.0.0 => /usr/lib/aarch64-linux-gnu/tegra/libnvbuf_utils.so.1.0.0 (0x0000007f9a87e000)
       libnvjpeg.so => /usr/lib/aarch64-linux-gnu/tegra/libnvjpeg.so (0x0000007f9a826000)
       libdrm.so.2 => /usr/lib/aarch64-linux-gnu/libdrm.so.2 (0x0000007f9a7f4000)
       libstdc++.so.6 => /usr/lib/aarch64-linux-gnu/libstdc++.so.6 (0x0000007f9a661000)
       libgcc_s.so.1 => /lib/aarch64-linux-gnu/libgcc_s.so.1 (0x0000007f9a63d000)
       libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000007f9a4e4000)
       /lib/ld-linux-aarch64.so.1 (0x0000007f9abd4000)
       libv4lconvert.so.0 => not found
       libdl.so.2 => /lib/aarch64-linux-gnu/libdl.so.2 (0x0000007f9a4cf000)
       libGLdispatch.so.0 => /usr/lib/aarch64-linux-gnu/libGLdispatch.so.0 (0x0000007f9a3a3000)
       libxcb.so.1 => /usr/lib/aarch64-linux-gnu/libxcb.so.1 (0x0000007f9a373000)
       libnvrm.so => /usr/lib/aarch64-linux-gnu/tegra/libnvrm.so (0x0000007f9a331000)
       libnvrm_graphics.so => /usr/lib/aarch64-linux-gnu/tegra/libnvrm_graphics.so (0x0000007f9a312000)
       libnvddk_vic.so => /usr/lib/aarch64-linux-gnu/tegra/libnvddk_vic.so (0x0000007f9a2f4000)
       libnvbuf_fdmap.so.1.0.0 => /usr/lib/aarch64-linux-gnu/tegra/libnvbuf_fdmap.so.1.0.0 (0x0000007f9a2e1000)
       libnvddk_2d_v2.so => /usr/lib/aarch64-linux-gnu/tegra/libnvddk_2d_v2.so (0x0000007f9a2bc000)
       libnvbufsurface.so.1.0.0 => not found
       libnvll.so => /usr/lib/aarch64-linux-gnu/tegra/libnvll.so (0x0000007f9a2a0000)
       libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000007f9a1e6000)
       libXau.so.6 => /usr/lib/aarch64-linux-gnu/libXau.so.6 (0x0000007f9a1d3000)
       libXdmcp.so.6 => /usr/lib/aarch64-linux-gnu/libXdmcp.so.6 (0x0000007f9a1be000)
       libnvos.so => /usr/lib/aarch64-linux-gnu/tegra/libnvos.so (0x0000007f9a1a0000)
       libnvdc.so => /usr/lib/aarch64-linux-gnu/tegra/libnvdc.so (0x0000007f9a181000)
       libbsd.so.0 => /lib/aarch64-linux-gnu/libbsd.so.0 (0x0000007f9a15f000)
       librt.so.1 => /lib/aarch64-linux-gnu/librt.so.1 (0x0000007f9a148000)
       libnvimp.so => /usr/lib/aarch64-linux-gnu/tegra/libnvimp.so (0x0000007f9a133000)

After restore libv4lconvert and libnvbufsurface just like libv4l2, all dependencies are available now.

 nvidia@nvidia-desktop:~/jetson_multimedia_api/samples/00_video_decode$ ldd video_decode
       linux-vdso.so.1 (0x0000007f8bb5c000)
       libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0 (0x0000007f8ba8a000)
       libv4l2.so.0 => /usr/lib/aarch64-linux-gnu/libv4l2.so.0 (0x0000007f8b975000)
       libEGL.so.1 => /usr/lib/aarch64-linux-gnu/libEGL.so.1 (0x0000007f8b954000)
       libGLESv2.so.2 => /usr/lib/aarch64-linux-gnu/libGLESv2.so.2 (0x0000007f8b91e000)
       libX11.so.6 => /usr/lib/aarch64-linux-gnu/libX11.so.6 (0x0000007f8b7f5000)
       libnvbuf_utils.so.1.0.0 => /usr/lib/aarch64-linux-gnu/tegra/libnvbuf_utils.so.1.0.0 (0x0000007f8b7db000)
       libnvjpeg.so => /usr/lib/aarch64-linux-gnu/tegra/libnvjpeg.so (0x0000007f8b783000)
       libdrm.so.2 => /usr/lib/aarch64-linux-gnu/libdrm.so.2 (0x0000007f8b751000)
       libstdc++.so.6 => /usr/lib/aarch64-linux-gnu/libstdc++.so.6 (0x0000007f8b5be000)
       libgcc_s.so.1 => /lib/aarch64-linux-gnu/libgcc_s.so.1 (0x0000007f8b59a000)
       libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000007f8b441000)
       /lib/ld-linux-aarch64.so.1 (0x0000007f8bb31000)
       libv4lconvert.so.0 => /usr/lib/aarch64-linux-gnu/libv4lconvert.so.0 (0x0000007f8b3bc000)
       libdl.so.2 => /lib/aarch64-linux-gnu/libdl.so.2 (0x0000007f8b3a7000)
       libGLdispatch.so.0 => /usr/lib/aarch64-linux-gnu/libGLdispatch.so.0 (0x0000007f8b27b000)
       libxcb.so.1 => /usr/lib/aarch64-linux-gnu/libxcb.so.1 (0x0000007f8b24b000)
       libnvrm.so => /usr/lib/aarch64-linux-gnu/tegra/libnvrm.so (0x0000007f8b209000)
       libnvrm_graphics.so => /usr/lib/aarch64-linux-gnu/tegra/libnvrm_graphics.so (0x0000007f8b1ea000)
       libnvddk_vic.so => /usr/lib/aarch64-linux-gnu/tegra/libnvddk_vic.so (0x0000007f8b1cc000)
       libnvbuf_fdmap.so.1.0.0 => /usr/lib/aarch64-linux-gnu/tegra/libnvbuf_fdmap.so.1.0.0 (0x0000007f8b1b9000)
       libnvddk_2d_v2.so => /usr/lib/aarch64-linux-gnu/tegra/libnvddk_2d_v2.so (0x0000007f8b194000)
       libnvbufsurface.so.1.0.0 => /usr/lib/aarch64-linux-gnu/tegra/libnvbufsurface.so.1.0.0 (0x0000007f8b116000)
       libnvll.so => /usr/lib/aarch64-linux-gnu/tegra/libnvll.so (0x0000007f8b0fa000)
       libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000007f8b040000)
       librt.so.1 => /lib/aarch64-linux-gnu/librt.so.1 (0x0000007f8b029000)
       libXau.so.6 => /usr/lib/aarch64-linux-gnu/libXau.so.6 (0x0000007f8b016000)
       libXdmcp.so.6 => /usr/lib/aarch64-linux-gnu/libXdmcp.so.6 (0x0000007f8b001000)
       libnvos.so => /usr/lib/aarch64-linux-gnu/tegra/libnvos.so (0x0000007f8afe3000)
       libnvdc.so => /usr/lib/aarch64-linux-gnu/tegra/libnvdc.so (0x0000007f8afc4000)
       libbsd.so.0 => /lib/aarch64-linux-gnu/libbsd.so.0 (0x0000007f8afa2000)
       libnvimp.so => /usr/lib/aarch64-linux-gnu/tegra/libnvimp.so (0x0000007f8af8d000)