View source for Jetson/FAQ/BSP/RootFS Reduction

= How to reduce disk usage on Jetson platform =

Jetson platform files system is based on ubuntu sample file system. Some Nvidia
platform related binaries (BSP related) are added to support platform basic function.
Jetson SDK Components including CUDA, cuDNN, TensorRT, OpenCV, VisionWorks, DeepStream
and multimedia API provide AI related support.

This page breaks down Nvidia platform related binaries and Jetson SDK Component, so user
can remove unused files to save space. A command to list all installed ubuntu packages by
size is also provided for user to remove unused package based on needs.

== Nvidia platform related binaries breakdown ==
Below is a breakdown of Nvidia platform related binaries.

  /
  ├── boot (43M)           Image(kernel binary), dtb and extlinux.conf. All are not used by Xavier, Image and extlinux.conf are used by Nano, TX1 and TX2
  ├── etc (536K)           Nvidia specified config file. DO NOT touch
  ├── lib
  │   ├── firmware (4.3M)  Firmwares used by platform. DO NOT touch
  │   ├── modules (47M)    Modules(.ko) used by platform. Can delete some based on needs
  │   └── systemd (32K)    Nvidia specified config file. DO NOT touch
  ├── opt
  │   └── nvidia  (17M)    Jetson gpio python lib and usb device mode service. Can delete based on needs
  ├── usr
  │   ├── bin (356K)       Nvidia speficied tools. DO NOT touch
  │   ├── lib (130M)       Nvidia speficied libs. DO NOT touch
  │   ├── sbin (2M)        Nvidia speficied tools. DO NOT touch
  │   ├── share (2.5M)     Nvidia specified boot logo, license, icon, etc. DO NOT touch
  │   └── src (234M)       Linux header files and grahics demos. Can delete based on needs
  └── var
      └── nvidia (188K)    Placeholder for nvcam. Can delete based on needs

== Jetson SDK Components ==
Based on Jetson OS, Jetson SDK Components, including CUDA, cuDNN, TensorRT, OpenCV, VisionWorks,
MultiMedia API and DeepStream, are installed by sdkmanager(or jetpack tool for older version).
MultiMedia API is installed by tbz2 package while others are installed by deb package.

Both Jetson SDK libs(both static and dynamic libs) and sample code are installed and sample code can be deleted to save space.
And if user's application is dynamicly linked to these libs, all static libs can be deleted. 

After Jetson SDK Components are installed, run below commands can remove all deb files.
  sudo apt clean    # clean debs in /var/cache/apt/archives
  sudo rm /etc/apt/sources.list.d/*        # remove /var/cudaxxxx, /var/visonworksxxxx from apt source list
  sudo rm /var/cuda-repo-10-0-local-10.0.326/ /var/visionworks-repo/ /var/visionworks-sfm-repo/ /var/visionworks-tracking-repo/ -rf     # remove nvidia's debs

Below table list all Jetson SDK Components brief breakdown.
==== CUDA ====
{| class="wikitable"
|-
! Elememt !! Path !! Size
|-
| sample || /usr/local/cuda/samples || 198M
|-
| dynamic lib || /usr/local/cuda/targets/aarch64-linux/lib/*.so* || 723M
|-
| static lib || /usr/local/cuda/targets/aarch64-linux/lib/*.a || 898M
|-
| doc || /usr/local/cuda/doc || 231M
|}

==== cuDNN ====
{| class="wikitable"
|-
!Elememt!!Path!!Size
|-
|sample||/usr/src/cudnn_samples_v7||11M
|-
|dynamic lib||/usr/lib/aarch64-linux-gnu/libcudnn*.so*||367M
|-
|static lib||/usr/lib/aarch64-linux-gnu/libcudnn*.a||358M
|}

==== TensorRT ====
{| class="wikitable"
|-
!Elememt!!Path!!Size
|-
|sample||/usr/src/tensorrt||783M
|-
|dynamic lib||/usr/lib/aarch64-linux-gnu/libnvcaffe_parser*.so*<br/>/usr/lib/aarch64-linux-gnu/libnvinfer*.so*<br/>/usr/lib/aarch64-linux-gnu/libnvonnxparser*.so*<br/>/usr/lib/aarch64-linux-gnu/libnvparsers*.so*||156M
|-
|static lib||/usr/lib/aarch64-linux-gnu/libnvcaffe_parser*.a<br/>/usr/lib/aarch64-linux-gnu/libnvinfer*.a<br/>/usr/lib/aarch64-linux-gnu/libnvonnxparser*.a<br/>/usr/lib/aarch64-linux-gnu/libnvparsers*.a||172M
|-
|doc||/usr/share/doc/libnvinfer*||24K
|}

==== OpenCV ====
{| class="wikitable"
|-
!Elememt!!Path!!Size
|-
|sample||/usr/share/OpenCV||11M
|-
|dynamic lib||/usr/lib/libopencv*.so*||19M
|}

==== VisionWorks ====
{| class="wikitable"
|-
!Elememt!!Path!!Size
|-
|sample||/usr/share/visionworks*<br/>~/VisionWorks-SFM*Samples||131M
|-
|dynamic lib||/usr/lib/libvisionworks*.so*||28M
|-
|doc||/usr/share/doc/libvisionworks*||352K
|}

==== DeepStream ====
{| class="wikitable"
|-
!Elememt!!Path!!Size
|-
|sample||/opt/nvidia/deepstream/deepstream*/samples||205M
|-
|dynamic lib||/opt/nvidia/deepstream/deepstream*/lib||52M
|-
|doc||/opt/nvidia/deepstream/deepstream*/doc||16K
|}

==== MultiMedia API ====
{| class="wikitable"
|-
!Path!!Size
|-
|/usr/src/tegra_multimedia_api||107M
|}

== Remove installed deb packages ==
Run below command can remove all cached .deb file.
  sudo apt clean
Run below command can remove all non-required installed packages.
  sudo apt autoremove
Below command can list all install deb packages by size. User can remove the undesired packages by hand to save space.
  dpkg-query -Wf '${Installed-Size}\t${Package}\n' | sort -rn

= Example =
== Remove all deb installation packages ==
  sudo apt clean
  sudo rm /etc/apt/sources.list.d/*
  sudo rm -rf /var/cuda-repo-10-0-local-10.0.326/ /var/visionworks-repo/ /var/visionworks-sfm-repo/ /var/visionworks-tracking-repo
== Remove ubuntu-desktop and the accessories ==
  sudo apt-get purge gnome-shell
  sudo apt-get purge docker*
  sudo apt-get purge chromium-browser thunderbird fonts-noto-cjk  libreoffice-common  containerd  snapd
  sudo apt-get remove --purge python* libpython*
  sudo apt-get purge libwebkit2gtk-4.0-37  libqt5webkit5  ubuntu-wallpapers-bionic freepats  libreoffice-writer libicu60  libreoffice-calc vim-runtime libflite1  libperl5.26 libmozjs-52-0 humanity-icon-theme samba-libs  iso-codes kwin-data perl-modules-5.26 light-themes libjavascriptcoregtk-4.0-18 chromium-browser-l10n
  sudo apt-get purge libvisionworks libvisionworks-sfm-dev libvisionworks-sfm-repo
  sudo apt-get autoremove
== Remove some source code and sample code ==
  cd /usr/src && sudo rm -rf cudnn_samples_v7 linux-headers-4.9.140-tegra-ubuntu18.04_aarch64 linux-headers-4.9.140-tegra-linux_x86_64  tensorrt nvidia
  rm -rf /home/nvidia/VisionWorks-SFM-0.90-Samples

= How to fix library missing issue after reduction =
Sometimes after removing ubuntu packages, application execution may fail due to depended libraries missing.

For example, after removing packages by above example, Jetson Multimedia API sample cannot run like below.
  nvidia@nvidia-desktop:~/jetson_multimedia_api/samples/00_video_decode$ ./video_decode
  ./video_decode: error while loading shared libraries: libv4l2.so.0: cannot open shared object file: No such file or directory

ldd command shows both libv4l2.so.0 and libnvjpeg.so are missing like below. So we need to copy these libraries from clean platform.
  nvidia@nvidia-desktop:~/jetson_multimedia_api/samples/00_video_decode$ ldd video_decode
        linux-vdso.so.1 (0x0000007f869e8000)
        libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0 (0x0000007f86916000)
        libv4l2.so.0 => not found
        libEGL.so.1 => /usr/lib/aarch64-linux-gnu/libEGL.so.1 (0x0000007f868f5000)
        libGLESv2.so.2 => /usr/lib/aarch64-linux-gnu/libGLESv2.so.2 (0x0000007f868bf000)
        libX11.so.6 => /usr/lib/aarch64-linux-gnu/libX11.so.6 (0x0000007f86796000)
        libnvbuf_utils.so.1.0.0 => /usr/lib/aarch64-linux-gnu/tegra/libnvbuf_utils.so.1.0.0 (0x0000007f8677c000)
        libnvjpeg.so => not found
        libdrm.so.2 => /usr/lib/aarch64-linux-gnu/libdrm.so.2 (0x0000007f8674a000)
        libstdc++.so.6 => /usr/lib/aarch64-linux-gnu/libstdc++.so.6 (0x0000007f865b7000)
        libgcc_s.so.1 => /lib/aarch64-linux-gnu/libgcc_s.so.1 (0x0000007f86593000)
        libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000007f8643a000)
        /lib/ld-linux-aarch64.so.1 (0x0000007f869bd000)
        libGLdispatch.so.0 => /usr/lib/aarch64-linux-gnu/libGLdispatch.so.0 (0x0000007f8630e000)
        libdl.so.2 => /lib/aarch64-linux-gnu/libdl.so.2 (0x0000007f862f9000)
        libxcb.so.1 => /usr/lib/aarch64-linux-gnu/libxcb.so.1 (0x0000007f862c9000)
        libnvrm.so => /usr/lib/aarch64-linux-gnu/tegra/libnvrm.so (0x0000007f86287000)
        libnvrm_graphics.so => /usr/lib/aarch64-linux-gnu/tegra/libnvrm_graphics.so (0x0000007f86268000)
        libnvddk_vic.so => /usr/lib/aarch64-linux-gnu/tegra/libnvddk_vic.so (0x0000007f8624a000)
        libnvbuf_fdmap.so.1.0.0 => /usr/lib/aarch64-linux-gnu/tegra/libnvbuf_fdmap.so.1.0.0 (0x0000007f86237000)
        libnvll.so => /usr/lib/aarch64-linux-gnu/tegra/libnvll.so (0x0000007f8621b000)
        libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000007f86161000)
        libXau.so.6 => /usr/lib/aarch64-linux-gnu/libXau.so.6 (0x0000007f8614e000)
        libXdmcp.so.6 => /usr/lib/aarch64-linux-gnu/libXdmcp.so.6 (0x0000007f86139000)
        libnvos.so => /usr/lib/aarch64-linux-gnu/tegra/libnvos.so (0x0000007f8611b000)
        libnvdc.so => /usr/lib/aarch64-linux-gnu/tegra/libnvdc.so (0x0000007f860fc000)
        libbsd.so.0 => /lib/aarch64-linux-gnu/libbsd.so.0 (0x0000007f860da000)
        librt.so.1 => /lib/aarch64-linux-gnu/librt.so.1 (0x0000007f860c3000)
        libnvimp.so => /usr/lib/aarch64-linux-gnu/tegra/libnvimp.so (0x0000007f860ae000)

On clean platform, we can use "find" and "ls" command to get all the necessary libraries like below.
  nvidia@nvidia-xavier:~$ find /usr/lib/ -name "libv4l2.so*"
  /usr/lib/aarch64-linux-gnu/libv4l2.so.0
  /usr/lib/aarch64-linux-gnu/libv4l2.so
  /usr/lib/aarch64-linux-gnu/libv4l2.so.0.0.0
  /usr/lib/aarch64-linux-gnu/tegra/libv4l2.so.0
  /usr/lib/aarch64-linux-gnu/libv4l2.so.0.0.999999

  nvidia@nvidia-xavier:/usr/lib/aarch64-linux-gnu$ ls -l /usr/lib/aarch64-linux-gnu/libv4l2.so*
  lrwxrwxrwx 1 root root    12 Dec 26 16:15 /usr/lib/aarch64-linux-gnu/libv4l2.so -> libv4l2.so.0
  lrwxrwxrwx 1 root root    21 Dec 19 14:43 /usr/lib/aarch64-linux-gnu/libv4l2.so.0 -> libv4l2.so.0.0.999999
  -rw-r--r-- 1 root root 55424 Feb 10  2018 /usr/lib/aarch64-linux-gnu/libv4l2.so.0.0.0
  lrwxrwxrwx 1 root root    18 Dec 10 15:08 /usr/lib/aarch64-linux-gnu/libv4l2.so.0.0.999999 -> tegra/libnvv4l2.so

After restore libv4l2 and libnvjpeg, remember to run "sudo ldconfig" to update /etc/ld.so.cache

Now, re-run video_decode, it still fails. That's because the new added libv4l2 and libnvjpeg depends on other missing libraries like below:
  nvidia@nvidia-desktop:~/jetson_multimedia_api/samples/00_video_decode$ ldd video_decode
        linux-vdso.so.1 (0x0000007f9abff000)
        libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0 (0x0000007f9ab2d000)
        libv4l2.so.0 => /usr/lib/aarch64-linux-gnu/libv4l2.so.0 (0x0000007f9aa18000)
        libEGL.so.1 => /usr/lib/aarch64-linux-gnu/libEGL.so.1 (0x0000007f9a9f7000)
        libGLESv2.so.2 => /usr/lib/aarch64-linux-gnu/libGLESv2.so.2 (0x0000007f9a9c1000)
        libX11.so.6 => /usr/lib/aarch64-linux-gnu/libX11.so.6 (0x0000007f9a898000)
        libnvbuf_utils.so.1.0.0 => /usr/lib/aarch64-linux-gnu/tegra/libnvbuf_utils.so.1.0.0 (0x0000007f9a87e000)
        libnvjpeg.so => /usr/lib/aarch64-linux-gnu/tegra/libnvjpeg.so (0x0000007f9a826000)
        libdrm.so.2 => /usr/lib/aarch64-linux-gnu/libdrm.so.2 (0x0000007f9a7f4000)
        libstdc++.so.6 => /usr/lib/aarch64-linux-gnu/libstdc++.so.6 (0x0000007f9a661000)
        libgcc_s.so.1 => /lib/aarch64-linux-gnu/libgcc_s.so.1 (0x0000007f9a63d000)
        libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000007f9a4e4000)
        /lib/ld-linux-aarch64.so.1 (0x0000007f9abd4000)
        libv4lconvert.so.0 => not found
        libdl.so.2 => /lib/aarch64-linux-gnu/libdl.so.2 (0x0000007f9a4cf000)
        libGLdispatch.so.0 => /usr/lib/aarch64-linux-gnu/libGLdispatch.so.0 (0x0000007f9a3a3000)
        libxcb.so.1 => /usr/lib/aarch64-linux-gnu/libxcb.so.1 (0x0000007f9a373000)
        libnvrm.so => /usr/lib/aarch64-linux-gnu/tegra/libnvrm.so (0x0000007f9a331000)
        libnvrm_graphics.so => /usr/lib/aarch64-linux-gnu/tegra/libnvrm_graphics.so (0x0000007f9a312000)
        libnvddk_vic.so => /usr/lib/aarch64-linux-gnu/tegra/libnvddk_vic.so (0x0000007f9a2f4000)
        libnvbuf_fdmap.so.1.0.0 => /usr/lib/aarch64-linux-gnu/tegra/libnvbuf_fdmap.so.1.0.0 (0x0000007f9a2e1000)
        libnvddk_2d_v2.so => /usr/lib/aarch64-linux-gnu/tegra/libnvddk_2d_v2.so (0x0000007f9a2bc000)
        libnvbufsurface.so.1.0.0 => not found
        libnvll.so => /usr/lib/aarch64-linux-gnu/tegra/libnvll.so (0x0000007f9a2a0000)
        libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000007f9a1e6000)
        libXau.so.6 => /usr/lib/aarch64-linux-gnu/libXau.so.6 (0x0000007f9a1d3000)
        libXdmcp.so.6 => /usr/lib/aarch64-linux-gnu/libXdmcp.so.6 (0x0000007f9a1be000)
        libnvos.so => /usr/lib/aarch64-linux-gnu/tegra/libnvos.so (0x0000007f9a1a0000)
        libnvdc.so => /usr/lib/aarch64-linux-gnu/tegra/libnvdc.so (0x0000007f9a181000)
        libbsd.so.0 => /lib/aarch64-linux-gnu/libbsd.so.0 (0x0000007f9a15f000)
        librt.so.1 => /lib/aarch64-linux-gnu/librt.so.1 (0x0000007f9a148000)
        libnvimp.so => /usr/lib/aarch64-linux-gnu/tegra/libnvimp.so (0x0000007f9a133000)

After restore libv4lconvert and libnvbufsurface just like libv4l2, all dependencies are available now.
  nvidia@nvidia-desktop:~/jetson_multimedia_api/samples/00_video_decode$ ldd video_decode
        linux-vdso.so.1 (0x0000007f8bb5c000)
        libpthread.so.0 => /lib/aarch64-linux-gnu/libpthread.so.0 (0x0000007f8ba8a000)
        libv4l2.so.0 => /usr/lib/aarch64-linux-gnu/libv4l2.so.0 (0x0000007f8b975000)
        libEGL.so.1 => /usr/lib/aarch64-linux-gnu/libEGL.so.1 (0x0000007f8b954000)
        libGLESv2.so.2 => /usr/lib/aarch64-linux-gnu/libGLESv2.so.2 (0x0000007f8b91e000)
        libX11.so.6 => /usr/lib/aarch64-linux-gnu/libX11.so.6 (0x0000007f8b7f5000)
        libnvbuf_utils.so.1.0.0 => /usr/lib/aarch64-linux-gnu/tegra/libnvbuf_utils.so.1.0.0 (0x0000007f8b7db000)
        libnvjpeg.so => /usr/lib/aarch64-linux-gnu/tegra/libnvjpeg.so (0x0000007f8b783000)
        libdrm.so.2 => /usr/lib/aarch64-linux-gnu/libdrm.so.2 (0x0000007f8b751000)
        libstdc++.so.6 => /usr/lib/aarch64-linux-gnu/libstdc++.so.6 (0x0000007f8b5be000)
        libgcc_s.so.1 => /lib/aarch64-linux-gnu/libgcc_s.so.1 (0x0000007f8b59a000)
        libc.so.6 => /lib/aarch64-linux-gnu/libc.so.6 (0x0000007f8b441000)
        /lib/ld-linux-aarch64.so.1 (0x0000007f8bb31000)
        libv4lconvert.so.0 => /usr/lib/aarch64-linux-gnu/libv4lconvert.so.0 (0x0000007f8b3bc000)
        libdl.so.2 => /lib/aarch64-linux-gnu/libdl.so.2 (0x0000007f8b3a7000)
        libGLdispatch.so.0 => /usr/lib/aarch64-linux-gnu/libGLdispatch.so.0 (0x0000007f8b27b000)
        libxcb.so.1 => /usr/lib/aarch64-linux-gnu/libxcb.so.1 (0x0000007f8b24b000)
        libnvrm.so => /usr/lib/aarch64-linux-gnu/tegra/libnvrm.so (0x0000007f8b209000)
        libnvrm_graphics.so => /usr/lib/aarch64-linux-gnu/tegra/libnvrm_graphics.so (0x0000007f8b1ea000)
        libnvddk_vic.so => /usr/lib/aarch64-linux-gnu/tegra/libnvddk_vic.so (0x0000007f8b1cc000)
        libnvbuf_fdmap.so.1.0.0 => /usr/lib/aarch64-linux-gnu/tegra/libnvbuf_fdmap.so.1.0.0 (0x0000007f8b1b9000)
        libnvddk_2d_v2.so => /usr/lib/aarch64-linux-gnu/tegra/libnvddk_2d_v2.so (0x0000007f8b194000)
        libnvbufsurface.so.1.0.0 => /usr/lib/aarch64-linux-gnu/tegra/libnvbufsurface.so.1.0.0 (0x0000007f8b116000)
        libnvll.so => /usr/lib/aarch64-linux-gnu/tegra/libnvll.so (0x0000007f8b0fa000)
        libm.so.6 => /lib/aarch64-linux-gnu/libm.so.6 (0x0000007f8b040000)
        librt.so.1 => /lib/aarch64-linux-gnu/librt.so.1 (0x0000007f8b029000)
        libXau.so.6 => /usr/lib/aarch64-linux-gnu/libXau.so.6 (0x0000007f8b016000)
        libXdmcp.so.6 => /usr/lib/aarch64-linux-gnu/libXdmcp.so.6 (0x0000007f8b001000)
        libnvos.so => /usr/lib/aarch64-linux-gnu/tegra/libnvos.so (0x0000007f8afe3000)
        libnvdc.so => /usr/lib/aarch64-linux-gnu/tegra/libnvdc.so (0x0000007f8afc4000)
        libbsd.so.0 => /lib/aarch64-linux-gnu/libbsd.so.0 (0x0000007f8afa2000)
        libnvimp.so => /usr/lib/aarch64-linux-gnu/tegra/libnvimp.so (0x0000007f8af8d000)