Difference between revisions of "Jetson/General debug"

From eLinux.org
Jump to: navigation, search
 
(28 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 +
Currently this page is applicable to release <= rel-35.3.1 <br>
 +
==Before start==
 +
Please be aware that below sections are all targeting NV developer kits (devkit).  You have to make sure if your board is NV devkit or custom board.<br>
 +
For any board that is not NV devkit, it is a custom board. For custom board, please contact the board vendor for debug tips (e.g. which port/pin to dump log).<br>
 +
 
==Get prepared==
 
==Get prepared==
 
This section is to share some debug tips for users to understand the basic flow of how to use/debug jetson platform. <br>
 
This section is to share some debug tips for users to understand the basic flow of how to use/debug jetson platform. <br>
Before we start to describe each kind of error, you should check below pages to understand how to dump a serial log from UART.  
+
Before starting to describe each kind of error, you should check below pages to understand how to dump a serial log from UART.  Please note that only UART console can dump the bootloader logs.  If your board hangs in somewhere prior to the kernel, then you must dump the log from UART.
 
  Tutorials:
 
  Tutorials:
   https://www.jetsonhacks.com/2017/03/24/serial-console-nvidia-jetson-tx2/  
+
   [TX1/TX2]    https://www.jetsonhacks.com/2017/03/24/serial-console-nvidia-jetson-tx2/  
   https://www.jetsonhacks.com/2019/04/19/jetson-nano-serial-console/
+
  [Nano/NX]   https://www.jetsonhacks.com/2019/04/19/jetson-nano-serial-console/
   https://developer.ridgerun.com/wiki/index.php?title=Xavier/In_Board/Getting_in_Board/Serial_Console
+
   [Xavier AGX] https://developer.ridgerun.com/wiki/index.php?title=Xavier/In_Board/Getting_in_Board/Serial_Console
 +
  [Orin AGX]  https://developer.ridgerun.com/wiki/index.php/NVIDIA_Jetson_Orin/In_Board/Getting_in_Board/Serial_Console
  
Also, we put all the official document in the dlc here:
+
Please be aware that only the "micro usb port" on jetson Xaiver/Orin AGX devkit can dump the serial log from uart. The usb ports that for flashing the device can only dump kernel log.<br>
  https://developer.nvidia.com/embedded/downloads
+
 
 +
Also, we put all the official document here:
 +
  https://developer.nvidia.com/embedded/downloads   -> This will have hardware related docuemnt
 +
https://developer.nvidia.com/embedded/jetson-linux-archive -> This will have software document for each software release
 +
<br>
 +
And if you want some help or report issue, please file a topic on forum.
 +
https://forums.developer.nvidia.com/
 +
 
 +
When reporting issue, you may tell your problem with below details.
 +
1. The software release version you are using (e.g. which jetpack version).
 +
2. Is this issue reproducible on developer kit(devkit) or custom board?
 +
3. How to reproduce this issue? It would be good if a sample code is attached.
 +
4. What peripherals are connected on device?
 +
5. The log from uart and dmesg
  
Currently this page is applicable to L4T- rel-32.3 and rel-32.4 <br>
 
 
==Flash problem==
 
==Flash problem==
 +
===SD card image===
 +
Sdcard image is not a true method to reflash the board. For anyone who hits boot up issue and cannot be resolved by updating sdcard image. Please directly try sdkmanager.<br>
 +
The reason that sdcard image does not work is mostly because the bootloader software on the jetson module is corrupted. These software are on the QSPI flash of the module. <br>
 +
If you just keep changing the sdcard, it won't have any effect to your QSPI flash. Which means it cannot save the corrupted software back. Only sdkmanger can reflash it. <br>
 
===SDKM===
 
===SDKM===
SDKManager is a GUI tool to replace old Jetpack.  This tool has two basic functions: '''flash the board(driver package/BSP)''' and '''install the SDK(cuda/tensorRT..etc.)'''.
+
SDK Manager is a GUI tool for installing L4T and other JetPack components to '''NV devkit'''.  This tool has two basic functions: '''flash the Jetson module''' (driver package/BSP) and '''install SDKs''' (CUDA, TensorRT, etc.).
 
<br>
 
<br>
 
https://developer.nvidia.com/nvidia-sdk-manager
 
https://developer.nvidia.com/nvidia-sdk-manager
 
<br>
 
<br>
To run sdkm, you have to prepare a x86 host with '''ubuntu 16.04''' or '''18.04'''.   
+
To run SDKM, you have to prepare a x86 host with '''Ubuntu 18.04 (jetpack4/jetpack5)''' (or '''16.04 for jetpack4 only''' or '''20.04 for jetpack5 only''').   
Some users asked whehter it supports ARM version so that they can flash jetson by another ARM device.<br>
+
Some users asked whether SDKM can be installed on ARM version of Ubuntu so that they can flash Jetson from another ARM device.<br>
Currently, it is not supported.  
+
Currently, this is not supported.  
 
<br>
 
<br>
Flash and install SDK could be separateIt means you could flash the board and skip the installation of SDK and vice versa.   
+
Flashing and SDK installation can be done separately with SDKMYou could flash the board and skip the installation of SDKs, or vice versa.   
 
<br>
 
<br>
 
<br>
 
<br>
*How to debug if flashing process is stuck in sdkm?
+
*How to debug if flashing process is stuck in SDKM?
In general, the flash process would finish in 10 minutes.  If it lasts longer than this, the flash process probably has problem.
+
In general, the flash process should finish in '''10 minutes'''.  If it lasts longer than this, the flash process probably has problem.
To debug flash, I would suggest to use flash.sh directly instead of SDKM GUI tool.
+
To debug flash, I would suggest to use flash.sh or l4t_initrd_flash.sh directly instead of SDKM. (SDKM uses flash.sh or initrd flash tool in the background.) <br>
  '''flash.sh'''
+
If your jetson module has internal emmc or sdcard slot, then it can use flash.sh to flash <br>
  This is a official tool to flash jetson platform. This tool is also installed and used by SDKM.  
+
   
 +
'''flash.sh'''
 +
This is the official tool to flash jetson modules. This tool is also installed and used by SDKM.  
 
  After you choose the flash button in sdkm, it would download a package "Linux_for_Tegra" on your host.  The flash.sh is inside this folder.
 
  After you choose the flash button in sdkm, it would download a package "Linux_for_Tegra" on your host.  The flash.sh is inside this folder.
 
  For more detail about how to use flash.sh, please refer to Jetson quick start guide -> https://developer.nvidia.com/embedded/downloads
 
  For more detail about how to use flash.sh, please refer to Jetson quick start guide -> https://developer.nvidia.com/embedded/downloads
 +
 +
If you wants to flash external nvme or usb drive, then you need to use initrd flash to flash. <br>
 +
'''l4t_initrd_flash.sh'''
 +
This is the official tool to flash jetson module. The difference between this tool and flash.sh is initrd flash tool can flash external drive while flash.sh '''cannot'''.
 +
To know how to use initrd flash tool, you can refer to the README file in your BSP direcotry: Linux_for_Tegra/tools/kernel_flash/README_initrd_flash.txt
 +
Or check the example of how to flash nvme/usb drive on Orin NX/Nano from document: https://docs.nvidia.com/jetson/archives/r35.3.1/DeveloperGuide/text/IN/QuickStart.html#to-flash-the-jetson-developer-kit-operating-software
  
 
*What log should I prepare for flash problem?
 
*What log should I prepare for flash problem?
 
To debug flash problem. You need two parts of log for forum admin to check.
 
To debug flash problem. You need two parts of log for forum admin to check.
  '''1. The result of flash.sh '''  
+
  '''1. The result of flash.sh and initrd flash '''  
   This log should be seen after you run above flash.sh
+
   This log should be seen after you run above flash.sh or l4t_initrd_flash.sh
 
  '''2. The result from device'''  
 
  '''2. The result from device'''  
   During the flash, the host would send some binary to device and use it to flash. It is a protocol over usb. To dump this log, you need to enable serial console on UART as above tutorial.
+
   During the flash, the host would send some binary to target device and use this binary to flash. It is a protocol over usb. To dump this log, you need to enable serial console on UART as above tutorial (methods in the top section)
 
  You shall see the log start to dump in the meantime when flash.sh starts to flash your board.
 
  You shall see the log start to dump in the meantime when flash.sh starts to flash your board.
  
Line 43: Line 73:
 
  The SDK installation could be done through ethernet/wifi or usb device mode. You could just ping or ssh before sdk installation starts.
 
  The SDK installation could be done through ethernet/wifi or usb device mode. You could just ping or ssh before sdk installation starts.
 
  USB device mode is a virtual internet interface created when you connect micro usb cable to jetson.  It would use a IP 192.168.55.x as device IP and create another 192.168.55.100 on your host.
 
  USB device mode is a virtual internet interface created when you connect micro usb cable to jetson.  It would use a IP 192.168.55.x as device IP and create another 192.168.55.100 on your host.
  However, this method requires your host to have driver support.  Some users' host may not support it.  In this case, please use wired ethernet to install.
+
  However, this method requires your host to have driver support.  Some users' host may not support it.  In this case, please use wired ethernet to install.
 +
 
 
==Boot==
 
==Boot==
 
After your flash process finishes, the system would boot up no matter you install sdk or not.
 
After your flash process finishes, the system would boot up no matter you install sdk or not.
Line 56: Line 87:
 
   For kernel log, before dumping it, you need to make sure there is no "'''quiet'''" in kernel cmdline. If you see this, it would disable the boot log.
 
   For kernel log, before dumping it, you need to make sure there is no "'''quiet'''" in kernel cmdline. If you see this, it would disable the boot log.
 
   The way to remove this keyword is check the '''extlinux.conf''' under "'''Linux_for_Tegra/rootfs/boot/extlinux'''"on your host.  If you don't know how to tell whether it is in kernel cmdline. Just see the length of your kernel log.  The full log should have usb/pcie init log which are available on all kinds of jetson platforms.  If you don't see them, then your log is in silent mode.
 
   The way to remove this keyword is check the '''extlinux.conf''' under "'''Linux_for_Tegra/rootfs/boot/extlinux'''"on your host.  If you don't know how to tell whether it is in kernel cmdline. Just see the length of your kernel log.  The full log should have usb/pcie init log which are available on all kinds of jetson platforms.  If you don't see them, then your log is in silent mode.
 +
 +
*I see some logs are printed on the screen and there is no desktop for me to operate'''
 +
Such issue is somehow very common on forum and the root cause may differ case by case. For example, such error may happen even when running out of disk space.
 +
When such case happens, enable the uart console and make sure your board is still booting up.  Try to backup the system under such case and file the topic for forum moderator.
  
 
==Board name==
 
==Board name==
 
Some users may get confused by the board ID within Jetpack. Here is the list to check.
 
Some users may get confused by the board ID within Jetpack. Here is the list to check.
   P2180  -> Jetson TX1
+
   P2180  -> Jetson TX1 (T210 SoC)
   P3310  -> Jetson TX2
+
   P3310  -> Jetson TX2 (T186 SoC)
   P3489  -> Jetson TX2i
+
   P3489  -> Jetson TX2i/TX2-4GB (T186 SoC)
   P3448  -> Jetson Nano devkit
+
   P3448  -> Jetson Nano devkit (T210 SoC)
   P3448-0020 -> Jetson Nano production module
+
   P3448-0020 -> Jetson Nano production module (T210 SoC)
   P2888 -> Jetson Xavier
+
   P2888 -> Jetson Xavier (T194 SoC)
   P2888-0060 -> Jetson Xavier-8GB
+
   P2888-0060 -> Jetson Xavier-8GB (T194 SoC)
 +
  P3636 -> Jetson TX2-NX (T186 SoC)
 +
  P3668 -> Jetson Xavier NX (T194 SoC)
 +
  P3701 -> Jetson AGX Orin (T234 SoC)
 +
  P3767 -> Jetson Orin NX/Nano (T234 SoC)
 +
 
 
When you run "sudo ./flash.sh <BoardID> mmcblk0p1" command to flash your board. It is actually searching the config file <boardID>.conf under Linux_for_Tegra.<br>
 
When you run "sudo ./flash.sh <BoardID> mmcblk0p1" command to flash your board. It is actually searching the config file <boardID>.conf under Linux_for_Tegra.<br>
 
For example,
 
For example,
Line 85: Line 125:
 
  nvidia@nvidia-desktop:/proc/device-tree$ xxd i2c@7000c000/status  
 
  nvidia@nvidia-desktop:/proc/device-tree$ xxd i2c@7000c000/status  
 
  00000000: 6f6b 6179 00                            okay.
 
  00000000: 6f6b 6179 00                            okay.
 +
 +
3. How to check if the pinmux file I modified gets flashed to the board?
 +
<br> Ans: You can read the flash log when the flash process starts. The log will tell which pinmux cfg/dts gets in use

Latest revision as of 00:24, 27 April 2023

Currently this page is applicable to release <= rel-35.3.1

Before start

Please be aware that below sections are all targeting NV developer kits (devkit). You have to make sure if your board is NV devkit or custom board.
For any board that is not NV devkit, it is a custom board. For custom board, please contact the board vendor for debug tips (e.g. which port/pin to dump log).

Get prepared

This section is to share some debug tips for users to understand the basic flow of how to use/debug jetson platform.
Before starting to describe each kind of error, you should check below pages to understand how to dump a serial log from UART. Please note that only UART console can dump the bootloader logs. If your board hangs in somewhere prior to the kernel, then you must dump the log from UART.

Tutorials:
 [TX1/TX2]    https://www.jetsonhacks.com/2017/03/24/serial-console-nvidia-jetson-tx2/ 
 [Nano/NX]   https://www.jetsonhacks.com/2019/04/19/jetson-nano-serial-console/
 [Xavier AGX] https://developer.ridgerun.com/wiki/index.php?title=Xavier/In_Board/Getting_in_Board/Serial_Console
 [Orin AGX]   https://developer.ridgerun.com/wiki/index.php/NVIDIA_Jetson_Orin/In_Board/Getting_in_Board/Serial_Console

Please be aware that only the "micro usb port" on jetson Xaiver/Orin AGX devkit can dump the serial log from uart. The usb ports that for flashing the device can only dump kernel log.

Also, we put all the official document here:

https://developer.nvidia.com/embedded/downloads   -> This will have hardware related docuemnt
https://developer.nvidia.com/embedded/jetson-linux-archive -> This will have software document for each software release


And if you want some help or report issue, please file a topic on forum.

https://forums.developer.nvidia.com/

When reporting issue, you may tell your problem with below details.

1. The software release version you are using (e.g. which jetpack version).
2. Is this issue reproducible on developer kit(devkit) or custom board?
3. How to reproduce this issue? It would be good if a sample code is attached.
4. What peripherals are connected on device?
5. The log from uart and dmesg

Flash problem

SD card image

Sdcard image is not a true method to reflash the board. For anyone who hits boot up issue and cannot be resolved by updating sdcard image. Please directly try sdkmanager.
The reason that sdcard image does not work is mostly because the bootloader software on the jetson module is corrupted. These software are on the QSPI flash of the module.
If you just keep changing the sdcard, it won't have any effect to your QSPI flash. Which means it cannot save the corrupted software back. Only sdkmanger can reflash it.

SDKM

SDK Manager is a GUI tool for installing L4T and other JetPack components to NV devkit. This tool has two basic functions: flash the Jetson module (driver package/BSP) and install SDKs (CUDA, TensorRT, etc.).
https://developer.nvidia.com/nvidia-sdk-manager
To run SDKM, you have to prepare a x86 host with Ubuntu 18.04 (jetpack4/jetpack5) (or 16.04 for jetpack4 only or 20.04 for jetpack5 only). Some users asked whether SDKM can be installed on ARM version of Ubuntu so that they can flash Jetson from another ARM device.
Currently, this is not supported.
Flashing and SDK installation can be done separately with SDKM. You could flash the board and skip the installation of SDKs, or vice versa.

  • How to debug if flashing process is stuck in SDKM?

In general, the flash process should finish in 10 minutes. If it lasts longer than this, the flash process probably has problem. To debug flash, I would suggest to use flash.sh or l4t_initrd_flash.sh directly instead of SDKM. (SDKM uses flash.sh or initrd flash tool in the background.)
If your jetson module has internal emmc or sdcard slot, then it can use flash.sh to flash

flash.sh

This is the official tool to flash jetson modules. This tool is also installed and used by SDKM. 
After you choose the flash button in sdkm, it would download a package "Linux_for_Tegra" on your host.   The flash.sh is inside this folder.
For more detail about how to use flash.sh, please refer to Jetson quick start guide -> https://developer.nvidia.com/embedded/downloads

If you wants to flash external nvme or usb drive, then you need to use initrd flash to flash.
l4t_initrd_flash.sh

This is the official tool to flash jetson module. The difference between this tool and flash.sh is initrd flash tool can flash external drive while flash.sh cannot.
To know how to use initrd flash tool, you can refer to the README file in your BSP direcotry: Linux_for_Tegra/tools/kernel_flash/README_initrd_flash.txt
Or check the example of how to flash nvme/usb drive on Orin NX/Nano from document: https://docs.nvidia.com/jetson/archives/r35.3.1/DeveloperGuide/text/IN/QuickStart.html#to-flash-the-jetson-developer-kit-operating-software
  • What log should I prepare for flash problem?

To debug flash problem. You need two parts of log for forum admin to check.

1. The result of flash.sh and initrd flash  
 This log should be seen after you run above flash.sh or l4t_initrd_flash.sh
2. The result from device 
 During the flash, the host would send some binary to target device and use this binary to flash. It is a protocol over usb. To dump this log, you need to enable serial console on UART as above tutorial (methods in the top section)
You shall see the log start to dump in the meantime when flash.sh starts to flash your board.
  • The installation of SDK is stuck, why?
The SDK installation could be done through ethernet/wifi or usb device mode. You could just ping or ssh before sdk installation starts.
USB device mode is a virtual internet interface created when you connect micro usb cable to jetson.  It would use a IP 192.168.55.x as device IP and create another 192.168.55.100 on your host.
However, this method requires your host to have driver support.  Some users' host may not support it.  In this case, please use wired ethernet to install.

Boot

After your flash process finishes, the system would boot up no matter you install sdk or not.

  • My system gets hang and doesn't respond. How to resolve?
There are lots of reason that might cause system hang.  Here are some common cases.
1. Need to configure user account
 This is a new feature after rel-32.1.  User needs to configure their own ubuntu account with USB keyboard and HDMI monitor.
If you don't have HDMI monitor for it, there is also alternative method- oem config to help.  You could refer the detail in L4T dev guide: https://docs.nvidia.com/jetson/l4t/index.html
2. There is bootloader failure or kernel failure
 To dump the log, you need to enable the serial console on UART.  This is the only way to debug bootloader log.
 For kernel log, before dumping it, you need to make sure there is no "quiet" in kernel cmdline. If you see this, it would disable the boot log.
 The way to remove this keyword is check the extlinux.conf under "Linux_for_Tegra/rootfs/boot/extlinux"on your host.  If you don't know how to tell whether it is in kernel cmdline. Just see the length of your kernel log.  The full log should have usb/pcie init log which are available on all kinds of jetson platforms.  If you don't see them, then your log is in silent mode.

  • I see some logs are printed on the screen and there is no desktop for me to operate
Such issue is somehow very common on forum and the root cause may differ case by case. For example, such error may happen even when running out of disk space.
When such case happens, enable the uart console and make sure your board is still booting up.  Try to backup the system under such case and file the topic for forum moderator.

Board name

Some users may get confused by the board ID within Jetpack. Here is the list to check.

 P2180  -> Jetson TX1 (T210 SoC)
 P3310  -> Jetson TX2 (T186 SoC)
 P3489  -> Jetson TX2i/TX2-4GB (T186 SoC)
 P3448  -> Jetson Nano devkit (T210 SoC)
 P3448-0020 -> Jetson Nano production module (T210 SoC)
 P2888 -> Jetson Xavier (T194 SoC)
 P2888-0060 -> Jetson Xavier-8GB (T194 SoC)
 P3636 -> Jetson TX2-NX (T186 SoC)
 P3668 -> Jetson Xavier NX (T194 SoC)
 P3701 -> Jetson AGX Orin (T234 SoC)
 P3767 -> Jetson Orin NX/Nano (T234 SoC)

When you run "sudo ./flash.sh <BoardID> mmcblk0p1" command to flash your board. It is actually searching the config file <boardID>.conf under Linux_for_Tegra.
For example,

sudo ./flash.sh jetson-tx2i mmcblk0p1 

This one would try to check the file in jetson-tx2i.conf. It means you could also create your own board name and it would work if the format in this config file is correct.

Device tree debug tips

1. There are lots of dtb file in source. Which one should I modify?
Ans: If you already flashed your board and it can boot up, please try "dmesg |grep dts". This will tell the name of dts.

nvidia@nvidia-desktop:~$ dmesg |grep dts
DTS File Name: /dvs/git/dirty/git-master_linux/kernel/kernel-4.9/arch/arm64/boot/dts/../../../../../../hardware/nvidia/platform/t210/porg/kernel-dts/tegra210-p3448-0000-p3449-0000-b00.dts

2. How to check if my change/patch take effect in dts?
Ans: You can check each property under /proc/device-tree/. For example, if you want to check the status of i2c@7000c000, then

nvidia@nvidia-desktop:/proc/device-tree$ xxd i2c@7000c000/status 
00000000: 6f6b 6179 00                             okay.

3. How to check if the pinmux file I modified gets flashed to the board?
Ans: You can read the flash log when the flash process starts. The log will tell which pinmux cfg/dts gets in use