Difference between revisions of "AoEForEmbeddedLinuxDevEnvEngText"
(→(slide 5)Networking the Block Device Layer) |
(→(slide 4)Issues when using NFS) |
||
Line 59: | Line 59: | ||
* NFS has prove track record(+) and it is convenient to develop under NFS environment, but there are issues ... | * NFS has prove track record(+) and it is convenient to develop under NFS environment, but there are issues ... | ||
− | |||
** Different software configuration in operation (after development) may cause misterious behavior | ** Different software configuration in operation (after development) may cause misterious behavior | ||
− | |||
** Debugging problems attributed to file system is impossible/hard | ** Debugging problems attributed to file system is impossible/hard | ||
Revision as of 02:33, 2 June 2009
[original file is http://elinux.org/upload/3/38/AoEForEmbeddedLinuxDevEnv-CELF200905shigeoka-jp.pdf]
[translated by ikoma]
[see original slide for figures]
CELF Technical Jamboree #27
May, 22, 2009
Copyright c Hitachi Ltd. 2008 All rights reserved.
Contents
- 1 (slide 1)
- 2 (slide 2)Background
- 3 (slide 3)Typical Development Environemnt using NFS
- 4 (slide 4)Issues when using NFS
- 5 (slide 5)Networking the Block Device Layer
- 6 (slide 6)ATA over Ethernet (AoE) Overview
- 7 (slide 7)System Structure Diagram of AoE
- 8 (slide 8)iSCSI Overview
- 9 (slide 9)System Structure Diagram of iSCSI
- 10 (slide 10)Network Block Device (NBD) Overview
- 11 (slide 11)Use of rootfs with each Method
- 12 (slide 12)MTD Emulation
- 13 (slide 13)AoE for Embedded Development Environment
- 14 (slide 14)Evaluation of AoE Use
- 15 (slide 15)Evaluation of AoE Use (Size/Memory Usage)
- 16 (slide 16)Evaluation of AoE Use (Behavior)
- 17 (slide 17)Consideration
- 18 (slide 18)Summary
- 19 (slide 19)References
(slide 1)
CE Linux Forum Japan Technical Jamboree #27 2009/5/22
ATA over Ethernet for Embedded Linux Developmnet Environment
Tomohiko Shigeoka Embedded System Platform Research Laboratory, Hitachi Ltd.
(slide 2)Background
- NFS is typically used to share files (rootfs) between the host and the target in embedded Linux development enviroment
- When burnt onto devices, file systems are built on flash memory
- File system other than extN is often used
- cramfs/squashfs/jffs2/ubifs/ etc.
- Modifying files placed on flash memory is not easy (in many cases)
- Writing with ROM Writer, JTAG-ICE etc. is cumbersome
- If failed, further troublesome
(slide 3)Typical Development Environemnt using NFS
- Developing software on host system using cross tool chain
- A file system on the host is used as the root file system on the target via NFS
- Later written onto flash memory by generating file system image
(Figure)
(slide 4)Issues when using NFS
- NFS has prove track record(+) and it is convenient to develop under NFS environment, but there are issues ...
- Different software configuration in operation (after development) may cause misterious behavior
- Debugging problems attributed to file system is impossible/hard
- (+) Sun publicly announced NFSv2 in 1984; NFS implemented on Linux since kernel circa 1.2
(slide 5)Networking the Block Device Layer
- Network connection in block device layer, not file system layer
- Same file system as on flash memory can be used
- Network connectable block devices:
- ATA over Ethernet (AoE)
- iSCSI (internet SCSI)
- Network Block Device (NBD)
(slide 6)ATA over Ethernet (AoE) Overview
- Low cost SAN technology lead by coraid.com
- Light-weight protocol to communicate by loading ATA command/data on Ethernet frames (does not use TCP/IP)
- Client side driver mainlined since kernel 2.6.11 (drivers/block/aoe/)
- Configuration commands (aoetools) available on client side
- Server side is special hardware or software implementation - userland daemon, kernel driver etc.
- vblade, kvblade, qaoed, etc.
- Server side is not required to be an ATA
- Supported on numbers of OSes and bootloaders
(slide 7)System Structure Diagram of AoE
- vblade daemon on server publishes disk image file on HDD
# vblade 0 0 eth0 image_file
- AoE driver on client presents an image publised on server as a block device to file system layer
# mount -t ext2 /dev/etherd/e0.0 /mnt
(Figure)
(slide 8)iSCSI Overview
- SAN technology gettin popular in enterprise arena
- Protocol to communicate by loading SCSI commands/data on TCP/IP
- Client side driver (initiator) mainlined since kernel 2.6.12 (drivers/scsi/iscsi_tcp.c)
- Block devices can be used with the same interface with ordinaly SCSI devices (/dev/sda etc.)
- On client side, configuration with userland tool (Open-iSCSI) is required
- Server side runs on high-end special hardware as well as various software implementation (iSCSI Enterprise Target, etc.)
(slide 9)System Structure Diagram of iSCSI
- iSCSI Target on server publishes a disk image file or a partition on HDD
- iSCSI Initiator on client connects to the target, and presents the disk image as a SCSI disk block device
(Figure)
(slide 10)Network Block Device (NBD) Overview
- Treats server data via TCP/IP as block device
- Protocol irelevalent to ATA (compression supported)
- Client side driver mainlined since kernel 2.1.101 (drivers/block/nbd.c)
- On client side, connection management daemon (nbd-client) required
# nbd-client server 2000 /dev/nb0
- Server is implemented as software (nbd-server)
# nbd-server 2000 image_file
(Figure)
(slide 11)Use of rootfs with each Method
- Only AoE support the same style as NFS
- AoE root patch required (http://support.coraid.com/support/linux/contrib/mcmullan/aoeroot-2.6.15.diff
- With this patch, AoE server to be root must be specifed in kernel config - Can not be specified in kernel command line (root=???) - Setting with command (aoetools) is not required
- Setting up with command is required for iSCSI or NBD
- Setting up using initramfs/initrd - Direct mounting as rootfs is not possible; must add initramfs/initrd with setting up --> this is why we use AoE here
(slide 12)MTD Emulation
- Typical file systems (ext3/cramfs etc.) assume block devices
- jffs2 etc. assumes MTD, not block devices
- MTD inteface can be used with block2mtd
# insmod block2mtd.ko block2mtd=/dev/hda1 # mount -t jffs2 /dev/mtdblock0 /mnt
(Figure)
(slide 13)AoE for Embedded Development Environment
- Software developed with cross tool chain in the same manner as when NFS used
- File system image must be created to be executed on target
- File system image to be written on flash memory can be directly used as root file system via network with AoE
(Figure)
(slide 14)Evaluation of AoE Use
- Evaluation items (Comparison among AoE/NFS/Flash memory)
- Object size of target kernel - Memory usage at boot with identical rootfs - Comparison of behavior with compression file system (cramfs) -- read time of a highly compressionable file (all zero) and a less compressionable file (random data) of the same data volume
- Evaluation environment
- Target: OSK (TI OMAP 192MHz, RAM 32M), kernel 2.6.20.20, 10BASE-T - Host: x86_64(3.4GHz), kernel 2.4.21(RHEL3), 1000BASE-T
(slide 15)Evaluation of AoE Use (Size/Memory Usage)
- Target kernel object size comparison
+----------------------+---------+---------------------+
| AoE | NFS | Flash memory |
+-----------+----------+---------+-----------+---------+
|file system| aoe | NFS |file system| mtd |
| (cramfs) | | client | (cramfs) | |
+-----------+----------+---------+-----------+---------+
| 10,433 | 28,326 | | 10,433 | 73.401 |
+-----------+----------+ +-----------+---------+
| 38,759 | 322,503 | 83,834 |
+----------------------+---------+---------------------+
size of build-in.o in each directory (bytes)
- Memory usage comparison
- After booting up with an identical rootfs (busybox), logging in and executing free commaand
+-------+--------+--------+--------+---------+
| | total | used | free | buffers |
+-------+--------+--------+--------+---------+
| AoE | 30,220 | 5,000 | 25,220 | 724 |
+-------+--------+--------+--------+---------+
| NFS | 29,888 | 4,352 | 25,536 | 0 |
+-------+--------+--------+--------+---------+
| Flash | 30,188 | 5,040 | 25,148 | 736 |
+-------+--------+--------+--------+---------+
free command output (Kbytes)
(slide 16)Evaluation of AoE Use (Behavior)
- File read time (CPU time)
- "echo 3 > /proc/sys/vm/drop_caches", then read out with "time wc -c"
+-------+--------------------+--------------------+
| | all zero data | random data |
| +------+------+------+------+------+------+
| | real | user | sys | real | user | sys |
+-------+------+------+------+------+------+------+
| AoE | 1.57 | 0.71 | 0.63 | 7.09 | 0.66 | 1.14 |
+-------+------+------+------+------+------+------+
| NFS | 6.28 | 0.72 | 0.88 | 6.39 | 0.72 | 0.86 |
+-------+------+------+------+------+------+------+
| Flash | 1.44 | 0.72 | 0.61 | 3.23 | 0.69 | 0.88 |
+-------+------+------+------+------+------+------+
unit: second (read 5MB; avarage of 10 trials)
- Data transfer volume
- Measured as the difference between /proc/diskstats and /proc/net/dev
+-------+--------------------------+--------------------------+
| | all zero data | random data |
| +------------+-------------+------------+-------------+
| | eth recv | sector read | eth recv | sector read |
+-------+------------+-------------+------------+-------------+
| AoE | 352,040 | 339,968 | 5,770,820 | 5,570,560 |
+-------+------------+-------------+------------+-------------+
| NFS | 5,666,068 | - | 5,663,699 | - |
+-------+------------+-------------+------------+-------------+
| Flash | - | 339,968 | - | 5,570,560 |
+-------+------------+-------------+------------+-------------+
unit: byte (when read 5MB)
(slide 17)Consideration
- Kernel CPU time reflects the difference of compression processing
- Expanding highly compressible data (all zero) does not use CPU as much as random data (Assuming that data transfer does not need so much CPU power; if data transfer needs much CPU power, volume of memory copy would be also reflected)
- In case of NFS, it is independent of data contents as there is no expanding (data transfer volume is also uniform)
- Data transfer volume is reflects the data contens to be transferred
- Transferring compressed data causes highly compressible data is transferred as less volume - NFS does not compress data, so data transfer volume is constant
(slide 18)Summary
- AoE is usable even for embedded Linux systems
- AoE make some debuging impossible with NFS possible
- Object size is smaller with AoE than with NFS
- AoE user-friendliness is still evolving