Difference between revisions of "AoEForEmbeddedLinuxDevEnvEngText"
(→(slide 15)Evaluation of AoE Use (Size/Memory Usage)) |
(Add category description) |
||
(3 intermediate revisions by 2 users not shown) | |||
Line 22: | Line 22: | ||
− | ATA over Ethernet for Embedded Linux | + | ATA over Ethernet for Embedded Linux Development Environment |
Tomohiko Shigeoka | Tomohiko Shigeoka | ||
Line 32: | Line 32: | ||
− | * NFS is typically used to share files (rootfs) between the host and the target in embedded Linux development | + | * NFS is typically used to share files (rootfs) between the host and the target in embedded Linux development environment |
* When burnt onto devices, file systems are built on flash memory | * When burnt onto devices, file systems are built on flash memory | ||
Line 43: | Line 43: | ||
[http://www.example.com link title] | [http://www.example.com link title] | ||
− | ==(slide 3)Typical Development | + | ==(slide 3)Typical Development Environment using NFS== |
* Developing software on host system using cross tool chain | * Developing software on host system using cross tool chain | ||
Line 56: | Line 56: | ||
* NFS has prove track record(+) and it is convenient to develop under NFS environment, but there are issues ... | * NFS has prove track record(+) and it is convenient to develop under NFS environment, but there are issues ... | ||
− | ** Different software configuration in operation (after development) may cause | + | ** Different software configuration in operation (after development) may cause mysterious behavior |
** Debugging problems attributed to file system is impossible/hard | ** Debugging problems attributed to file system is impossible/hard | ||
Line 95: | Line 95: | ||
# vblade 0 0 eth0 image_file | # vblade 0 0 eth0 image_file | ||
− | * AoE driver on client presents an image | + | * AoE driver on client presents an image published on server as a block device to file system layer |
# mount -t ext2 /dev/etherd/e0.0 /mnt | # mount -t ext2 /dev/etherd/e0.0 /mnt | ||
Line 104: | Line 104: | ||
==(slide 8)iSCSI Overview== | ==(slide 8)iSCSI Overview== | ||
− | * SAN technology | + | * SAN technology getting popular in enterprise arena |
* Protocol to communicate by loading SCSI commands/data on TCP/IP | * Protocol to communicate by loading SCSI commands/data on TCP/IP | ||
Line 110: | Line 110: | ||
* Client side driver (initiator) mainlined since kernel 2.6.12 (drivers/scsi/iscsi_tcp.c) | * Client side driver (initiator) mainlined since kernel 2.6.12 (drivers/scsi/iscsi_tcp.c) | ||
− | * Block devices can be used with the same interface with | + | * Block devices can be used with the same interface with ordinary SCSI devices (/dev/sda etc.) |
* On client side, configuration with userland tool (Open-iSCSI) is required | * On client side, configuration with userland tool (Open-iSCSI) is required | ||
Line 128: | Line 128: | ||
* Treats server data via TCP/IP as block device | * Treats server data via TCP/IP as block device | ||
− | * Protocol | + | * Protocol irrelevant to ATA (compression supported) |
* Client side driver mainlined since kernel 2.1.101 (drivers/block/nbd.c) | * Client side driver mainlined since kernel 2.1.101 (drivers/block/nbd.c) | ||
Line 145: | Line 145: | ||
* Only AoE support the same style as NFS | * Only AoE support the same style as NFS | ||
** AoE root patch required(http://support.coraid.com/support/linux/contrib/mcmullan/aoeroot-2.6.15.diff) | ** AoE root patch required(http://support.coraid.com/support/linux/contrib/mcmullan/aoeroot-2.6.15.diff) | ||
− | ** With this patch, AoE server to be root must be | + | ** With this patch, AoE server to be root must be specified in kernel config |
** Can not be specified in kernel command line (root=???) | ** Can not be specified in kernel command line (root=???) | ||
** Setting with command (aoetools) is not required | ** Setting with command (aoetools) is not required | ||
Line 160: | Line 160: | ||
* jffs2 etc. assumes MTD, not block devices | * jffs2 etc. assumes MTD, not block devices | ||
− | * MTD | + | * MTD interface can be used with block2mtd |
# insmod block2mtd.ko block2mtd=/dev/hda1 | # insmod block2mtd.ko block2mtd=/dev/hda1 | ||
# mount -t jffs2 /dev/mtdblock0 /mnt | # mount -t jffs2 /dev/mtdblock0 /mnt | ||
Line 184: | Line 184: | ||
** Memory usage at boot with identical rootfs | ** Memory usage at boot with identical rootfs | ||
** Comparison of behavior with compression file system (cramfs) | ** Comparison of behavior with compression file system (cramfs) | ||
− | *** read time of a highly | + | *** read time of a highly compressible file (all zero) and a less compressible file (random data) of the same data volume |
* Evaluation environment | * Evaluation environment | ||
Line 217: | Line 217: | ||
* Memory usage comparison | * Memory usage comparison | ||
− | ** After booting up with an identical rootfs (busybox), logging in and executing free | + | ** After booting up with an identical rootfs (busybox), logging in and executing free command |
+-------+--------+--------+--------+---------+ | +-------+--------+--------+--------+---------+ | ||
Line 242: | Line 242: | ||
* File read time (CPU time) | * File read time (CPU time) | ||
− | + | ** "echo 3 > /proc/sys/vm/drop_caches", then read out with "time wc -c" | |
+-------+--------------------+--------------------+ | +-------+--------------------+--------------------+ | ||
Line 266: | Line 266: | ||
+-------+------+------+------+------+------+------+ | +-------+------+------+------+------+------+------+ | ||
− | + | : unit: second (read 5MB; avarage of 10 trials) | |
* Data transfer volume | * Data transfer volume | ||
− | + | ** Measured as the difference between /proc/diskstats and /proc/net/dev | |
+-------+--------------------------+--------------------------+ | +-------+--------------------------+--------------------------+ | ||
Line 294: | Line 294: | ||
+-------+------------+-------------+------------+-------------+ | +-------+------------+-------------+------------+-------------+ | ||
− | + | : unit: byte (when read 5MB) | |
==(slide 17)Consideration== | ==(slide 17)Consideration== | ||
* Kernel CPU time reflects the difference of compression processing | * Kernel CPU time reflects the difference of compression processing | ||
− | + | ** Expanding highly compressible data (all zero) does not use CPU as much as random data (Assuming that data transfer does not need so much CPU power; if data transfer needs much CPU power, volume of memory copy would be also reflected) | |
− | + | ** In case of NFS, it is independent of data contents as there is no expanding (data transfer volume is also uniform) | |
− | |||
− | |||
− | |||
− | |||
− | |||
+ | * Data transfer volume is reflects the data contents to be transferred | ||
+ | ** Transferring compressed data causes highly compressible data is transferred as less volume | ||
+ | ** NFS does not compress data, so data transfer volume is constant | ||
==(slide 18)Summary== | ==(slide 18)Summary== | ||
Line 312: | Line 310: | ||
* AoE is usable even for embedded Linux systems | * AoE is usable even for embedded Linux systems | ||
− | * AoE make some | + | * AoE make some debugging impossible with NFS possible |
* Object size is smaller with AoE than with NFS | * Object size is smaller with AoE than with NFS | ||
Line 329: | Line 327: | ||
* http://nbd.sourceforge.net/ | * http://nbd.sourceforge.net/ | ||
+ | [[Category:Presentations]] |
Latest revision as of 07:08, 5 May 2011
[original file is http://elinux.org/upload/3/38/AoEForEmbeddedLinuxDevEnv-CELF200905shigeoka-jp.pdf]
[translated by ikoma]
[see original slide for figures]
CELF Technical Jamboree #27
May, 22, 2009
Copyright c Hitachi Ltd. 2008 All rights reserved.
Contents
- 1 (slide 1)
- 2 (slide 2)Background
- 3 (slide 3)Typical Development Environment using NFS
- 4 (slide 4)Issues when using NFS
- 5 (slide 5)Networking the Block Device Layer
- 6 (slide 6)ATA over Ethernet (AoE) Overview
- 7 (slide 7)System Structure Diagram of AoE
- 8 (slide 8)iSCSI Overview
- 9 (slide 9)System Structure Diagram of iSCSI
- 10 (slide 10)Network Block Device (NBD) Overview
- 11 (slide 11)Use of rootfs with each Method
- 12 (slide 12)MTD Emulation
- 13 (slide 13)AoE for Embedded Development Environment
- 14 (slide 14)Evaluation of AoE Use
- 15 (slide 15)Evaluation of AoE Use (Size/Memory Usage)
- 16 (slide 16)Evaluation of AoE Use (Behavior)
- 17 (slide 17)Consideration
- 18 (slide 18)Summary
- 19 (slide 19)References
(slide 1)
CE Linux Forum Japan Technical Jamboree #27 2009/5/22
ATA over Ethernet for Embedded Linux Development Environment
Tomohiko Shigeoka Embedded System Platform Research Laboratory, Hitachi Ltd.
(slide 2)Background
- NFS is typically used to share files (rootfs) between the host and the target in embedded Linux development environment
- When burnt onto devices, file systems are built on flash memory
- File system other than extN is often used
- cramfs/squashfs/jffs2/ubifs/ etc.
- Modifying files placed on flash memory is not easy (in many cases)
- Writing with ROM Writer, JTAG-ICE etc. is cumbersome
- If failed, further troublesome
(slide 3)Typical Development Environment using NFS
- Developing software on host system using cross tool chain
- A file system on the host is used as the root file system on the target via NFS
- Later written onto flash memory by generating file system image
(Figure)
(slide 4)Issues when using NFS
- NFS has prove track record(+) and it is convenient to develop under NFS environment, but there are issues ...
- Different software configuration in operation (after development) may cause mysterious behavior
- Debugging problems attributed to file system is impossible/hard
- (+) Sun publicly announced NFSv2 in 1984; NFS implemented on Linux since kernel circa 1.2
(slide 5)Networking the Block Device Layer
- Network connection in block device layer, not file system layer
- Same file system as on flash memory can be used
- Network connectable block devices:
- ATA over Ethernet (AoE)
- iSCSI (internet SCSI)
- Network Block Device (NBD)
(slide 6)ATA over Ethernet (AoE) Overview
- Low cost SAN technology lead by coraid.com
- Light-weight protocol to communicate by loading ATA command/data on Ethernet frames (does not use TCP/IP)
- Client side driver mainlined since kernel 2.6.11 (drivers/block/aoe/)
- Configuration commands (aoetools) available on client side
- Server side is special hardware or software implementation - userland daemon, kernel driver etc.
- vblade, kvblade, qaoed, etc.
- Server side is not required to be an ATA
- Supported on numbers of OSes and bootloaders
(slide 7)System Structure Diagram of AoE
- vblade daemon on server publishes disk image file on HDD
# vblade 0 0 eth0 image_file
- AoE driver on client presents an image published on server as a block device to file system layer
# mount -t ext2 /dev/etherd/e0.0 /mnt
(Figure)
(slide 8)iSCSI Overview
- SAN technology getting popular in enterprise arena
- Protocol to communicate by loading SCSI commands/data on TCP/IP
- Client side driver (initiator) mainlined since kernel 2.6.12 (drivers/scsi/iscsi_tcp.c)
- Block devices can be used with the same interface with ordinary SCSI devices (/dev/sda etc.)
- On client side, configuration with userland tool (Open-iSCSI) is required
- Server side runs on high-end special hardware as well as various software implementation (iSCSI Enterprise Target, etc.)
(slide 9)System Structure Diagram of iSCSI
- iSCSI Target on server publishes a disk image file or a partition on HDD
- iSCSI Initiator on client connects to the target, and presents the disk image as a SCSI disk block device
(Figure)
(slide 10)Network Block Device (NBD) Overview
- Treats server data via TCP/IP as block device
- Protocol irrelevant to ATA (compression supported)
- Client side driver mainlined since kernel 2.1.101 (drivers/block/nbd.c)
- On client side, connection management daemon (nbd-client) required
# nbd-client server 2000 /dev/nb0
- Server is implemented as software (nbd-server)
# nbd-server 2000 image_file
(Figure)
(slide 11)Use of rootfs with each Method
- Only AoE support the same style as NFS
- AoE root patch required(http://support.coraid.com/support/linux/contrib/mcmullan/aoeroot-2.6.15.diff)
- With this patch, AoE server to be root must be specified in kernel config
- Can not be specified in kernel command line (root=???)
- Setting with command (aoetools) is not required
- Setting up with command is required for iSCSI or NBD
- Setting up using initramfs/initrd
- Direct mounting as rootfs is not possible; must add initramfs/initrd with setting up
- --> this is why we use AoE here
(slide 12)MTD Emulation
- Typical file systems (ext3/cramfs etc.) assume block devices
- jffs2 etc. assumes MTD, not block devices
- MTD interface can be used with block2mtd
# insmod block2mtd.ko block2mtd=/dev/hda1 # mount -t jffs2 /dev/mtdblock0 /mnt
(Figure)
(slide 13)AoE for Embedded Development Environment
- Software developed with cross tool chain in the same manner as when NFS used
- File system image must be created to be executed on target
- File system image to be written on flash memory can be directly used as root file system via network with AoE
(Figure)
(slide 14)Evaluation of AoE Use
- Evaluation items (Comparison among AoE/NFS/Flash memory)
- Object size of target kernel
- Memory usage at boot with identical rootfs
- Comparison of behavior with compression file system (cramfs)
- read time of a highly compressible file (all zero) and a less compressible file (random data) of the same data volume
- Evaluation environment
- Target: OSK (TI OMAP 192MHz, RAM 32M), kernel 2.6.20.20, 10BASE-T
- Host: x86_64(3.4GHz), kernel 2.4.21(RHEL3), 1000BASE-T
(slide 15)Evaluation of AoE Use (Size/Memory Usage)
- Target kernel object size comparison
+----------------------+---------+---------------------+
| AoE | NFS | Flash memory |
+-----------+----------+---------+-----------+---------+
|file system| aoe | NFS |file system| mtd |
| (cramfs) | | client | (cramfs) | |
+-----------+----------+---------+-----------+---------+
| 10,433 | 28,326 | | 10,433 | 73.401 |
+-----------+----------+ +-----------+---------+
| 38,759 | 322,503 | 83,834 |
+----------------------+---------+---------------------+
- size of build-in.o in each directory (bytes)
- Memory usage comparison
- After booting up with an identical rootfs (busybox), logging in and executing free command
+-------+--------+--------+--------+---------+
| | total | used | free | buffers |
+-------+--------+--------+--------+---------+
| AoE | 30,220 | 5,000 | 25,220 | 724 |
+-------+--------+--------+--------+---------+
| NFS | 29,888 | 4,352 | 25,536 | 0 |
+-------+--------+--------+--------+---------+
| Flash | 30,188 | 5,040 | 25,148 | 736 |
+-------+--------+--------+--------+---------+
- free command output (Kbytes)
(slide 16)Evaluation of AoE Use (Behavior)
- File read time (CPU time)
- "echo 3 > /proc/sys/vm/drop_caches", then read out with "time wc -c"
+-------+--------------------+--------------------+
| | all zero data | random data |
| +------+------+------+------+------+------+
| | real | user | sys | real | user | sys |
+-------+------+------+------+------+------+------+
| AoE | 1.57 | 0.71 | 0.63 | 7.09 | 0.66 | 1.14 |
+-------+------+------+------+------+------+------+
| NFS | 6.28 | 0.72 | 0.88 | 6.39 | 0.72 | 0.86 |
+-------+------+------+------+------+------+------+
| Flash | 1.44 | 0.72 | 0.61 | 3.23 | 0.69 | 0.88 |
+-------+------+------+------+------+------+------+
- unit: second (read 5MB; avarage of 10 trials)
- Data transfer volume
- Measured as the difference between /proc/diskstats and /proc/net/dev
+-------+--------------------------+--------------------------+
| | all zero data | random data |
| +------------+-------------+------------+-------------+
| | eth recv | sector read | eth recv | sector read |
+-------+------------+-------------+------------+-------------+
| AoE | 352,040 | 339,968 | 5,770,820 | 5,570,560 |
+-------+------------+-------------+------------+-------------+
| NFS | 5,666,068 | - | 5,663,699 | - |
+-------+------------+-------------+------------+-------------+
| Flash | - | 339,968 | - | 5,570,560 |
+-------+------------+-------------+------------+-------------+
- unit: byte (when read 5MB)
(slide 17)Consideration
- Kernel CPU time reflects the difference of compression processing
- Expanding highly compressible data (all zero) does not use CPU as much as random data (Assuming that data transfer does not need so much CPU power; if data transfer needs much CPU power, volume of memory copy would be also reflected)
- In case of NFS, it is independent of data contents as there is no expanding (data transfer volume is also uniform)
- Data transfer volume is reflects the data contents to be transferred
- Transferring compressed data causes highly compressible data is transferred as less volume
- NFS does not compress data, so data transfer volume is constant
(slide 18)Summary
- AoE is usable even for embedded Linux systems
- AoE make some debugging impossible with NFS possible
- Object size is smaller with AoE than with NFS
- AoE user-friendliness is still evolving