Difference between revisions of "AoEForEmbeddedLinuxDevEnvEngText"

From eLinux.org
Jump to: navigation, search
((slide 6)ATA over Ethernet (AoE) Overview)
(Add category description)
 
(6 intermediate revisions by 2 users not shown)
Line 22: Line 22:
  
  
ATA over Ethernet for Embedded Linux Developmnet Environment
+
ATA over Ethernet for Embedded Linux Development Environment
  
 
Tomohiko Shigeoka
 
Tomohiko Shigeoka
Line 32: Line 32:
  
  
* NFS is typically used to share files (rootfs) between the host and the target in embedded Linux development enviroment
+
* NFS is typically used to share files (rootfs) between the host and the target in embedded Linux development environment
  
 
* When burnt onto devices, file systems are built on flash memory
 
* When burnt onto devices, file systems are built on flash memory
Line 43: Line 43:
 
[http://www.example.com link title]
 
[http://www.example.com link title]
  
==(slide 3)Typical Development Environemnt using NFS==
+
==(slide 3)Typical Development Environment using NFS==
  
 
* Developing software on host system using cross tool chain
 
* Developing software on host system using cross tool chain
Line 56: Line 56:
  
 
* NFS has prove track record(+) and it is convenient to develop under NFS environment, but there are issues ...
 
* NFS has prove track record(+) and it is convenient to develop under NFS environment, but there are issues ...
** Different software configuration in operation (after development) may cause misterious behavior
+
** Different software configuration in operation (after development) may cause mysterious behavior
 
** Debugging problems attributed to file system is impossible/hard
 
** Debugging problems attributed to file system is impossible/hard
  
Line 95: Line 95:
 
   # vblade 0 0 eth0 image_file
 
   # vblade 0 0 eth0 image_file
  
* AoE driver on client presents an image publised on server as a block device to file system layer
+
* AoE driver on client presents an image published on server as a block device to file system layer
  
 
   # mount -t ext2 /dev/etherd/e0.0 /mnt
 
   # mount -t ext2 /dev/etherd/e0.0 /mnt
Line 104: Line 104:
 
==(slide 8)iSCSI Overview==
 
==(slide 8)iSCSI Overview==
  
* SAN technology gettin popular in enterprise arena
+
* SAN technology getting popular in enterprise arena
  
 
* Protocol to communicate by loading SCSI commands/data on TCP/IP
 
* Protocol to communicate by loading SCSI commands/data on TCP/IP
Line 110: Line 110:
 
* Client side driver (initiator) mainlined since kernel 2.6.12 (drivers/scsi/iscsi_tcp.c)
 
* Client side driver (initiator) mainlined since kernel 2.6.12 (drivers/scsi/iscsi_tcp.c)
  
* Block devices can be used with the same interface with ordinaly SCSI devices (/dev/sda etc.)
+
* Block devices can be used with the same interface with ordinary SCSI devices (/dev/sda etc.)
  
 
* On client side, configuration with userland tool (Open-iSCSI) is required
 
* On client side, configuration with userland tool (Open-iSCSI) is required
Line 128: Line 128:
 
* Treats server data via TCP/IP as block device  
 
* Treats server data via TCP/IP as block device  
  
* Protocol irelevalent to ATA (compression supported)
+
* Protocol irrelevant to ATA (compression supported)
  
 
* Client side driver mainlined since kernel 2.1.101 (drivers/block/nbd.c)
 
* Client side driver mainlined since kernel 2.1.101 (drivers/block/nbd.c)
Line 144: Line 144:
  
 
* Only AoE support the same style as NFS
 
* Only AoE support the same style as NFS
  - AoE root patch required
+
** AoE root patch required(http://support.coraid.com/support/linux/contrib/mcmullan/aoeroot-2.6.15.diff)
    (http://support.coraid.com/support/linux/contrib/mcmullan/aoeroot-2.6.15.diff
+
** With this patch, AoE server to be root must be specified in kernel config
 
+
** Can not be specified in kernel command line (root=???)
  - With this patch, AoE server to be root must be specifed in kernel config
+
** Setting with command (aoetools) is not required
  - Can not be specified in kernel command line (root=???)
+
  - Setting with command (aoetools) is not required
+
  
 
* Setting up with command is required for iSCSI or NBD
 
* Setting up with command is required for iSCSI or NBD
  - Setting up using initramfs/initrd
+
** Setting up using initramfs/initrd
  - Direct mounting as rootfs is not possible; must add initramfs/initrd with setting up
+
** Direct mounting as rootfs is not possible; must add initramfs/initrd with setting up
    --> this is why we use AoE here
+
:    --> this is why we use AoE here
  
 
==(slide 12)MTD Emulation==
 
==(slide 12)MTD Emulation==
Line 162: Line 160:
 
* jffs2 etc. assumes MTD, not block devices
 
* jffs2 etc. assumes MTD, not block devices
  
* MTD inteface can be used with block2mtd
+
* MTD interface can be used with block2mtd
 
   # insmod block2mtd.ko block2mtd=/dev/hda1
 
   # insmod block2mtd.ko block2mtd=/dev/hda1
 
   # mount -t jffs2 /dev/mtdblock0 /mnt
 
   # mount -t jffs2 /dev/mtdblock0 /mnt
Line 183: Line 181:
  
 
* Evaluation items (Comparison among AoE/NFS/Flash memory)
 
* Evaluation items (Comparison among AoE/NFS/Flash memory)
  - Object size of target kernel
+
** Object size of target kernel
  - Memory usage at boot with identical rootfs
+
** Memory usage at boot with identical rootfs
  - Comparison of behavior with compression file system (cramfs)
+
** Comparison of behavior with compression file system (cramfs)
    -- read time of a highly compressionable file (all zero) and a less compressionable file (random data) of the same data volume
+
*** read time of a highly compressible file (all zero) and a less compressible file (random data) of the same data volume
  
 
* Evaluation environment
 
* Evaluation environment
  - Target: OSK (TI OMAP 192MHz, RAM 32M), kernel 2.6.20.20, 10BASE-T
+
** Target: OSK (TI OMAP 192MHz, RAM 32M), kernel 2.6.20.20, 10BASE-T
  - Host: x86_64(3.4GHz), kernel 2.4.21(RHEL3), 1000BASE-T
+
** Host: x86_64(3.4GHz), kernel 2.4.21(RHEL3), 1000BASE-T
 
+
  
 
==(slide 15)Evaluation of AoE Use (Size/Memory Usage)==
 
==(slide 15)Evaluation of AoE Use (Size/Memory Usage)==
Line 217: Line 214:
 
+----------------------+---------+---------------------+
 
+----------------------+---------+---------------------+
  
              size of build-in.o in each directory (bytes)
+
:              size of build-in.o in each directory (bytes)
  
 
* Memory usage comparison
 
* Memory usage comparison
 
+
** After booting up with an identical rootfs (busybox), logging in and executing free command
  - After booting up with an identical rootfs (busybox), logging in and executing free commaand
+
  
 
+-------+--------+--------+--------+---------+
 
+-------+--------+--------+--------+---------+
Line 241: Line 237:
 
+-------+--------+--------+--------+---------+
 
+-------+--------+--------+--------+---------+
  
                  free command output (Kbytes)
+
:                  free command output (Kbytes)
 
+
  
 
==(slide 16)Evaluation of AoE Use (Behavior)==
 
==(slide 16)Evaluation of AoE Use (Behavior)==
  
 
* File read time (CPU time)
 
* File read time (CPU time)
  - "echo 3 > /proc/sys/vm/drop_caches", then read out with "time wc -c"
+
** "echo 3 > /proc/sys/vm/drop_caches", then read out with "time wc -c"
  
 
+-------+--------------------+--------------------+
 
+-------+--------------------+--------------------+
Line 271: Line 266:
 
+-------+------+------+------+------+------+------+
 
+-------+------+------+------+------+------+------+
  
        unit: second (read 5MB; avarage of 10 trials)
+
:        unit: second (read 5MB; avarage of 10 trials)
  
  
 
* Data transfer volume
 
* Data transfer volume
  - Measured as the difference between /proc/diskstats and /proc/net/dev
+
** Measured as the difference between /proc/diskstats and /proc/net/dev
  
 
+-------+--------------------------+--------------------------+
 
+-------+--------------------------+--------------------------+
Line 299: Line 294:
 
+-------+------------+-------------+------------+-------------+
 
+-------+------------+-------------+------------+-------------+
  
                                    unit: byte (when read 5MB)
+
:                                    unit: byte (when read 5MB)
  
 
==(slide 17)Consideration==
 
==(slide 17)Consideration==
  
 
* Kernel CPU time reflects the difference of compression processing
 
* Kernel CPU time reflects the difference of compression processing
  - Expanding highly compressible data (all zero) does not use CPU as much as random data (Assuming that data transfer does not need so much CPU power; if data transfer needs much CPU power, volume of memory copy would be also reflected)
+
** Expanding highly compressible data (all zero) does not use CPU as much as random data (Assuming that data transfer does not need so much CPU power; if data transfer needs much CPU power, volume of memory copy would be also reflected)
 
+
** In case of NFS, it is independent of data contents as there is no expanding (data transfer volume is also uniform)
  - In case of NFS, it is independent of data contents as there is no expanding (data transfer volume is also uniform)
+
 
+
* Data transfer volume is reflects the data contens to be transferred
+
  - Transferring compressed data causes highly compressible data is transferred as less volume
+
  - NFS does not compress data, so data transfer volume is constant
+
  
 +
* Data transfer volume is reflects the data contents to be transferred
 +
** Transferring compressed data causes highly compressible data is transferred as less volume
 +
** NFS does not compress data, so data transfer volume is constant
  
 
==(slide 18)Summary==
 
==(slide 18)Summary==
Line 317: Line 310:
 
* AoE is usable even for embedded Linux systems
 
* AoE is usable even for embedded Linux systems
  
* AoE make some debuging impossible with NFS possible
+
* AoE make some debugging impossible with NFS possible
  
 
* Object size is smaller with AoE than with NFS
 
* Object size is smaller with AoE than with NFS
Line 334: Line 327:
  
 
* http://nbd.sourceforge.net/
 
* http://nbd.sourceforge.net/
 +
[[Category:Presentations]]

Latest revision as of 14:08, 5 May 2011

[original file is http://elinux.org/upload/3/38/AoEForEmbeddedLinuxDevEnv-CELF200905shigeoka-jp.pdf]

[translated by ikoma]

[see original slide for figures]


CELF Technical Jamboree #27

May, 22, 2009




Copyright c Hitachi Ltd. 2008 All rights reserved.



(slide 1)

CE Linux Forum Japan Technical Jamboree #27 2009/5/22


ATA over Ethernet for Embedded Linux Development Environment

Tomohiko Shigeoka Embedded System Platform Research Laboratory, Hitachi Ltd.


(slide 2)Background

  • NFS is typically used to share files (rootfs) between the host and the target in embedded Linux development environment
  • When burnt onto devices, file systems are built on flash memory
    • File system other than extN is often used
    • cramfs/squashfs/jffs2/ubifs/ etc.
  • Modifying files placed on flash memory is not easy (in many cases)
    • Writing with ROM Writer, JTAG-ICE etc. is cumbersome
    • If failed, further troublesome

link title

(slide 3)Typical Development Environment using NFS

  • Developing software on host system using cross tool chain
  • A file system on the host is used as the root file system on the target via NFS
  • Later written onto flash memory by generating file system image
              (Figure)

(slide 4)Issues when using NFS

  • NFS has prove track record(+) and it is convenient to develop under NFS environment, but there are issues ...
    • Different software configuration in operation (after development) may cause mysterious behavior
    • Debugging problems attributed to file system is impossible/hard


(+) Sun publicly announced NFSv2 in 1984; NFS implemented on Linux since kernel circa 1.2

(slide 5)Networking the Block Device Layer

  • Network connection in block device layer, not file system layer
    • Same file system as on flash memory can be used
  • Network connectable block devices:
    • ATA over Ethernet (AoE)
    • iSCSI (internet SCSI)
    • Network Block Device (NBD)

(slide 6)ATA over Ethernet (AoE) Overview

  • Low cost SAN technology lead by coraid.com
  • Light-weight protocol to communicate by loading ATA command/data on Ethernet frames (does not use TCP/IP)
  • Client side driver mainlined since kernel 2.6.11 (drivers/block/aoe/)
  • Configuration commands (aoetools) available on client side
  • Server side is special hardware or software implementation - userland daemon, kernel driver etc.
    • vblade, kvblade, qaoed, etc.
  • Server side is not required to be an ATA
  • Supported on numbers of OSes and bootloaders

(slide 7)System Structure Diagram of AoE

  • vblade daemon on server publishes disk image file on HDD
 # vblade 0 0 eth0 image_file
  • AoE driver on client presents an image published on server as a block device to file system layer
 # mount -t ext2 /dev/etherd/e0.0 /mnt


    (Figure)

(slide 8)iSCSI Overview

  • SAN technology getting popular in enterprise arena
  • Protocol to communicate by loading SCSI commands/data on TCP/IP
  • Client side driver (initiator) mainlined since kernel 2.6.12 (drivers/scsi/iscsi_tcp.c)
  • Block devices can be used with the same interface with ordinary SCSI devices (/dev/sda etc.)
  • On client side, configuration with userland tool (Open-iSCSI) is required
  • Server side runs on high-end special hardware as well as various software implementation (iSCSI Enterprise Target, etc.)

(slide 9)System Structure Diagram of iSCSI

  • iSCSI Target on server publishes a disk image file or a partition on HDD
  • iSCSI Initiator on client connects to the target, and presents the disk image as a SCSI disk block device
    (Figure)

(slide 10)Network Block Device (NBD) Overview

  • Treats server data via TCP/IP as block device
  • Protocol irrelevant to ATA (compression supported)
  • Client side driver mainlined since kernel 2.1.101 (drivers/block/nbd.c)
  • On client side, connection management daemon (nbd-client) required
 # nbd-client server 2000 /dev/nb0
  • Server is implemented as software (nbd-server)
 # nbd-server 2000 image_file
   (Figure)


(slide 11)Use of rootfs with each Method

  • Setting up with command is required for iSCSI or NBD
    • Setting up using initramfs/initrd
    • Direct mounting as rootfs is not possible; must add initramfs/initrd with setting up
--> this is why we use AoE here

(slide 12)MTD Emulation

  • Typical file systems (ext3/cramfs etc.) assume block devices
  • jffs2 etc. assumes MTD, not block devices
  • MTD interface can be used with block2mtd
 # insmod block2mtd.ko block2mtd=/dev/hda1
 # mount -t jffs2 /dev/mtdblock0 /mnt
     (Figure)


(slide 13)AoE for Embedded Development Environment

  • Software developed with cross tool chain in the same manner as when NFS used
  • File system image must be created to be executed on target
  • File system image to be written on flash memory can be directly used as root file system via network with AoE
    (Figure)


(slide 14)Evaluation of AoE Use

  • Evaluation items (Comparison among AoE/NFS/Flash memory)
    • Object size of target kernel
    • Memory usage at boot with identical rootfs
    • Comparison of behavior with compression file system (cramfs)
      • read time of a highly compressible file (all zero) and a less compressible file (random data) of the same data volume
  • Evaluation environment
    • Target: OSK (TI OMAP 192MHz, RAM 32M), kernel 2.6.20.20, 10BASE-T
    • Host: x86_64(3.4GHz), kernel 2.4.21(RHEL3), 1000BASE-T

(slide 15)Evaluation of AoE Use (Size/Memory Usage)

  • Target kernel object size comparison

+----------------------+---------+---------------------+

| AoE | NFS | Flash memory |

+-----------+----------+---------+-----------+---------+

|file system| aoe | NFS |file system| mtd |

| (cramfs) | | client | (cramfs) | |

+-----------+----------+---------+-----------+---------+

| 10,433 | 28,326 | | 10,433 | 73.401 |

+-----------+----------+ +-----------+---------+

| 38,759 | 322,503 | 83,834 |

+----------------------+---------+---------------------+

size of build-in.o in each directory (bytes)
  • Memory usage comparison
    • After booting up with an identical rootfs (busybox), logging in and executing free command

+-------+--------+--------+--------+---------+

| | total | used | free | buffers |

+-------+--------+--------+--------+---------+

| AoE | 30,220 | 5,000 | 25,220 | 724 |

+-------+--------+--------+--------+---------+

| NFS | 29,888 | 4,352 | 25,536 | 0 |

+-------+--------+--------+--------+---------+

| Flash | 30,188 | 5,040 | 25,148 | 736 |

+-------+--------+--------+--------+---------+

free command output (Kbytes)

(slide 16)Evaluation of AoE Use (Behavior)

  • File read time (CPU time)
    • "echo 3 > /proc/sys/vm/drop_caches", then read out with "time wc -c"

+-------+--------------------+--------------------+

| | all zero data | random data |

| +------+------+------+------+------+------+

| | real | user | sys | real | user | sys |

+-------+------+------+------+------+------+------+

| AoE | 1.57 | 0.71 | 0.63 | 7.09 | 0.66 | 1.14 |

+-------+------+------+------+------+------+------+

| NFS | 6.28 | 0.72 | 0.88 | 6.39 | 0.72 | 0.86 |

+-------+------+------+------+------+------+------+

| Flash | 1.44 | 0.72 | 0.61 | 3.23 | 0.69 | 0.88 |

+-------+------+------+------+------+------+------+

unit: second (read 5MB; avarage of 10 trials)


  • Data transfer volume
    • Measured as the difference between /proc/diskstats and /proc/net/dev

+-------+--------------------------+--------------------------+

| | all zero data | random data |

| +------------+-------------+------------+-------------+

| | eth recv | sector read | eth recv | sector read |

+-------+------------+-------------+------------+-------------+

| AoE | 352,040 | 339,968 | 5,770,820 | 5,570,560 |

+-------+------------+-------------+------------+-------------+

| NFS | 5,666,068 | - | 5,663,699 | - |

+-------+------------+-------------+------------+-------------+

| Flash | - | 339,968 | - | 5,570,560 |

+-------+------------+-------------+------------+-------------+

unit: byte (when read 5MB)

(slide 17)Consideration

  • Kernel CPU time reflects the difference of compression processing
    • Expanding highly compressible data (all zero) does not use CPU as much as random data (Assuming that data transfer does not need so much CPU power; if data transfer needs much CPU power, volume of memory copy would be also reflected)
    • In case of NFS, it is independent of data contents as there is no expanding (data transfer volume is also uniform)
  • Data transfer volume is reflects the data contents to be transferred
    • Transferring compressed data causes highly compressible data is transferred as less volume
    • NFS does not compress data, so data transfer volume is constant

(slide 18)Summary

  • AoE is usable even for embedded Linux systems
  • AoE make some debugging impossible with NFS possible
  • Object size is smaller with AoE than with NFS
  • AoE user-friendliness is still evolving

(slide 19)References