142 lines · plain
1=========2dm-switch3=========4 5The device-mapper switch target creates a device that supports an6arbitrary mapping of fixed-size regions of I/O across a fixed set of7paths. The path used for any specific region can be switched8dynamically by sending the target a message.9 10It maps I/O to underlying block devices efficiently when there is a large11number of fixed-sized address regions but there is no simple pattern12that would allow for a compact representation of the mapping such as13dm-stripe.14 15Background16----------17 18Dell EqualLogic and some other iSCSI storage arrays use a distributed19frameless architecture. In this architecture, the storage group20consists of a number of distinct storage arrays ("members") each having21independent controllers, disk storage and network adapters. When a LUN22is created it is spread across multiple members. The details of the23spreading are hidden from initiators connected to this storage system.24The storage group exposes a single target discovery portal, no matter25how many members are being used. When iSCSI sessions are created, each26session is connected to an eth port on a single member. Data to a LUN27can be sent on any iSCSI session, and if the blocks being accessed are28stored on another member the I/O will be forwarded as required. This29forwarding is invisible to the initiator. The storage layout is also30dynamic, and the blocks stored on disk may be moved from member to31member as needed to balance the load.32 33This architecture simplifies the management and configuration of both34the storage group and initiators. In a multipathing configuration, it35is possible to set up multiple iSCSI sessions to use multiple network36interfaces on both the host and target to take advantage of the37increased network bandwidth. An initiator could use a simple round38robin algorithm to send I/O across all paths and let the storage array39members forward it as necessary, but there is a performance advantage to40sending data directly to the correct member.41 42A device-mapper table already lets you map different regions of a43device onto different targets. However in this architecture the LUN is44spread with an address region size on the order of 10s of MBs, which45means the resulting table could have more than a million entries and46consume far too much memory.47 48Using this device-mapper switch target we can now build a two-layer49device hierarchy:50 51 Upper Tier - Determine which array member the I/O should be sent to.52 Lower Tier - Load balance amongst paths to a particular member.53 54The lower tier consists of a single dm multipath device for each member.55Each of these multipath devices contains the set of paths directly to56the array member in one priority group, and leverages existing path57selectors to load balance amongst these paths. We also build a58non-preferred priority group containing paths to other array members for59failover reasons.60 61The upper tier consists of a single dm-switch device. This device uses62a bitmap to look up the location of the I/O and choose the appropriate63lower tier device to route the I/O. By using a bitmap we are able to64use 4 bits for each address range in a 16 member group (which is very65large for us). This is a much denser representation than the dm table66b-tree can achieve.67 68Construction Parameters69=======================70 71 <num_paths> <region_size> <num_optional_args> [<optional_args>...] [<dev_path> <offset>]+72 <num_paths>73 The number of paths across which to distribute the I/O.74 75 <region_size>76 The number of 512-byte sectors in a region. Each region can be redirected77 to any of the available paths.78 79 <num_optional_args>80 The number of optional arguments. Currently, no optional arguments81 are supported and so this must be zero.82 83 <dev_path>84 The block device that represents a specific path to the device.85 86 <offset>87 The offset of the start of data on the specific <dev_path> (in units88 of 512-byte sectors). This number is added to the sector number when89 forwarding the request to the specific path. Typically it is zero.90 91Messages92========93 94set_region_mappings <index>:<path_nr> [<index>]:<path_nr> [<index>]:<path_nr>...95 96Modify the region table by specifying which regions are redirected to97which paths.98 99<index>100 The region number (region size was specified in constructor parameters).101 If index is omitted, the next region (previous index + 1) is used.102 Expressed in hexadecimal (WITHOUT any prefix like 0x).103 104<path_nr>105 The path number in the range 0 ... (<num_paths> - 1).106 Expressed in hexadecimal (WITHOUT any prefix like 0x).107 108R<n>,<m>109 This parameter allows repetitive patterns to be loaded quickly. <n> and <m>110 are hexadecimal numbers. The last <n> mappings are repeated in the next <m>111 slots.112 113Status114======115 116No status line is reported.117 118Example119=======120 121Assume that you have volumes vg1/switch0 vg1/switch1 vg1/switch2 with122the same size.123 124Create a switch device with 64kB region size::125 126 dmsetup create switch --table "0 `blockdev --getsz /dev/vg1/switch0`127 switch 3 128 0 /dev/vg1/switch0 0 /dev/vg1/switch1 0 /dev/vg1/switch2 0"128 129Set mappings for the first 7 entries to point to devices switch0, switch1,130switch2, switch0, switch1, switch2, switch1::131 132 dmsetup message switch 0 set_region_mappings 0:0 :1 :2 :0 :1 :2 :1133 134Set repetitive mapping. This command::135 136 dmsetup message switch 0 set_region_mappings 1000:1 :2 R2,10137 138is equivalent to::139 140 dmsetup message switch 0 set_region_mappings 1000:1 :2 :1 :2 :1 :2 :1 :2 \141 :1 :2 :1 :2 :1 :2 :1 :2 :1 :2142