386 lines · plain
1==========2MD Cluster3==========4 5The cluster MD is a shared-device RAID for a cluster, it supports6two levels: raid1 and raid10 (limited support).7 8 91. On-disk format10=================11 12Separate write-intent-bitmaps are used for each cluster node.13The bitmaps record all writes that may have been started on that node,14and may not yet have finished. The on-disk layout is::15 16 0 4k 8k 12k17 -------------------------------------------------------------------18 | idle | md super | bm super [0] + bits |19 | bm bits[0, contd] | bm super[1] + bits | bm bits[1, contd] |20 | bm super[2] + bits | bm bits [2, contd] | bm super[3] + bits |21 | bm bits [3, contd] | | |22 23During "normal" functioning we assume the filesystem ensures that only24one node writes to any given block at a time, so a write request will25 26 - set the appropriate bit (if not already set)27 - commit the write to all mirrors28 - schedule the bit to be cleared after a timeout.29 30Reads are just handled normally. It is up to the filesystem to ensure31one node doesn't read from a location where another node (or the same32node) is writing.33 34 352. DLM Locks for management36===========================37 38There are three groups of locks for managing the device:39 402.1 Bitmap lock resource (bm_lockres)41-------------------------------------42 43 The bm_lockres protects individual node bitmaps. They are named in44 the form bitmap000 for node 1, bitmap001 for node 2 and so on. When a45 node joins the cluster, it acquires the lock in PW mode and it stays46 so during the lifetime the node is part of the cluster. The lock47 resource number is based on the slot number returned by the DLM48 subsystem. Since DLM starts node count from one and bitmap slots49 start from zero, one is subtracted from the DLM slot number to arrive50 at the bitmap slot number.51 52 The LVB of the bitmap lock for a particular node records the range53 of sectors that are being re-synced by that node. No other54 node may write to those sectors. This is used when a new nodes55 joins the cluster.56 572.2 Message passing locks58-------------------------59 60 Each node has to communicate with other nodes when starting or ending61 resync, and for metadata superblock updates. This communication is62 managed through three locks: "token", "message", and "ack", together63 with the Lock Value Block (LVB) of one of the "message" lock.64 652.3 new-device management66-------------------------67 68 A single lock: "no-new-dev" is used to coordinate the addition of69 new devices - this must be synchronized across the array.70 Normally all nodes hold a concurrent-read lock on this device.71 723. Communication73================74 75 Messages can be broadcast to all nodes, and the sender waits for all76 other nodes to acknowledge the message before proceeding. Only one77 message can be processed at a time.78 793.1 Message Types80-----------------81 82 There are six types of messages which are passed:83 843.1.1 METADATA_UPDATED85^^^^^^^^^^^^^^^^^^^^^^86 87 informs other nodes that the metadata has88 been updated, and the node must re-read the md superblock. This is89 performed synchronously. It is primarily used to signal device90 failure.91 923.1.2 RESYNCING93^^^^^^^^^^^^^^^94 informs other nodes that a resync is initiated or95 ended so that each node may suspend or resume the region. Each96 RESYNCING message identifies a range of the devices that the97 sending node is about to resync. This overrides any previous98 notification from that node: only one ranged can be resynced at a99 time per-node.100 1013.1.3 NEWDISK102^^^^^^^^^^^^^103 104 informs other nodes that a device is being added to105 the array. Message contains an identifier for that device. See106 below for further details.107 1083.1.4 REMOVE109^^^^^^^^^^^^110 111 A failed or spare device is being removed from the112 array. The slot-number of the device is included in the message.113 114 3.1.5 RE_ADD:115 116 A failed device is being re-activated - the assumption117 is that it has been determined to be working again.118 119 3.1.6 BITMAP_NEEDS_SYNC:120 121 If a node is stopped locally but the bitmap122 isn't clean, then another node is informed to take the ownership of123 resync.124 1253.2 Communication mechanism126---------------------------127 128 The DLM LVB is used to communicate within nodes of the cluster. There129 are three resources used for the purpose:130 1313.2.1 token132^^^^^^^^^^^133 The resource which protects the entire communication134 system. The node having the token resource is allowed to135 communicate.136 1373.2.2 message138^^^^^^^^^^^^^139 The lock resource which carries the data to communicate.140 1413.2.3 ack142^^^^^^^^^143 144 The resource, acquiring which means the message has been145 acknowledged by all nodes in the cluster. The BAST of the resource146 is used to inform the receiving node that a node wants to147 communicate.148 149The algorithm is:150 151 1. receive status - all nodes have concurrent-reader lock on "ack"::152 153 sender receiver receiver154 "ack":CR "ack":CR "ack":CR155 156 2. sender get EX on "token",157 sender get EX on "message"::158 159 sender receiver receiver160 "token":EX "ack":CR "ack":CR161 "message":EX162 "ack":CR163 164 Sender checks that it still needs to send a message. Messages165 received or other events that happened while waiting for the166 "token" may have made this message inappropriate or redundant.167 168 3. sender writes LVB169 170 sender down-convert "message" from EX to CW171 172 sender try to get EX of "ack"173 174 ::175 176 [ wait until all receivers have *processed* the "message" ]177 178 [ triggered by bast of "ack" ]179 receiver get CR on "message"180 receiver read LVB181 receiver processes the message182 [ wait finish ]183 receiver releases "ack"184 receiver tries to get PR on "message"185 186 sender receiver receiver187 "token":EX "message":CR "message":CR188 "message":CW189 "ack":EX190 191 4. triggered by grant of EX on "ack" (indicating all receivers192 have processed message)193 194 sender down-converts "ack" from EX to CR195 196 sender releases "message"197 198 sender releases "token"199 200 ::201 202 receiver upconvert to PR on "message"203 receiver get CR of "ack"204 receiver release "message"205 206 sender receiver receiver207 "ack":CR "ack":CR "ack":CR208 209 2104. Handling Failures211====================212 2134.1 Node Failure214----------------215 216 When a node fails, the DLM informs the cluster with the slot217 number. The node starts a cluster recovery thread. The cluster218 recovery thread:219 220 - acquires the bitmap<number> lock of the failed node221 - opens the bitmap222 - reads the bitmap of the failed node223 - copies the set bitmap to local node224 - cleans the bitmap of the failed node225 - releases bitmap<number> lock of the failed node226 - initiates resync of the bitmap on the current node227 md_check_recovery is invoked within recover_bitmaps,228 then md_check_recovery -> metadata_update_start/finish,229 it will lock the communication by lock_comm.230 Which means when one node is resyncing it blocks all231 other nodes from writing anywhere on the array.232 233 The resync process is the regular md resync. However, in a clustered234 environment when a resync is performed, it needs to tell other nodes235 of the areas which are suspended. Before a resync starts, the node236 send out RESYNCING with the (lo,hi) range of the area which needs to237 be suspended. Each node maintains a suspend_list, which contains the238 list of ranges which are currently suspended. On receiving RESYNCING,239 the node adds the range to the suspend_list. Similarly, when the node240 performing resync finishes, it sends RESYNCING with an empty range to241 other nodes and other nodes remove the corresponding entry from the242 suspend_list.243 244 A helper function, ->area_resyncing() can be used to check if a245 particular I/O range should be suspended or not.246 2474.2 Device Failure248==================249 250 Device failures are handled and communicated with the metadata update251 routine. When a node detects a device failure it does not allow252 any further writes to that device until the failure has been253 acknowledged by all other nodes.254 2555. Adding a new Device256----------------------257 258 For adding a new device, it is necessary that all nodes "see" the new259 device to be added. For this, the following algorithm is used:260 261 1. Node 1 issues mdadm --manage /dev/mdX --add /dev/sdYY which issues262 ioctl(ADD_NEW_DISK with disc.state set to MD_DISK_CLUSTER_ADD)263 2. Node 1 sends a NEWDISK message with uuid and slot number264 3. Other nodes issue kobject_uevent_env with uuid and slot number265 (Steps 4,5 could be a udev rule)266 4. In userspace, the node searches for the disk, perhaps267 using blkid -t SUB_UUID=""268 5. Other nodes issue either of the following depending on whether269 the disk was found:270 ioctl(ADD_NEW_DISK with disc.state set to MD_DISK_CANDIDATE and271 disc.number set to slot number)272 ioctl(CLUSTERED_DISK_NACK)273 6. Other nodes drop lock on "no-new-devs" (CR) if device is found274 7. Node 1 attempts EX lock on "no-new-dev"275 8. If node 1 gets the lock, it sends METADATA_UPDATED after276 unmarking the disk as SpareLocal277 9. If not (get "no-new-dev" lock), it fails the operation and sends278 METADATA_UPDATED.279 10. Other nodes get the information whether a disk is added or not280 by the following METADATA_UPDATED.281 2826. Module interface283===================284 285 There are 17 call-backs which the md core can make to the cluster286 module. Understanding these can give a good overview of the whole287 process.288 2896.1 join(nodes) and leave()290---------------------------291 292 These are called when an array is started with a clustered bitmap,293 and when the array is stopped. join() ensures the cluster is294 available and initializes the various resources.295 Only the first 'nodes' nodes in the cluster can use the array.296 2976.2 slot_number()298-----------------299 300 Reports the slot number advised by the cluster infrastructure.301 Range is from 0 to nodes-1.302 3036.3 resync_info_update()304------------------------305 306 This updates the resync range that is stored in the bitmap lock.307 The starting point is updated as the resync progresses. The308 end point is always the end of the array.309 It does *not* send a RESYNCING message.310 3116.4 resync_start(), resync_finish()312-----------------------------------313 314 These are called when resync/recovery/reshape starts or stops.315 They update the resyncing range in the bitmap lock and also316 send a RESYNCING message. resync_start reports the whole317 array as resyncing, resync_finish reports none of it.318 319 resync_finish() also sends a BITMAP_NEEDS_SYNC message which320 allows some other node to take over.321 3226.5 metadata_update_start(), metadata_update_finish(), metadata_update_cancel()323-------------------------------------------------------------------------------324 325 metadata_update_start is used to get exclusive access to326 the metadata. If a change is still needed once that access is327 gained, metadata_update_finish() will send a METADATA_UPDATE328 message to all other nodes, otherwise metadata_update_cancel()329 can be used to release the lock.330 3316.6 area_resyncing()332--------------------333 334 This combines two elements of functionality.335 336 Firstly, it will check if any node is currently resyncing337 anything in a given range of sectors. If any resync is found,338 then the caller will avoid writing or read-balancing in that339 range.340 341 Secondly, while node recovery is happening it reports that342 all areas are resyncing for READ requests. This avoids races343 between the cluster-filesystem and the cluster-RAID handling344 a node failure.345 3466.7 add_new_disk_start(), add_new_disk_finish(), new_disk_ack()347---------------------------------------------------------------348 349 These are used to manage the new-disk protocol described above.350 When a new device is added, add_new_disk_start() is called before351 it is bound to the array and, if that succeeds, add_new_disk_finish()352 is called the device is fully added.353 354 When a device is added in acknowledgement to a previous355 request, or when the device is declared "unavailable",356 new_disk_ack() is called.357 3586.8 remove_disk()359-----------------360 361 This is called when a spare or failed device is removed from362 the array. It causes a REMOVE message to be send to other nodes.363 3646.9 gather_bitmaps()365--------------------366 367 This sends a RE_ADD message to all other nodes and then368 gathers bitmap information from all bitmaps. This combined369 bitmap is then used to recovery the re-added device.370 3716.10 lock_all_bitmaps() and unlock_all_bitmaps()372------------------------------------------------373 374 These are called when change bitmap to none. If a node plans375 to clear the cluster raid's bitmap, it need to make sure no other376 nodes are using the raid which is achieved by lock all bitmap377 locks within the cluster, and also those locks are unlocked378 accordingly.379 3807. Unsupported features381=======================382 383There are somethings which are not supported by cluster MD yet.384 385- change array_sectors.386