459 lines · plain
1.. SPDX-License-Identifier: GPL-2.02 3=========4SAS Layer5=========6 7The SAS Layer is a management infrastructure which manages8SAS LLDDs. It sits between SCSI Core and SAS LLDDs. The9layout is as follows: while SCSI Core is concerned with10SAM/SPC issues, and a SAS LLDD+sequencer is concerned with11phy/OOB/link management, the SAS layer is concerned with:12 13 * SAS Phy/Port/HA event management (LLDD generates,14 SAS Layer processes),15 * SAS Port management (creation/destruction),16 * SAS Domain discovery and revalidation,17 * SAS Domain device management,18 * SCSI Host registration/unregistration,19 * Device registration with SCSI Core (SAS) or libata20 (SATA), and21 * Expander management and exporting expander control22 to user space.23 24A SAS LLDD is a PCI device driver. It is concerned with25phy/OOB management, and vendor specific tasks and generates26events to the SAS layer.27 28The SAS Layer does most SAS tasks as outlined in the SAS 1.129spec.30 31The sas_ha_struct describes the SAS LLDD to the SAS layer.32Most of it is used by the SAS Layer but a few fields need to33be initialized by the LLDDs.34 35After initializing your hardware, from the probe() function36you call sas_register_ha(). It will register your LLDD with37the SCSI subsystem, creating a SCSI host and it will38register your SAS driver with the sysfs SAS tree it creates.39It will then return. Then you enable your phys to actually40start OOB (at which point your driver will start calling the41notify_* event callbacks).42 43Structure descriptions44======================45 46``struct sas_phy``47------------------48 49Normally this is statically embedded to your driver's50phy structure::51 52 struct my_phy {53 blah;54 struct sas_phy sas_phy;55 bleh;56 };57 58And then all the phys are an array of my_phy in your HA59struct (shown below).60 61Then as you go along and initialize your phys you also62initialize the sas_phy struct, along with your own63phy structure.64 65In general, the phys are managed by the LLDD and the ports66are managed by the SAS layer. So the phys are initialized67and updated by the LLDD and the ports are initialized and68updated by the SAS layer.69 70There is a scheme where the LLDD can RW certain fields,71and the SAS layer can only read such ones, and vice versa.72The idea is to avoid unnecessary locking.73 74enabled75 - must be set (0/1)76 77id78 - must be set [0,MAX_PHYS)]79 80class, proto, type, role, oob_mode, linkrate81 - must be set82 83oob_mode84 - you set this when OOB has finished and then notify85 the SAS Layer.86 87sas_addr88 - this normally points to an array holding the sas89 address of the phy, possibly somewhere in your my_phy90 struct.91 92attached_sas_addr93 - set this when you (LLDD) receive an94 IDENTIFY frame or a FIS frame, _before_ notifying the SAS95 layer. The idea is that sometimes the LLDD may want to fake96 or provide a different SAS address on that phy/port and this97 allows it to do this. At best you should copy the sas98 address from the IDENTIFY frame or maybe generate a SAS99 address for SATA directly attached devices. The Discover100 process may later change this.101 102frame_rcvd103 - this is where you copy the IDENTIFY/FIS frame104 when you get it; you lock, copy, set frame_rcvd_size and105 unlock the lock, and then call the event. It is a pointer106 since there's no way to know your hw frame size _exactly_,107 so you define the actual array in your phy struct and let108 this pointer point to it. You copy the frame from your109 DMAable memory to that area holding the lock.110 111sas_prim112 - this is where primitives go when they're113 received. See sas.h. Grab the lock, set the primitive,114 release the lock, notify.115 116port117 - this points to the sas_port if the phy belongs118 to a port -- the LLDD only reads this. It points to the119 sas_port this phy is part of. Set by the SAS Layer.120 121ha122 - may be set; the SAS layer sets it anyway.123 124lldd_phy125 - you should set this to point to your phy so you126 can find your way around faster when the SAS layer calls one127 of your callbacks and passes you a phy. If the sas_phy is128 embedded you can also use container_of -- whatever you129 prefer.130 131 132``struct sas_port``133-------------------134 135The LLDD doesn't set any fields of this struct -- it only136reads them. They should be self explanatory.137 138phy_mask is 32 bit, this should be enough for now, as I139haven't heard of a HA having more than 8 phys.140 141lldd_port142 - I haven't found use for that -- maybe other143 LLDD who wish to have internal port representation can make144 use of this.145 146``struct sas_ha_struct``147------------------------148 149It normally is statically declared in your own LLDD150structure describing your adapter::151 152 struct my_sas_ha {153 blah;154 struct sas_ha_struct sas_ha;155 struct my_phy phys[MAX_PHYS];156 struct sas_port sas_ports[MAX_PHYS]; /* (1) */157 bleh;158 };159 160 (1) If your LLDD doesn't have its own port representation.161 162What needs to be initialized (sample function given below).163 164pcidev165^^^^^^166 167sas_addr168 - since the SAS layer doesn't want to mess with169 memory allocation, etc, this points to statically170 allocated array somewhere (say in your host adapter171 structure) and holds the SAS address of the host172 adapter as given by you or the manufacturer, etc.173 174sas_port175^^^^^^^^176 177sas_phy178 - an array of pointers to structures. (see179 note above on sas_addr).180 These must be set. See more notes below.181 182num_phys183 - the number of phys present in the sas_phy array,184 and the number of ports present in the sas_port185 array. There can be a maximum num_phys ports (one per186 port) so we drop the num_ports, and only use187 num_phys.188 189The event interface::190 191 /* LLDD calls these to notify the class of an event. */192 void sas_notify_port_event(struct sas_phy *, enum port_event, gfp_t);193 void sas_notify_phy_event(struct sas_phy *, enum phy_event, gfp_t);194 195The port notification::196 197 /* The class calls these to notify the LLDD of an event. */198 void (*lldd_port_formed)(struct sas_phy *);199 void (*lldd_port_deformed)(struct sas_phy *);200 201If the LLDD wants notification when a port has been formed202or deformed it sets those to a function satisfying the type.203 204A SAS LLDD should also implement at least one of the Task205Management Functions (TMFs) described in SAM::206 207 /* Task Management Functions. Must be called from process context. */208 int (*lldd_abort_task)(struct sas_task *);209 int (*lldd_abort_task_set)(struct domain_device *, u8 *lun);210 int (*lldd_clear_task_set)(struct domain_device *, u8 *lun);211 int (*lldd_I_T_nexus_reset)(struct domain_device *);212 int (*lldd_lu_reset)(struct domain_device *, u8 *lun);213 int (*lldd_query_task)(struct sas_task *);214 215For more information please read SAM from T10.org.216 217Port and Adapter management::218 219 /* Port and Adapter management */220 int (*lldd_clear_nexus_port)(struct sas_port *);221 int (*lldd_clear_nexus_ha)(struct sas_ha_struct *);222 223A SAS LLDD should implement at least one of those.224 225Phy management::226 227 /* Phy management */228 int (*lldd_control_phy)(struct sas_phy *, enum phy_func);229 230lldd_ha231 - set this to point to your HA struct. You can also232 use container_of if you embedded it as shown above.233 234A sample initialization and registration function235can look like this (called last thing from probe())236*but* before you enable the phys to do OOB::237 238 static int register_sas_ha(struct my_sas_ha *my_ha)239 {240 int i;241 static struct sas_phy *sas_phys[MAX_PHYS];242 static struct sas_port *sas_ports[MAX_PHYS];243 244 my_ha->sas_ha.sas_addr = &my_ha->sas_addr[0];245 246 for (i = 0; i < MAX_PHYS; i++) {247 sas_phys[i] = &my_ha->phys[i].sas_phy;248 sas_ports[i] = &my_ha->sas_ports[i];249 }250 251 my_ha->sas_ha.sas_phy = sas_phys;252 my_ha->sas_ha.sas_port = sas_ports;253 my_ha->sas_ha.num_phys = MAX_PHYS;254 255 my_ha->sas_ha.lldd_port_formed = my_port_formed;256 257 my_ha->sas_ha.lldd_dev_found = my_dev_found;258 my_ha->sas_ha.lldd_dev_gone = my_dev_gone;259 260 my_ha->sas_ha.lldd_execute_task = my_execute_task;261 262 my_ha->sas_ha.lldd_abort_task = my_abort_task;263 my_ha->sas_ha.lldd_abort_task_set = my_abort_task_set;264 my_ha->sas_ha.lldd_clear_task_set = my_clear_task_set;265 my_ha->sas_ha.lldd_I_T_nexus_reset= NULL; (2)266 my_ha->sas_ha.lldd_lu_reset = my_lu_reset;267 my_ha->sas_ha.lldd_query_task = my_query_task;268 269 my_ha->sas_ha.lldd_clear_nexus_port = my_clear_nexus_port;270 my_ha->sas_ha.lldd_clear_nexus_ha = my_clear_nexus_ha;271 272 my_ha->sas_ha.lldd_control_phy = my_control_phy;273 274 return sas_register_ha(&my_ha->sas_ha);275 }276 277(2) SAS 1.1 does not define I_T Nexus Reset TMF.278 279Events280======281 282Events are **the only way** a SAS LLDD notifies the SAS layer283of anything. There is no other method or way a LLDD to tell284the SAS layer of anything happening internally or in the SAS285domain.286 287Phy events::288 289 PHYE_LOSS_OF_SIGNAL, (C)290 PHYE_OOB_DONE,291 PHYE_OOB_ERROR, (C)292 PHYE_SPINUP_HOLD.293 294Port events, passed on a _phy_::295 296 PORTE_BYTES_DMAED, (M)297 PORTE_BROADCAST_RCVD, (E)298 PORTE_LINK_RESET_ERR, (C)299 PORTE_TIMER_EVENT, (C)300 PORTE_HARD_RESET.301 302Host Adapter event:303 HAE_RESET304 305A SAS LLDD should be able to generate306 307 - at least one event from group C (choice),308 - events marked M (mandatory) are mandatory (only one),309 - events marked E (expander) if it wants the SAS layer310 to handle domain revalidation (only one such).311 - Unmarked events are optional.312 313Meaning:314 315HAE_RESET316 - when your HA got internal error and was reset.317 318PORTE_BYTES_DMAED319 - on receiving an IDENTIFY/FIS frame320 321PORTE_BROADCAST_RCVD322 - on receiving a primitive323 324PORTE_LINK_RESET_ERR325 - timer expired, loss of signal, loss of DWS, etc. [1]_326 327PORTE_TIMER_EVENT328 - DWS reset timeout timer expired [1]_329 330PORTE_HARD_RESET331 - Hard Reset primitive received.332 333PHYE_LOSS_OF_SIGNAL334 - the device is gone [1]_335 336PHYE_OOB_DONE337 - OOB went fine and oob_mode is valid338 339PHYE_OOB_ERROR340 - Error while doing OOB, the device probably341 got disconnected. [1]_342 343PHYE_SPINUP_HOLD344 - SATA is present, COMWAKE not sent.345 346.. [1] should set/clear the appropriate fields in the phy,347 or alternatively call the inlined sas_phy_disconnected()348 which is just a helper, from their tasklet.349 350The Execute Command SCSI RPC::351 352 int (*lldd_execute_task)(struct sas_task *, gfp_t gfp_flags);353 354Used to queue a task to the SAS LLDD. @task is the task to be executed.355@gfp_mask is the gfp_mask defining the context of the caller.356 357This function should implement the Execute Command SCSI RPC,358 359That is, when lldd_execute_task() is called, the command360go out on the transport *immediately*. There is *no*361queuing of any sort and at any level in a SAS LLDD.362 363Returns:364 365 * -SAS_QUEUE_FULL, -ENOMEM, nothing was queued;366 * 0, the task(s) were queued.367 368::369 370 struct sas_task {371 dev -- the device this task is destined to372 task_proto -- _one_ of enum sas_proto373 scatter -- pointer to scatter gather list array374 num_scatter -- number of elements in scatter375 total_xfer_len -- total number of bytes expected to be transferred376 data_dir -- PCI_DMA_...377 task_done -- callback when the task has finished execution378 };379 380Discovery381=========382 383The sysfs tree has the following purposes:384 385 a) It shows you the physical layout of the SAS domain at386 the current time, i.e. how the domain looks in the387 physical world right now.388 b) Shows some device parameters _at_discovery_time_.389 390This is a link to the tree(1) program, very useful in391viewing the SAS domain:392ftp://mama.indstate.edu/linux/tree/393 394I expect user space applications to actually create a395graphical interface of this.396 397That is, the sysfs domain tree doesn't show or keep state if398you e.g., change the meaning of the READY LED MEANING399setting, but it does show you the current connection status400of the domain device.401 402Keeping internal device state changes is responsibility of403upper layers (Command set drivers) and user space.404 405When a device or devices are unplugged from the domain, this406is reflected in the sysfs tree immediately, and the device(s)407removed from the system.408 409The structure domain_device describes any device in the SAS410domain. It is completely managed by the SAS layer. A task411points to a domain device, this is how the SAS LLDD knows412where to send the task(s) to. A SAS LLDD only reads the413contents of the domain_device structure, but it never creates414or destroys one.415 416Expander management from User Space417===================================418 419In each expander directory in sysfs, there is a file called420"smp_portal". It is a binary sysfs attribute file, which421implements an SMP portal (Note: this is *NOT* an SMP port),422to which user space applications can send SMP requests and423receive SMP responses.424 425Functionality is deceptively simple:426 4271. Build the SMP frame you want to send. The format and layout428 is described in the SAS spec. Leave the CRC field equal 0.429 430open(2)431 4322. Open the expander's SMP portal sysfs file in RW mode.433 434write(2)435 4363. Write the frame you built in 1.437 438read(2)439 4404. Read the amount of data you expect to receive for the frame you built.441 If you receive different amount of data you expected to receive,442 then there was some kind of error.443 444close(2)445 446All this process is shown in detail in the function do_smp_func()447and its callers, in the file "expander_conf.c".448 449The kernel functionality is implemented in the file450"sas_expander.c".451 452The program "expander_conf.c" implements this. It takes one453argument, the sysfs file name of the SMP portal to the454expander, and gives expander information, including routing455tables.456 457The SMP portal gives you complete control of the expander,458so please be careful.459