491 lines · plain
1*****************************************************2Support for AArch64 Scalable Matrix Extension in LLVM3*****************************************************4 5.. contents::6 :local:7 81. Introduction9===============10 11The :ref:`AArch64 SME ACLE <aarch64_sme_acle>` provides a number of12attributes for users to control PSTATE.SM and PSTATE.ZA.13The :ref:`AArch64 SME ABI<aarch64_sme_abi>` describes the requirements for14calls between functions when at least one of those functions uses PSTATE.SM or15PSTATE.ZA.16 17This document describes how the SME ACLE attributes map to LLVM IR18attributes and how LLVM lowers these attributes to implement the rules and19requirements of the ABI.20 21Below, we describe the LLVM IR attributes and their relation to the22C/C++-level ACLE attributes:23 24``aarch64_pstate_sm_enabled``25 is used for functions with ``__arm_streaming``26 27``aarch64_pstate_sm_compatible``28 is used for functions with ``__arm_streaming_compatible``29 30``aarch64_pstate_sm_body``31 is used for functions with ``__arm_locally_streaming`` and is32 only valid on function definitions (not declarations)33 34``aarch64_new_za``35 is used for functions with ``__arm_new("za")``36 37``aarch64_in_za``38 is used for functions with ``__arm_in("za")``39 40``aarch64_out_za``41 is used for functions with ``__arm_out("za")``42 43``aarch64_inout_za``44 is used for functions with ``__arm_inout("za")``45 46``aarch64_preserves_za``47 is used for functions with ``__arm_preserves("za")``48 49``aarch64_expanded_pstate_za``50 is used for functions with ``__arm_new_za``51 52Clang must ensure that the above attributes are added both to the53function's declaration/definition as well as to their call-sites. This is54important for calls to attributed function pointers, where no55definition or declaration is available.56 57 582. Handling PSTATE.SM59=====================60 61When changing PSTATE.SM the execution of FP/vector operations may be transferred62to another processing element. This has three important implications:63 64* The runtime SVE vector length may change.65 66* The contents of FP/AdvSIMD/SVE registers are zeroed.67 68* The set of allowable instructions changes.69 70This leads to certain restrictions on IR and optimizations. For example, it71is undefined behaviour to share vector-length dependent state between functions72that may operate with different values for PSTATE.SM. Front-ends must honour73these restrictions when generating LLVM IR.74 75Even though the runtime SVE vector length may change, for the purpose of LLVM IR76and almost all parts of CodeGen we can assume that the runtime value for77``vscale`` does not. If we let the compiler insert the appropriate ``smstart``78and ``smstop`` instructions around call boundaries, then the effects on SVE79state can be mitigated. By limiting the state changes to a very brief window80around the call, we can control how the operations are scheduled and how live81values remain preserved between state transitions.82 83In order to control PSTATE.SM at this level of granularity, we use function and84callsite attributes rather than intrinsics.85 86 87Restrictions on attributes88--------------------------89 90* It is undefined behaviour to pass or return (pointers to) scalable vector91 objects to/from functions which may use a different SVE vector length.92 This includes functions with a non-streaming interface but marked with93 ``aarch64_pstate_sm_body``.94 95* It is not allowed for a function to be decorated with both96 ``aarch64_pstate_sm_compatible`` and ``aarch64_pstate_sm_enabled``.97 98* It is not allowed for a function to be decorated with more than one of the99 following attributes:100 ``aarch64_new_za``, ``aarch64_in_za``, ``aarch64_out_za``, ``aarch64_inout_za``,101 ``aarch64_preserves_za``.102 103These restrictions also apply in the higher-level SME ACLE, which means we can104emit diagnostics in Clang to signal users about incorrect behaviour.105 106 107Compiler inserted streaming-mode changes108----------------------------------------109 110The table below describes the transitions in PSTATE.SM the compiler has to111account for when doing calls between functions with different attributes.112In this table, we use the following abbreviations:113 114``N``115 functions with a normal interface (PSTATE.SM=0 on entry, PSTATE.SM=0 on116 return)117 118``S``119 functions with a Streaming interface (PSTATE.SM=1 on entry, PSTATE.SM=1120 on return)121 122``SC``123 functions with a Streaming-Compatible interface (PSTATE.SM can be124 either 0 or 1 on entry, and is unchanged on return).125 126Functions with ``__attribute__((arm_locally_streaming))`` are excluded from this127table because for the caller the attribute is synonymous with 'streaming', and128for the callee it is merely an implementation detail that is explicitly not129exposed to the caller.130 131.. table:: Combinations of calls for functions with different attributes132 133 ==== ==== =============================== ============================== ==============================134 From To Before call After call After exception135 ==== ==== =============================== ============================== ==============================136 N N137 N S SMSTART SMSTOP138 N SC139 S N SMSTOP SMSTART SMSTART140 S S SMSTART141 S SC SMSTART142 SC N If PSTATE.SM before call is 1, If PSTATE.SM before call is 1, If PSTATE.SM before call is 1,143 then SMSTOP then SMSTART then SMSTART144 SC S If PSTATE.SM before call is 0, If PSTATE.SM before call is 0, If PSTATE.SM before call is 1,145 then SMSTART then SMSTOP then SMSTART146 SC SC If PSTATE.SM before call is 1,147 then SMSTART148 ==== ==== =============================== ============================== ==============================149 150 151Because changing PSTATE.SM zeroes the FP/vector registers, it is best to emit152the ``smstart`` and ``smstop`` instructions before register allocation, so that153the register allocator can spill/reload registers around the mode change.154 155The compiler should also have sufficient information on which operations are156part of the call/function's arguments/result and which operations are part of157the function's body, so that it can place the mode changes in exactly the right158position. The suitable place to do this seems to be SelectionDAG, where it lowers159the call's arguments/return values to implement the specified calling convention.160SelectionDAG provides Chains and Glue to specify the order of operations and give161preliminary control over instruction scheduling.162 163 164Example of preserving state165---------------------------166 167When passing and returning a ``float`` value to/from a function168that has a streaming interface from a function that has a normal interface, the169call-site will need to ensure that the argument/result registers are preserved170and that no other code is scheduled in between the ``smstart/smstop`` and the call.171 172.. code-block:: llvm173 174 define float @foo(float %f) nounwind {175 %res = call float @bar(float %f) "aarch64_pstate_sm_enabled"176 ret float %res177 }178 179 declare float @bar(float) "aarch64_pstate_sm_enabled"180 181The program needs to preserve the value of the floating point argument and182return value in register ``s0``:183 184.. code-block:: none185 186 foo: // @foo187 // %bb.0:188 stp d15, d14, [sp, #-80]! // 16-byte Folded Spill189 stp d13, d12, [sp, #16] // 16-byte Folded Spill190 stp d11, d10, [sp, #32] // 16-byte Folded Spill191 stp d9, d8, [sp, #48] // 16-byte Folded Spill192 str x30, [sp, #64] // 8-byte Folded Spill193 str s0, [sp, #76] // 4-byte Folded Spill194 smstart sm195 ldr s0, [sp, #76] // 4-byte Folded Reload196 bl bar197 str s0, [sp, #76] // 4-byte Folded Spill198 smstop sm199 ldp d9, d8, [sp, #48] // 16-byte Folded Reload200 ldp d11, d10, [sp, #32] // 16-byte Folded Reload201 ldp d13, d12, [sp, #16] // 16-byte Folded Reload202 ldr s0, [sp, #76] // 4-byte Folded Reload203 ldr x30, [sp, #64] // 8-byte Folded Reload204 ldp d15, d14, [sp], #80 // 16-byte Folded Reload205 ret206 207Setting the correct register masks on the ISD nodes and inserting the208``smstart/smstop`` in the right places should ensure this is done correctly.209 210 211Instruction Selection Nodes212---------------------------213 214.. code-block:: none215 216 AArch64ISD::SMSTART Chain, [SM|ZA|Both][, RegMask]217 AArch64ISD::SMSTOP Chain, [SM|ZA|Both][, RegMask]218 AArch64ISD::COND_SMSTART Chain, [SM|ZA|Both], CurrentState, ExpectedState[, RegMask]219 AArch64ISD::COND_SMSTOP Chain, [SM|ZA|Both], CurrentState, ExpectedState[, RegMask]220 221The ``COND_SMSTART/COND_SMSTOP`` nodes additionally take ``CurrentState`` and222``ExpectedState``, in this case the instruction will only be executed if223``CurrentState != ExpectedState``.224 225When ``CurrentState`` and ``ExpectedState`` can be evaluated at compile-time226(i.e. they are both constants) then an unconditional ``smstart/smstop``227instruction is emitted. Otherwise, the node is matched to a Pseudo instruction228which expands to a compare/branch and a ``smstart/smstop``. This is necessary to229implement transitions from ``SC -> N`` and ``SC -> S``.230 231 232Unchained Function calls233------------------------234When a function with "``aarch64_pstate_sm_enabled``" calls a function that is not235streaming compatible, the compiler has to insert an SMSTOP before the call and236insert an SMSTOP after the call.237 238If the function that is called is an intrinsic with no side-effects which in239turn is lowered to a function call (e.g., ``@llvm.cos()``), then the call to240``@llvm.cos()`` is not part of any Chain; it can be scheduled freely.241 242Lowering of a Callsite creates a small chain of nodes which:243 244- starts a call sequence245 246- copies input values from virtual registers to physical registers specified by247 the ABI248 249- executes a branch-and-link250 251- stops the call sequence252 253- copies the output values from their physical registers to virtual registers254 255When the callsite's Chain is not used, only the result value from the chained256sequence is used, but the Chain itself is discarded.257 258The ``SMSTART`` and ``SMSTOP`` ISD nodes return a Chain, but no real259values, so when the ``SMSTART/SMSTOP`` nodes are part of a Chain that isn't260used, these nodes are not considered for scheduling and are261removed from the DAG. In order to prevent these nodes262from being removed, we need a way to ensure the results from the263``CopyFromReg`` can only be **used after** the ``SMSTART/SMSTOP`` has been264executed.265 266We can use a CopyToReg -> CopyFromReg sequence for this, which moves the267value to/from a virtual register and chains these nodes with the268SMSTART/SMSTOP to make them part of the expression that calculates269the result value. The resulting COPY nodes are removed by the register270allocator.271 272The example below shows how this is used in a DAG that does not link273together the result by a Chain, but rather by a value:274 275.. code-block:: none276 277 t0: ch,glue = AArch64ISD::SMSTOP ...278 t1: ch,glue = ISD::CALL ....279 t2: res,ch,glue = CopyFromReg t1, ...280 t3: ch,glue = AArch64ISD::SMSTART t2:1, .... <- this is now part of the expression that returns the result value.281 t4: ch = CopyToReg t3, Register:f64 %vreg, t2282 t5: res,ch = CopyFromReg t4, Register:f64 %vreg283 t6: res = FADD t5, t9284 285We also need this for locally streaming functions, where an ``SMSTART`` needs to286be inserted into the DAG at the start of the function.287 288Functions with __attribute__((arm_locally_streaming))289-----------------------------------------------------290 291If a function is marked as ``arm_locally_streaming``, then the runtime SVE292vector length in the prologue/epilogue may be different from the vector length293in the function's body. This happens because we invoke smstart after setting up294the stack-frame and similarly invoke smstop before deallocating the stack-frame.295 296To ensure we use the correct SVE vector length to allocate the locals with, we297can use the streaming vector-length to allocate the stack-slots through the298``ADDSVL`` instruction, even when the CPU is not yet in streaming mode.299 300This works only for locals and not callee-save slots, since LLVM doesn't support301mixing two different scalable vector lengths in one stack frame. That means that the302case where a function is marked ``arm_locally_streaming`` and needs to spill SVE303callee-saves in the prologue is currently unsupported. However, it is unlikely304for this to happen without user intervention because ``arm_locally_streaming``305functions cannot take or return vector-length-dependent values. This would otherwise306require forcing both the SVE PCS using '``aarch64_sve_pcs``' combined with using307``arm_locally_streaming`` in order to encounter this problem. This combination308can be prevented in Clang through emitting a diagnostic.309 310 311An example of how the prologue/epilogue would look for a function that is312attributed with ``arm_locally_streaming``:313 314.. code-block:: c++315 316 #define N 64317 318 void __attribute__((arm_streaming_compatible)) some_use(svfloat32_t *);319 320 // Use a float argument type, to check the value isn't clobbered by smstart.321 // Use a float return type to check the value isn't clobbered by smstop.322 float __attribute__((noinline, arm_locally_streaming)) foo(float arg) {323 // Create local for SVE vector to check local is created with correct324 // size when not yet in streaming mode (ADDSVL).325 float array[N];326 svfloat32_t vector;327 328 some_use(&vector);329 svst1_f32(svptrue_b32(), &array[0], vector);330 return array[N - 1] + arg;331 }332 333should use ``ADDSVL`` for allocating the stack space and should avoid clobbering334the return/argument values.335 336.. code-block:: none337 338 _Z3foof: // @_Z3foof339 // %bb.0: // %entry340 stp d15, d14, [sp, #-96]! // 16-byte Folded Spill341 stp d13, d12, [sp, #16] // 16-byte Folded Spill342 stp d11, d10, [sp, #32] // 16-byte Folded Spill343 stp d9, d8, [sp, #48] // 16-byte Folded Spill344 stp x29, x30, [sp, #64] // 16-byte Folded Spill345 add x29, sp, #64346 str x28, [sp, #80] // 8-byte Folded Spill347 addsvl sp, sp, #-1348 sub sp, sp, #256349 str s0, [x29, #28] // 4-byte Folded Spill350 smstart sm351 sub x0, x29, #64352 addsvl x0, x0, #-1353 bl _Z10some_usePu13__SVFloat32_t354 sub x8, x29, #64355 ptrue p0.s356 ld1w { z0.s }, p0/z, [x8, #-1, mul vl]357 ldr s1, [x29, #28] // 4-byte Folded Reload358 st1w { z0.s }, p0, [sp]359 ldr s0, [sp, #252]360 fadd s0, s0, s1361 str s0, [x29, #28] // 4-byte Folded Spill362 smstop sm363 ldr s0, [x29, #28] // 4-byte Folded Reload364 addsvl sp, sp, #1365 add sp, sp, #256366 ldp x29, x30, [sp, #64] // 16-byte Folded Reload367 ldp d9, d8, [sp, #48] // 16-byte Folded Reload368 ldp d11, d10, [sp, #32] // 16-byte Folded Reload369 ldp d13, d12, [sp, #16] // 16-byte Folded Reload370 ldr x28, [sp, #80] // 8-byte Folded Reload371 ldp d15, d14, [sp], #96 // 16-byte Folded Reload372 ret373 374 375Preventing the use of illegal instructions in Streaming Mode376------------------------------------------------------------377 378* When executing a program in streaming-mode (PSTATE.SM=1) a subset of SVE/SVE2379 instructions and most AdvSIMD/NEON instructions are invalid.380 381* When executing a program in normal mode (PSTATE.SM=0), a subset of SME382 instructions are invalid.383 384* Streaming-compatible functions must use only instructions that are valid when385 either PSTATE.SM=0 or PSTATE.SM=1.386 387The value of PSTATE.SM is not controlled by the feature flags, but rather by the388function attributes. This means that we can compile for '``+sme``', and the compiler389will code-generate any instructions, even if they are not legal under the requested390streaming mode. The compiler needs to use the function attributes to ensure the391compiler doesn't perform transformations under the assumption that certain operations392are available at runtime.393 394We made a conscious choice not to model this with feature flags because we395still want to support inline-asm in either mode (with the user placing396smstart/smstop manually), and this became rather complicated to implement at the397individual instruction level (see `D120261 <https://reviews.llvm.org/D120261>`_398and `D121208 <https://reviews.llvm.org/D121208>`_) because of limitations in399TableGen.400 401As a first step, this means we'll disable vectorization (LoopVectorize/SLP)402entirely when a function has either of the ``aarch64_pstate_sm_enabled``,403``aarch64_pstate_sm_body`` or ``aarch64_pstate_sm_compatible`` attributes,404in order to avoid the use of vector instructions.405 406Later on, we'll aim to relax these restrictions to enable scalable407auto-vectorization with a subset of streaming-compatible instructions, but that408requires changes to the CostModel, Legalization and SelectionDAG lowering.409 410We will also emit diagnostics in Clang to prevent the use of411non-streaming(-compatible) operations, e.g., through ACLE intrinsics, when a412function is decorated with the streaming mode attributes.413 414 415Other things to consider416------------------------417 418* Inlining must be disabled when the call-site needs to toggle PSTATE.SM or419 when the callee's function body is executed in a different streaming mode from420 its caller. This is needed because function calls are the boundaries for421 streaming mode changes.422 423* Tail call optimization must be disabled when the call-site needs to toggle424 PSTATE.SM, such that the caller can restore the original value of PSTATE.SM.425 426 4273. Handling PSTATE.ZA428=====================429 430In contrast to PSTATE.SM, enabling PSTATE.ZA does not affect the SVE vector431length and also doesn't clobber FP/AdvSIMD/SVE registers. This means it is safe432to toggle PSTATE.ZA using intrinsics. This also makes it simpler to setup a433lazy-save mechanism for calls to private-ZA functions (i.e. functions that may434either directly or indirectly clobber ZA state).435 436For the purpose of handling functions marked with ``aarch64_new_za``,437we have introduced a new LLVM IR pass (SMEABIPass) that runs just before438SelectionDAG. Any such functions handled by this pass are marked with439``aarch64_expanded_pstate_za``.440 441Setting up a lazy-save442----------------------443 444Committing a lazy-save445----------------------446 447Exception handling and ZA448-------------------------449 4504. Types451========452 453AArch64 Predicate-as-Counter Type454---------------------------------455 456:Overview:457 458The predicate-as-counter type represents the type of a predicate-as-counter459value held in an AArch64 SVE predicate register. Such a value contains460information about the number of active lanes, the element width and a bit that461indicates whether the generated mask should be inverted. ACLE intrinsics should be462used to move the predicate-as-counter value to/from a predicate vector.463 464There are certain limitations on the type:465 466* The type can be used for function parameters and return values.467 468* The supported LLVM operations on this type are limited to ``load``, ``store``,469 ``phi``, ``select``, and ``alloca`` instructions.470 471The predicate-as-counter type is a scalable type.472 473:Syntax:474 475::476 477 target("aarch64.svcount")478 479 480 4815. References482=============483 484 .. _aarch64_sme_acle:485 4861. `SME ACLE Pull-request <https://github.com/ARM-software/acle/pull/188>`__487 488 .. _aarch64_sme_abi:489 4902. `SME ABI Pull-request <https://github.com/ARM-software/abi-aa/pull/123>`__491