15210 lines · plain
1=============================2User Guide for AMDGPU Backend3=============================4 5.. contents::6 :local:7 8.. toctree::9 :hidden:10 11 AMDGPU/AMDGPUAsmGFX712 AMDGPU/AMDGPUAsmGFX813 AMDGPU/AMDGPUAsmGFX914 AMDGPU/AMDGPUAsmGFX90015 AMDGPU/AMDGPUAsmGFX90416 AMDGPU/AMDGPUAsmGFX90617 AMDGPU/AMDGPUAsmGFX90818 AMDGPU/AMDGPUAsmGFX90a19 AMDGPU/AMDGPUAsmGFX94020 AMDGPU/AMDGPUAsmGFX1021 AMDGPU/AMDGPUAsmGFX101122 AMDGPU/AMDGPUAsmGFX101323 AMDGPU/AMDGPUAsmGFX103024 AMDGPU/AMDGPUAsmGFX1125 AMDGPU/AMDGPUAsmGFX1226 AMDGPUModifierSyntax27 AMDGPUOperandSyntax28 AMDGPUInstructionSyntax29 AMDGPUInstructionNotation30 AMDGPUDwarfExtensionsForHeterogeneousDebugging31 AMDGPUDwarfExtensionAllowLocationDescriptionOnTheDwarfExpressionStack/AMDGPUDwarfExtensionAllowLocationDescriptionOnTheDwarfExpressionStack32 33Introduction34============35 36The AMDGPU backend provides ISA code generation for AMD GPUs, starting with the37R600 family up until the current GCN families. It lives in the38``llvm/lib/Target/AMDGPU`` directory.39 40LLVM41====42 43.. _amdgpu-target-triples:44 45Target Triples46--------------47 48Use the Clang option ``-target <Architecture>-<Vendor>-<OS>-<Environment>``49to specify the target triple:50 51 .. table:: AMDGPU Architectures52 :name: amdgpu-architecture-table53 54 ============ ==============================================================55 Architecture Description56 ============ ==============================================================57 ``r600`` AMD GPUs HD2XXX-HD6XXX for graphics and compute shaders.58 ``amdgcn`` AMD GPUs GCN GFX6 onwards for graphics and compute shaders.59 ============ ==============================================================60 61 .. table:: AMDGPU Vendors62 :name: amdgpu-vendor-table63 64 ============ ==============================================================65 Vendor Description66 ============ ==============================================================67 ``amd`` Can be used for all AMD GPU usage.68 ``mesa`` Can be used if the OS is ``mesa3d``.69 ============ ==============================================================70 71 .. table:: AMDGPU Operating Systems72 :name: amdgpu-os73 74 ============== ============================================================75 OS Description76 ============== ============================================================77 *<empty>* Defaults to the *unknown* OS.78 ``amdhsa`` Compute kernels executed on HSA [HSA]_ compatible runtimes79 such as:80 81 - AMD's ROCm™ runtime [AMD-ROCm]_ using the *rocm-amdhsa*82 loader on Linux. See *AMD ROCm Platform Release Notes*83 [AMD-ROCm-Release-Notes]_ for supported hardware and84 software.85 - AMD's PAL runtime using the *pal-amdhsa* loader on86 Windows.87 88 ``amdpal`` Graphic shaders and compute kernels executed on AMD's PAL89 runtime using the *pal-amdpal* loader on Windows and Linux90 Pro.91 ``mesa3d`` Graphic shaders and compute kernels executed on AMD's Mesa92 3D runtime using the *mesa-mesa3d* loader on Linux.93 ============== ============================================================94 95 .. table:: AMDGPU Environments96 :name: amdgpu-environment-table97 98 ============ ==============================================================99 Environment Description100 ============ ==============================================================101 *<empty>* Default.102 ============ ==============================================================103 104.. _amdgpu-processors:105 106Processors107----------108 109Use the Clang options ``-mcpu=<target-id>`` or ``--offload-arch=<target-id>`` to110specify the AMDGPU processor together with optional target features. See111:ref:`amdgpu-target-id` and :ref:`amdgpu-target-features` for AMD GPU target112specific information.113 114Every processor supports every OS ABI (see :ref:`amdgpu-os`) with the following exceptions:115 116* ``amdhsa`` is not supported in ``r600`` architecture (see :ref:`amdgpu-architecture-table`).117 118 119 .. table:: AMDGPU Processors120 :name: amdgpu-processor-table121 122 =========== =============== ============ ===== ================= =============== =============== ======================123 Processor Alternative Target dGPU/ Target Target OS Support Example124 Processor Triple APU Features Properties *(see* Products125 Architecture Supported `amdgpu-os`_126 *and127 corresponding128 runtime release129 notes for130 current131 information and132 level of133 support)*134 =========== =============== ============ ===== ================= =============== =============== ======================135 **Radeon HD 2000/3000 Series (R600)** [AMD-RADEON-HD-2000-3000]_136 -----------------------------------------------------------------------------------------------------------------------137 ``r600`` ``r600`` dGPU - Does not138 support139 generic140 address141 space142 ``r630`` ``r600`` dGPU - Does not143 support144 generic145 address146 space147 ``rs880`` ``r600`` dGPU - Does not148 support149 generic150 address151 space152 ``rv670`` ``r600`` dGPU - Does not153 support154 generic155 address156 space157 **Radeon HD 4000 Series (R700)** [AMD-RADEON-HD-4000]_158 -----------------------------------------------------------------------------------------------------------------------159 ``rv710`` ``r600`` dGPU - Does not160 support161 generic162 address163 space164 ``rv730`` ``r600`` dGPU - Does not165 support166 generic167 address168 space169 ``rv770`` ``r600`` dGPU - Does not170 support171 generic172 address173 space174 **Radeon HD 5000 Series (Evergreen)** [AMD-RADEON-HD-5000]_175 -----------------------------------------------------------------------------------------------------------------------176 ``cedar`` ``r600`` dGPU - Does not177 support178 generic179 address180 space181 ``cypress`` ``r600`` dGPU - Does not182 support183 generic184 address185 space186 ``juniper`` ``r600`` dGPU - Does not187 support188 generic189 address190 space191 ``redwood`` ``r600`` dGPU - Does not192 support193 generic194 address195 space196 ``sumo`` ``r600`` dGPU - Does not197 support198 generic199 address200 space201 **Radeon HD 6000 Series (Northern Islands)** [AMD-RADEON-HD-6000]_202 -----------------------------------------------------------------------------------------------------------------------203 ``barts`` ``r600`` dGPU - Does not204 support205 generic206 address207 space208 ``caicos`` ``r600`` dGPU - Does not209 support210 generic211 address212 space213 ``cayman`` ``r600`` dGPU - Does not214 support215 generic216 address217 space218 ``turks`` ``r600`` dGPU - Does not219 support220 generic221 address222 space223 **GCN GFX6 (Southern Islands (SI))** [AMD-GCN-GFX6]_224 -----------------------------------------------------------------------------------------------------------------------225 ``gfx600`` - ``tahiti`` ``amdgcn`` dGPU - Does not - *pal-amdpal*226 support227 generic228 address229 space230 ``gfx601`` - ``pitcairn`` ``amdgcn`` dGPU - Does not - *pal-amdpal*231 - ``verde`` support232 generic233 address234 space235 ``gfx602`` - ``hainan`` ``amdgcn`` dGPU - Does not - *pal-amdpal*236 - ``oland`` support237 generic238 address239 space240 **GCN GFX7 (Sea Islands (CI))** [AMD-GCN-GFX7]_241 -----------------------------------------------------------------------------------------------------------------------242 ``gfx700`` - ``kaveri`` ``amdgcn`` APU - Offset - *rocm-amdhsa* - A6-7000243 flat - *pal-amdhsa* - A6 Pro-7050B244 scratch - *pal-amdpal* - A8-7100245 - A8 Pro-7150B246 - A10-7300247 - A10 Pro-7350B248 - FX-7500249 - A8-7200P250 - A10-7400P251 - FX-7600P252 ``gfx701`` - ``hawaii`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - FirePro W8100253 flat - *pal-amdhsa* - FirePro W9100254 scratch - *pal-amdpal* - FirePro S9150255 - FirePro S9170256 ``gfx702`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon R9 290257 flat - *pal-amdhsa* - Radeon R9 290x258 scratch - *pal-amdpal* - Radeon R390259 - Radeon R390x260 ``gfx703`` - ``kabini`` ``amdgcn`` APU - Offset - *pal-amdhsa* - E1-2100261 - ``mullins`` flat - *pal-amdpal* - E1-2200262 scratch - E1-2500263 - E2-3000264 - E2-3800265 - A4-5000266 - A4-5100267 - A6-5200268 - A4 Pro-3340B269 ``gfx704`` - ``bonaire`` ``amdgcn`` dGPU - Offset - *pal-amdhsa* - Radeon HD 7790270 flat - *pal-amdpal* - Radeon HD 8770271 scratch - R7 260272 - R7 260X273 ``gfx705`` ``amdgcn`` APU - Offset - *pal-amdhsa* *TBA*274 flat - *pal-amdpal*275 scratch .. TODO::276 277 Add product278 names.279 280 **GCN GFX8 (Volcanic Islands (VI))** [AMD-GCN-GFX8]_281 -----------------------------------------------------------------------------------------------------------------------282 ``gfx801`` - ``carrizo`` ``amdgcn`` APU - xnack - Offset - *rocm-amdhsa* - A6-8500P283 flat - *pal-amdhsa* - Pro A6-8500B284 scratch - *pal-amdpal* - A8-8600P285 - Pro A8-8600B286 - FX-8800P287 - Pro A12-8800B288 - A10-8700P289 - Pro A10-8700B290 - A10-8780P291 - A10-9600P292 - A10-9630P293 - A12-9700P294 - A12-9730P295 - FX-9800P296 - FX-9830P297 - E2-9010298 - A6-9210299 - A9-9410300 ``gfx802`` - ``iceland`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon R9 285301 - ``tonga`` flat - *pal-amdhsa* - Radeon R9 380302 scratch - *pal-amdpal* - Radeon R9 385303 ``gfx803`` - ``fiji`` ``amdgcn`` dGPU - *rocm-amdhsa* - Radeon R9 Nano304 - *pal-amdhsa* - Radeon R9 Fury305 - *pal-amdpal* - Radeon R9 FuryX306 - Radeon Pro Duo307 - FirePro S9300x2308 - Radeon Instinct MI8309 \ - ``polaris10`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon RX 470310 flat - *pal-amdhsa* - Radeon RX 480311 scratch - *pal-amdpal* - Radeon Instinct MI6312 \ - ``polaris11`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - Radeon RX 460313 flat - *pal-amdhsa*314 scratch - *pal-amdpal*315 ``gfx805`` - ``tongapro`` ``amdgcn`` dGPU - Offset - *rocm-amdhsa* - FirePro S7150316 flat - *pal-amdhsa* - FirePro S7100317 scratch - *pal-amdpal* - FirePro W7100318 - Mobile FirePro319 M7170320 ``gfx810`` - ``stoney`` ``amdgcn`` APU - xnack - Offset - *rocm-amdhsa* *TBA*321 flat - *pal-amdhsa*322 scratch - *pal-amdpal* .. TODO::323 324 Add product325 names.326 327 **GCN GFX9 (Vega)** [AMD-GCN-GFX900-GFX904-VEGA]_ [AMD-GCN-GFX906-VEGA7NM]_ [AMD-GCN-GFX908-CDNA1]_ [AMD-GCN-GFX90A-CDNA2]_ [AMD-GCN-GFX942-CDNA3]_328 -----------------------------------------------------------------------------------------------------------------------329 ``gfx900`` ``amdgcn`` dGPU - xnack - Absolute - *rocm-amdhsa* - Radeon Vega330 flat - *pal-amdhsa* Frontier Edition331 scratch - *pal-amdpal* - Radeon RX Vega 56332 - Radeon RX Vega 64333 - Radeon RX Vega 64334 Liquid335 - Radeon Instinct MI25336 ``gfx902`` ``amdgcn`` APU - xnack - Absolute - *rocm-amdhsa* - Ryzen 3 2200G337 flat - *pal-amdhsa* - Ryzen 5 2400G338 scratch - *pal-amdpal*339 ``gfx904`` ``amdgcn`` dGPU - xnack - *rocm-amdhsa* *TBA*340 - *pal-amdhsa*341 - *pal-amdpal* .. TODO::342 343 Add product344 names.345 346 ``gfx906`` ``amdgcn`` dGPU - sramecc - Absolute - *rocm-amdhsa* - Radeon Instinct MI50347 - xnack flat - *pal-amdhsa* - Radeon Instinct MI60348 scratch - *pal-amdpal* - Radeon VII349 - Radeon Pro VII350 ``gfx908`` ``amdgcn`` dGPU - sramecc - *rocm-amdhsa* - AMD Instinct MI100 Accelerator351 - xnack - Absolute352 flat353 scratch354 ``gfx909`` ``amdgcn`` APU - xnack - Absolute - *pal-amdpal* *TBA*355 flat356 scratch .. TODO::357 358 Add product359 names.360 361 ``gfx90a`` ``amdgcn`` dGPU - sramecc - Absolute - *rocm-amdhsa* - AMD Instinct MI210 Accelerator362 - tgsplit flat - *rocm-amdhsa* - AMD Instinct MI250 Accelerator363 - xnack scratch - *rocm-amdhsa* - AMD Instinct MI250X Accelerator364 - kernarg preload - Packed365 (except MI210) work-item366 IDs367 368 ``gfx90c`` ``amdgcn`` APU - xnack - Absolute - *pal-amdpal* - Ryzen 7 4700G369 flat - Ryzen 7 4700GE370 scratch - Ryzen 5 4600G371 - Ryzen 5 4600GE372 - Ryzen 3 4300G373 - Ryzen 3 4300GE374 - Ryzen Pro 4000G375 - Ryzen 7 Pro 4700G376 - Ryzen 7 Pro 4750GE377 - Ryzen 5 Pro 4650G378 - Ryzen 5 Pro 4650GE379 - Ryzen 3 Pro 4350G380 - Ryzen 3 Pro 4350GE381 382 ``gfx942`` ``amdgcn`` dGPU - sramecc - Architected - AMD Instinct MI300X383 - tgsplit flat - AMD Instinct MI300A384 - xnack scratch385 - kernarg preload - Packed386 work-item387 IDs388 389 ``gfx950`` ``amdgcn`` dGPU - sramecc - Architected *TBA*390 - tgsplit flat391 - xnack scratch .. TODO::392 - kernarg preload - Packed393 work-item Add product394 IDs names.395 396 **GCN GFX10.1 (RDNA 1)** [AMD-GCN-GFX10-RDNA1]_397 -----------------------------------------------------------------------------------------------------------------------398 ``gfx1010`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon Pro 5600 XT399 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 5600M400 - xnack scratch - *pal-amdpal* - Radeon RX 5700401 - Radeon RX 5700 XT402 ``gfx1011`` ``amdgcn`` dGPU - cumode - *rocm-amdhsa* - Radeon Pro V520403 - wavefrontsize64 - Absolute - *pal-amdhsa* - Radeon Pro 5600M404 - xnack flat - *pal-amdpal*405 scratch406 ``gfx1012`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 5500407 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 5500 XT408 - xnack scratch - *pal-amdpal*409 ``gfx1013`` ``amdgcn`` APU - cumode - Absolute - *rocm-amdhsa* *TBA*410 - wavefrontsize64 flat - *pal-amdhsa*411 - xnack scratch - *pal-amdpal* .. TODO::412 413 Add product414 names.415 416 **GCN GFX10.3 (RDNA 2)** [AMD-GCN-GFX10-RDNA2]_417 -----------------------------------------------------------------------------------------------------------------------418 ``gfx1030`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 6800419 - wavefrontsize64 flat - *pal-amdhsa* - Radeon RX 6800 XT420 scratch - *pal-amdpal* - Radeon RX 6900 XT421 - Radeon PRO W6800422 - Radeon PRO V620423 ``gfx1031`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* - Radeon RX 6700 XT424 - wavefrontsize64 flat - *pal-amdhsa*425 scratch - *pal-amdpal*426 ``gfx1032`` ``amdgcn`` dGPU - cumode - Absolute - *rocm-amdhsa* *TBA*427 - wavefrontsize64 flat - *pal-amdhsa*428 scratch - *pal-amdpal* .. TODO::429 430 Add product431 names.432 433 ``gfx1033`` ``amdgcn`` APU - cumode - Absolute - *pal-amdpal* *TBA*434 - wavefrontsize64 flat435 scratch .. TODO::436 437 Add product438 names.439 ``gfx1034`` ``amdgcn`` dGPU - cumode - Absolute - *pal-amdpal* *TBA*440 - wavefrontsize64 flat441 scratch .. TODO::442 443 Add product444 names.445 446 ``gfx1035`` ``amdgcn`` APU - cumode - Absolute - *pal-amdpal* *TBA*447 - wavefrontsize64 flat448 scratch .. TODO::449 Add product450 names.451 452 ``gfx1036`` ``amdgcn`` APU - cumode - Absolute - *pal-amdpal* *TBA*453 - wavefrontsize64 flat454 scratch .. TODO::455 456 Add product457 names.458 459 **GCN GFX11 (RDNA 3)** [AMD-GCN-GFX11-RDNA3]_460 -----------------------------------------------------------------------------------------------------------------------461 ``gfx1100`` ``amdgcn`` dGPU - cumode - Architected - *pal-amdpal* - Radeon PRO W7900 Dual Slot462 - wavefrontsize64 flat - Radeon PRO W7900463 scratch - Radeon PRO W7800464 - Packed - Radeon RX 7900 XTX465 work-item - Radeon RX 7900 XT466 IDs - Radeon RX 7900 GRE467 468 ``gfx1101`` ``amdgcn`` dGPU - cumode - Architected *TBA*469 - wavefrontsize64 flat470 scratch .. TODO::471 - Packed472 work-item Add product473 IDs names.474 475 ``gfx1102`` ``amdgcn`` dGPU - cumode - Architected *TBA*476 - wavefrontsize64 flat477 scratch .. TODO::478 - Packed479 work-item Add product480 IDs names.481 482 ``gfx1103`` ``amdgcn`` APU - cumode - Architected *TBA*483 - wavefrontsize64 flat484 scratch .. TODO::485 - Packed486 work-item Add product487 IDs names.488 489 **GCN GFX11 (RDNA 3.5)** [AMD-GCN-GFX11-RDNA3.5]_490 -----------------------------------------------------------------------------------------------------------------------491 ``gfx1150`` ``amdgcn`` APU - cumode - Architected Radeon 890M492 - wavefrontsize64 flat493 scratch .. TODO::494 - Packed495 work-item Add product496 IDs names.497 498 ``gfx1151`` ``amdgcn`` APU - cumode - Architected Radeon 8060S499 - wavefrontsize64 flat500 scratch .. TODO::501 - Packed502 work-item Add product503 IDs names.504 505 ``gfx1152`` ``amdgcn`` APU - cumode - Architected Radeon 860M506 - wavefrontsize64 flat507 scratch .. TODO::508 - Packed509 work-item Add product510 IDs names.511 512 ``gfx1153`` ``amdgcn`` APU - cumode - Architected *TBA*513 - wavefrontsize64 flat514 scratch .. TODO::515 - Packed516 work-item Add product517 IDs names.518 519 **GCN GFX12 (RDNA 4)** [AMD-GCN-GFX12-RDNA4]_520 -----------------------------------------------------------------------------------------------------------------------521 ``gfx1200`` ``amdgcn`` dGPU - cumode - Architected - Radeon RX 9060522 - wavefrontsize64 flat - Radeon RX 9060 XT523 scratch524 - Packed525 work-item526 IDs527 528 ``gfx1201`` ``amdgcn`` dGPU - cumode - Architected - Radeon RX 9070529 - wavefrontsize64 flat - Radeon RX 9070 XT530 scratch - Radeon RX 9070 GRE531 - Packed532 work-item533 IDs534 535 ``gfx1250`` ``amdgcn`` APU - Architected *TBA*536 flat537 scratch .. TODO::538 - Packed539 work-item Add product540 IDs names.541 - Globally542 Accessible543 Scratch544 - Workgroup545 Clusters546 547 ``gfx1251`` ``amdgcn`` APU - Architected *TBA*548 flat549 scratch .. TODO::550 - Packed551 work-item Add product552 IDs names.553 - Globally554 Accessible555 Scratch556 - Workgroup557 Clusters558 559 =========== =============== ============ ===== ================= =============== =============== ======================560 561Generic processors allow execution of a single code object on any of the processors that562it supports. Such code objects may not perform as well as those for the non-generic processors.563 564Generic processors are only available on code object V6 and above (see :ref:`amdgpu-elf-code-object`).565 566Generic processor code objects are versioned. See :ref:`amdgpu-generic-processor-versioning` for more information on how versioning works.567 568 .. table:: AMDGPU Generic Processors569 :name: amdgpu-generic-processor-table570 571 ==================== ============== ================= ================== ================= =================================572 Processor Target Supported Target Features Target Properties Target Restrictions573 Triple Processors Supported574 Architecture575 576 ==================== ============== ================= ================== ================= =================================577 ``gfx9-generic`` ``amdgcn`` - ``gfx900`` - xnack - Absolute flat - ``v_mad_mix`` instructions578 - ``gfx902`` scratch are not available on579 - ``gfx904`` ``gfx900``, ``gfx902``,580 - ``gfx906`` ``gfx909``, ``gfx90c``581 - ``gfx909`` - ``v_fma_mix`` instructions582 - ``gfx90c`` are not available on ``gfx904``583 - sramecc is not available on584 ``gfx906``585 - The following instructions586 are not available on ``gfx906``:587 588 - ``v_fmac_f32``589 - ``v_xnor_b32``590 - ``v_dot4_i32_i8``591 - ``v_dot8_i32_i4``592 - ``v_dot2_i32_i16``593 - ``v_dot2_u32_u16``594 - ``v_dot4_u32_u8``595 - ``v_dot8_u32_u4``596 - ``v_dot2_f32_f16``597 598 599 ``gfx9-4-generic`` ``amdgcn`` - ``gfx942`` - sramecc - Architected FP8 and BF8 instructions,600 - ``gfx950`` - tgsplit flat scratch FP8 and BF8 conversion601 - xnack - Packed instructions, as well as602 - kernarg preload work-item instructions with XF32 format603 IDs support are not available.604 605 ``gfx10-1-generic`` ``amdgcn`` - ``gfx1010`` - xnack - Absolute flat - The following instructions are606 - ``gfx1011`` - wavefrontsize64 scratch not available on ``gfx1011``607 - ``gfx1012`` - cumode and ``gfx1012``608 - ``gfx1013``609 - ``v_dot4_i32_i8``610 - ``v_dot8_i32_i4``611 - ``v_dot2_i32_i16``612 - ``v_dot2_u32_u16``613 - ``v_dot2c_f32_f16``614 - ``v_dot4c_i32_i8``615 - ``v_dot4_u32_u8``616 - ``v_dot8_u32_u4``617 - ``v_dot2_f32_f16``618 619 - BVH Ray Tracing instructions620 are not available on621 ``gfx1013``622 623 624 ``gfx10-3-generic`` ``amdgcn`` - ``gfx1030`` - wavefrontsize64 - Absolute flat No restrictions.625 - ``gfx1031`` - cumode scratch626 - ``gfx1032``627 - ``gfx1033``628 - ``gfx1034``629 - ``gfx1035``630 - ``gfx1036``631 632 633 ``gfx11-generic`` ``amdgcn`` - ``gfx1100`` - wavefrontsize64 - Architected Various codegen pessimizations634 - ``gfx1101`` - cumode flat scratch are applied to work around some635 - ``gfx1102`` - Packed hazards specific to some targets636 - ``gfx1103`` work-item within this family.637 - ``gfx1150`` IDs638 - ``gfx1151``639 - ``gfx1152``640 - ``gfx1153`` Not all VGPRs can be used on:641 642 - ``gfx1100``643 - ``gfx1101``644 - ``gfx1151``645 646 SALU floating point instructions647 are not available on:648 649 - ``gfx1100``650 - ``gfx1101``651 - ``gfx1102``652 - ``gfx1103``653 654 SGPRs are not supported for src1655 in dpp instructions for:656 657 - ``gfx1100``658 - ``gfx1101``659 - ``gfx1102``660 - ``gfx1103``661 662 663 ``gfx12-generic`` ``amdgcn`` - ``gfx1200`` - wavefrontsize64 - Architected No restrictions.664 - ``gfx1201`` - cumode flat scratch665 - Packed666 work-item667 IDs668 ==================== ============== ================= ================== ================= =================================669 670.. _amdgpu-generic-processor-versioning:671 672Generic Processor Versioning673----------------------------674 675Generic processor (see :ref:`amdgpu-generic-processor-table`) code objects are versioned (see :ref:`amdgpu-elf-header-e_flags-table-v6-onwards`) between 1 and 255.676The version of non-generic code objects is always set to 0.677 678For a generic code object, adding a new supported processor may require the code generated for the generic target to be changed679so it can continue to execute on the previously supported processors as well as on the new one.680When this happens, the generic code object version number is incremented at the same time as the generic target is updated.681 682Each supported processor of a generic target is mapped to the version it was introduced in.683A generic code object can execute on a supported processor if the version of the code object being loaded is684greater than or equal to the version in which the processor was added to the generic target.685 686.. _amdgpu-target-features:687 688Target Features689---------------690 691Target features control how code is generated to support certain692processor specific features. Not all target features are supported by693all processors. The runtime must ensure that the features supported by694the device used to execute the code match the features enabled when695generating the code. A mismatch of features may result in incorrect696execution, or a reduction in performance.697 698The target features supported by each processor are listed in699:ref:`amdgpu-processors`.700 701Target features are controlled by exactly one of the following Clang702options:703 704``-mcpu=<target-id>`` or ``--offload-arch=<target-id>``705 706 The ``-mcpu`` and ``--offload-arch`` can specify the target feature as707 optional components of the target ID. If omitted, the target feature has the708 ``any`` value. See :ref:`amdgpu-target-id`.709 710``-m[no-]<target-feature>``711 712 Target features not specified by the target ID are specified using a713 separate option. These target features can have an ``on`` or ``off``714 value. ``on`` is specified by omitting the ``no-`` prefix, and715 ``off`` is specified by including the ``no-`` prefix. The default716 if not specified is ``off``.717 718For example:719 720``-mcpu=gfx908:xnack+``721 Enable the ``xnack`` feature.722``-mcpu=gfx908:xnack-``723 Disable the ``xnack`` feature.724``-mcumode``725 Enable the ``cumode`` feature.726``-mno-cumode``727 Disable the ``cumode`` feature.728 729 .. table:: AMDGPU Target Features730 :name: amdgpu-target-features-table731 732 =============== ============================ ==================================================733 Target Feature Clang Option to Control Description734 Name735 =============== ============================ ==================================================736 cumode - ``-m[no-]cumode`` Control the wavefront execution mode used737 when generating code for kernels. When disabled738 native WGP wavefront execution mode is used,739 when enabled CU wavefront execution mode is used740 (see :ref:`amdgpu-amdhsa-memory-model`).741 742 sramecc - ``-mcpu`` If specified, generate code that can only be743 - ``--offload-arch`` loaded and executed in a process that has a744 matching setting for SRAMECC.745 746 If not specified for code object V2 to V3, generate747 code that can be loaded and executed in a process748 with SRAMECC enabled.749 750 If not specified for code object V4 or above, generate751 code that can be loaded and executed in a process752 with either setting of SRAMECC.753 754 tgsplit ``-m[no-]tgsplit`` Enable/disable generating code that assumes755 work-groups are launched in threadgroup split mode.756 When enabled the waves of a work-group may be757 launched in different CUs.758 759 wavefrontsize64 - ``-m[no-]wavefrontsize64`` Control the wavefront size used when760 generating code for kernels. When disabled761 native wavefront size 32 is used, when enabled762 wavefront size 64 is used.763 764 xnack - ``-mcpu`` If specified, generate code that can only be765 - ``--offload-arch`` loaded and executed in a process that has a766 matching setting for XNACK replay.767 768 If not specified for code object V2 to V3, generate769 code that can be loaded and executed in a process770 with XNACK replay enabled.771 772 If not specified for code object V4 or above, generate773 code that can be loaded and executed in a process774 with either setting of XNACK replay.775 776 XNACK replay can be used for demand paging and777 page migration. If enabled in the device, then if778 a page fault occurs the code may execute779 incorrectly unless generated with XNACK replay780 enabled, or generated for code object V4 or above without781 specifying XNACK replay. Executing code that was782 generated with XNACK replay enabled, or generated783 for code object V4 or above without specifying XNACK replay,784 on a device that does not have XNACK replay785 enabled will execute correctly but may be less786 performant than code generated for XNACK replay787 disabled.788 789 =============== ============================ ==================================================790 791.. _amdgpu-target-id:792 793Target ID794---------795 796AMDGPU supports target IDs. See `Clang Offload Bundler797<https://clang.llvm.org/docs/ClangOffloadBundler.html>`_ for a general798description. The AMDGPU target specific information is:799 800**processor**801 Is an AMDGPU processor or alternative processor name specified in802 :ref:`amdgpu-processor-table`. The non-canonical form target ID allows both803 the primary processor and alternative processor names. The canonical form804 target ID only allows the primary processor name.805 806**target-feature**807 Is a target feature name specified in :ref:`amdgpu-target-features-table` that808 is supported by the processor. The target features supported by each processor809 is specified in :ref:`amdgpu-processor-table`. Those that can be specified in810 a target ID are marked as being controlled by ``-mcpu`` and811 ``--offload-arch``. Each target feature must appear at most once in a target812 ID. The non-canonical form target ID allows the target features to be813 specified in any order. The canonical form target ID requires the target814 features to be specified in alphabetical order.815 816.. _amdgpu-target-id-v2-v3:817 818Code Object V2 to V3 Target ID819~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~820 821The target ID syntax for code object V2 to V3 is the same as defined in `Clang822Offload Bundler <https://clang.llvm.org/docs/ClangOffloadBundler.html>`_ except823when used in the :ref:`amdgpu-assembler-directive-amdgcn-target` assembler824directive and the bundle entry ID. In those cases it has the following BNF825syntax:826 827.. code::828 829 <target-id> ::== <processor> ( "+" <target-feature> )*830 831Where a target feature is omitted if *Off* and present if *On* or *Any*.832 833.. note::834 835 The code object V2 to V3 cannot represent *Any* and treats it the same as836 *On*.837 838.. _amdgpu-embedding-bundled-objects:839 840Embedding Bundled Code Objects841------------------------------842 843AMDGPU supports the HIP and OpenMP languages that perform code object embedding844as described in `Clang Offload Bundler845<https://clang.llvm.org/docs/ClangOffloadBundler.html>`_.846 847.. note::848 849 The target ID syntax used for code object V2 to V3 for a bundle entry ID850 differs from that used elsewhere. See :ref:`amdgpu-target-id-v2-v3`.851 852.. _amdgpu-address-spaces:853 854Address Spaces855--------------856 857The AMDGPU architecture supports a number of memory address spaces. The address858space names use the OpenCL standard names, with some additions.859 860The AMDGPU address spaces correspond to target architecture specific LLVM861address space numbers used in LLVM IR.862 863The AMDGPU address spaces are described in864:ref:`amdgpu-address-spaces-table`. Only 64-bit process address spaces are865supported for the ``amdgcn`` target.866 867 .. table:: AMDGPU Address Spaces868 :name: amdgpu-address-spaces-table869 870 ===================================== =============== =========== ================ ======= ============================871 .. 64-Bit Process Address Space872 ------------------------------------- --------------- ----------- ---------------- ------------------------------------873 Address Space Name LLVM IR Address HSA Segment Hardware Address NULL Value874 Space Number Name Name Size875 ===================================== =============== =========== ================ ======= ============================876 Generic 0 flat flat 64 0x0000000000000000877 Global 1 global global 64 0x0000000000000000878 Region 2 N/A GDS 32 *not implemented for AMDHSA*879 Local 3 group LDS 32 0xFFFFFFFF880 Constant 4 constant *same as global* 64 0x0000000000000000881 Private 5 private scratch 32 0xFFFFFFFF882 Constant 32-bit 6 *TODO* 0x00000000883 Buffer Fat Pointer 7 N/A N/A 160 0884 Buffer Resource 8 N/A V# 128 0x00000000000000000000000000000000885 Buffer Strided Pointer (experimental) 9 *TODO*886 *reserved for future use* 10887 *reserved for future use* 11888 *reserved for downstream use (LLPC)* 12889 Streamout Registers 128 N/A GS_REGS890 ===================================== =============== =========== ================ ======= ============================891 892**Generic**893 The generic address space is supported unless the *Target Properties* column894 of :ref:`amdgpu-processor-table` specifies *Does not support generic address895 space*.896 897 The generic address space uses the hardware flat address support for two fixed898 ranges of virtual addresses (the private and local apertures), that are899 outside the range of addressable global memory, to map from a flat address to900 a private or local address. This uses FLAT instructions that can take a flat901 address and access global, private (scratch), and group (LDS) memory depending902 on if the address is within one of the aperture ranges.903 904 Flat access to scratch requires hardware aperture setup and setup in the905 kernel prologue (see :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat906 access to LDS requires hardware aperture setup and M0 (GFX7-GFX8) register907 setup (see :ref:`amdgpu-amdhsa-kernel-prolog-m0`).908 909 To convert between a private or group address space address (termed a segment910 address) and a flat address, the base address of the corresponding aperture911 can be used. For GFX7-GFX8 these are available in the912 :ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with913 Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For914 GFX9-GFX11 the aperture base addresses are directly available as inline915 constant registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``.916 In 64-bit address mode the aperture sizes are 2^32 bytes and the base is917 aligned to 2^32 which makes it easier to convert from flat to segment or918 segment to flat.919 920 A global address space address has the same value when used as a flat address921 so no conversion is needed.922 923**Global and Constant**924 The global and constant address spaces both use global virtual addresses,925 which are the same virtual address space used by the CPU. However, some926 virtual addresses may only be accessible to the CPU, some only accessible927 by the GPU, and some by both.928 929 Using the constant address space indicates that the data will not change930 during the execution of the kernel. This allows scalar read instructions to931 be used. As the constant address space could only be modified on the host932 side, a generic pointer loaded from the constant address space is safe to be933 assumed as a global pointer since only the device global memory is visible934 and managed on the host side. The vector and scalar L1 caches are invalidated935 of volatile data before each kernel dispatch execution to allow constant936 memory to change values between kernel dispatches.937 938**Region**939 The region address space uses the hardware Global Data Store (GDS). All940 wavefronts executing on the same device will access the same memory for any941 given region address. However, the same region address accessed by wavefronts942 executing on different devices will access different memory. It is higher943 performance than global memory. It is allocated by the runtime. The data944 store (DS) instructions can be used to access it.945 946**Local**947 The local address space uses the hardware Local Data Store (LDS) which is948 automatically allocated when the hardware creates the wavefronts of a949 work-group, and freed when all the wavefronts of a work-group have950 terminated. All wavefronts belonging to the same work-group will access the951 same memory for any given local address. However, the same local address952 accessed by wavefronts belonging to different work-groups will access953 different memory. It is higher performance than global memory. The data store954 (DS) instructions can be used to access it.955 956**Private**957 The private address space uses the hardware scratch memory support which958 automatically allocates memory when it creates a wavefront and frees it when959 a wavefronts terminates. The memory accessed by a lane of a wavefront for any960 given private address will be different to the memory accessed by another lane961 of the same or different wavefront for the same private address.962 963 If a kernel dispatch uses scratch, then the hardware allocates memory from a964 pool of backing memory allocated by the runtime for each wavefront. The lanes965 of the wavefront access this using dword (4 byte) interleaving. The mapping966 used from private address to backing memory address is:967 968 ``wavefront-scratch-base +969 ((private-address / 4) * wavefront-size * 4) +970 (wavefront-lane-id * 4) + (private-address % 4)``971 972 If each lane of a wavefront accesses the same private address, the973 interleaving results in adjacent dwords being accessed and hence requires974 fewer cache lines to be fetched.975 976 There are different ways that the wavefront scratch base address is977 determined by a wavefront (see978 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`).979 980 Scratch memory can be accessed in an interleaved manner using buffer981 instructions with the scratch buffer descriptor and per wavefront scratch982 offset, by the scratch instructions, or by flat instructions. Multi-dword983 access is not supported except by flat and scratch instructions in984 GFX9-GFX11.985 986 On targets without "Globally Accessible Scratch" (introduced in GFX125x), code that987 manipulates the stack values in other lanes of a wavefront, such as by988 ``addrspacecast``-ing stack pointers to generic ones and taking offsets that reach other989 lanes or by explicitly constructing the scratch buffer descriptor, triggers undefined990 behavior when it modifies the scratch values of other lanes. The compiler may assume991 that such modifications do not occur for such targets.992 993 When using code object V5 ``LIBOMPTARGET_STACK_SIZE`` may be used to provide the994 private segment size in bytes, for cases where a dynamic stack is used.995 996**Constant 32-bit**997 *TODO*998 999**Buffer Fat Pointer**1000 The buffer fat pointer is an experimental address space that is currently1001 unsupported in the backend. It exposes a non-integral pointer that is in1002 the future intended to support the modelling of 128-bit buffer descriptors1003 plus a 32-bit offset into the buffer (in total encapsulating a 160-bit1004 *pointer*), allowing normal LLVM load/store/atomic operations to be used to1005 model the buffer descriptors used heavily in graphics workloads targeting1006 the backend.1007 1008 The buffer descriptor used to construct a buffer fat pointer must be *raw*:1009 the stride must be 0, the "add tid" flag must be 0, the swizzle enable bits1010 must be off, and the extent must be measured in bytes. (On subtargets where1011 bounds checking may be disabled, buffer fat pointers may choose to enable1012 it or not). The cache swizzle support introduced in gfx942 may be used.1013 1014 These pointers can be created by ``addrspacecast`` from a buffer resource1015 (``ptr addrspace(8)```) or by using `llvm.amdgcn.make.buffer.rsrc` to produce a1016 ``ptr addrspace(7)`` directly, which produces a buffer fat pointer with an initial1017 offset of 0 and prevents the address space cast from being rewritten away.1018 1019 The ``align`` attribute on operations from buffer fat pointers is deemed to apply1020 to all componenents of the pointer - that is, an ``align 4`` load is expected to1021 both have the offset be a multiple of 4 and to have a base pointer with an1022 alignment of 4.1023 1024 This componentwise definition of alignment is needed to allow for promotion of1025 aligned loads to ``s_buffer_load``, which requires that both the base pointer and1026 offset be appropriately aligned.1027 1028**Buffer Resource**1029 The buffer resource pointer, in address space 8, is the newer form1030 for representing buffer descriptors in AMDGPU IR, replacing their1031 previous representation as ``<4 x i32>``. It is a non-integral pointer1032 that represents a 128-bit buffer descriptor resource (``V#``).1033 1034 Since, in general, a buffer resource supports complex addressing modes that cannot1035 be easily represented in LLVM (such as implicit swizzled access to structured1036 buffers), performing address computations such as ``getelementptr`` is not1037 recommended on ``ptr addrspace(8)``s (if such computations are performed, the1038 offset must be wavefront-uniform.) Note that such a usage of GEP is currently1039 **unimplemented** in the backend, as it would require a wrapping 48-bit1040 addition. Buffer resources may be passed to AMDGPU buffer intrinsics, and they1041 may be converted to and from ``i128``.1042 1043 Casting a buffer resource to a buffer fat pointer is permitted and adds an offset1044 of 0.1045 1046 Buffer resources can be created from 64-bit pointers (which should be either1047 generic or global) using the ``llvm.amdgcn.make.buffer.rsrc`` intrinsic, which1048 takes the pointer, which becomes the base of the resource,1049 the 16-bit stride (and swzizzle control) field stored in bits `63:48` of a `V#`,1050 the 32-bit NumRecords/extent field (bits `95:64`), and the 32-bit flags field1051 (bits `127:96`). The specific interpretation of these fields varies by the1052 target architecture and is detailed in the ISA descriptions.1053 1054 On gfx1250, the base pointer is instead truncated to 57 bits and the NumRecords1055 field is 45 bits, which necessitated a change to ``make.buffer.rsrcs``'s arguments1056 in order to make that field an ``i64``.1057 1058 When buffer resources are passed to buffer intrinsics such as1059 ``llvm.amdgcn.raw.ptr.buffer.load`` or1060 ``llvm.amdgcn.struct.ptr.buffer.store``, the ``align`` attribute on the1061 pointer is assumed to apply to both the offset and the base pointer value.1062 That is, ``align 8`` means that both the base address within the ``ptr1063 addrspace(8)`` and the ``offset`` argument have their three lowest bits set1064 to 0. If the stride of the resource is nonzero, the stride must be a multiple1065 of the given alignment.1066 1067 In other words, the ``align`` attribute specifies the alignment of the effective1068 address being loaded from/stored to *and* acts as a guarantee that this is1069 not achieved from adding lower-alignment parts (as hardware may not always1070 allow for such an addition). For example, if a buffer resource has the base1071 address ``0xfffe`` and is accessed with a ``raw.ptr.buffer.load`` with an offset1072 of ``2``, the load must **not** be marked ``align 4`` (even though the1073 effective adddress ``0x10000`` is so aligned) as this would permit the compiler1074 to make incorrect transformations (such as promotion to ``s_buffer_load``,1075 which requires such componentwise alignment).1076 1077**Buffer Strided Pointer**1078 The buffer index pointer is an experimental address space. It represents1079 a 128-bit buffer descriptor and a 32-bit offset, like the **Buffer Fat1080 Pointer**. Additionally, it contains an index into the buffer, which1081 allows the direct addressing of structured elements. These components appear1082 in that order, i.e., the descriptor comes first, then the 32-bit offset1083 followed by the 32-bit index.1084 1085 The bits in the buffer descriptor must meet the following requirements:1086 the stride is the size of a structured element, the "add tid" flag must be 0,1087 and the swizzle enable bits must be off.1088 1089 These pointers can be created by ``addrspacecast`` from a buffer resource1090 (``ptr addrspace(8)``) or by using ``llvm.amdgcn.make.buffer.rsrc`` to produce a1091 ``ptr addrspace(9)``` directly, which produces a buffer strided pointer whose initial1092 index and offset values are both 0. This prevents the address space cast from1093 being rewritten away.1094 1095 As with buffer fat pointers, alignment of a buffer strided pointer applies to1096 both the base pointer address and the offset. In addition, the alignment also1097 constrains the stride of the pointer. That is, if you do an ``align 4`` load from1098 a buffer strided pointer, this means that the base pointer is ``align(4)``, that1099 the offset is a multiple of 4 bytes, and that the stride is a multiple of 4.1100 1101**Streamout Registers**1102 Dedicated registers used by the GS NGG Streamout Instructions. The register1103 file is modelled as a memory in a distinct address space because it is indexed1104 by an address-like offset in place of named registers, and because register1105 accesses affect LGKMcnt. This is an internal address space used only by the1106 compiler. Do not use this address space for IR pointers.1107 1108.. _amdgpu-memory-scopes:1109 1110Memory Scopes1111-------------1112 1113This section provides LLVM memory synchronization scopes supported by the AMDGPU1114backend memory model when the target triple OS is ``amdhsa`` (see1115:ref:`amdgpu-amdhsa-memory-model` and :ref:`amdgpu-target-triples`).1116 1117The memory model supported is based on the HSA memory model [HSA]_ which is1118based in turn on HRF-indirect with scope inclusion [HRF]_. The happens-before1119relation is transitive over the synchronizes-with relation independent of scope1120and synchronizes-with allows the memory scope instances to be inclusive (see1121table :ref:`amdgpu-amdhsa-llvm-sync-scopes-table`).1122 1123This is different to the OpenCL [OpenCL]_ memory model which does not have scope1124inclusion and requires the memory scopes to exactly match. However, this1125is conservatively correct for OpenCL.1126 1127 .. table:: AMDHSA LLVM Sync Scopes1128 :name: amdgpu-amdhsa-llvm-sync-scopes-table1129 1130 ======================= ===================================================1131 LLVM Sync Scope Description1132 ======================= ===================================================1133 *none* The default: ``system``.1134 1135 Synchronizes with, and participates in modification1136 and seq_cst total orderings with, other operations1137 (except image operations) for all address spaces1138 (except private, or generic that accesses private)1139 provided the other operation's sync scope is:1140 1141 - ``system``.1142 - ``agent`` and executed by a thread on the same1143 agent.1144 - ``workgroup`` and executed by a thread in the1145 same work-group.1146 - ``wavefront`` and executed by a thread in the1147 same wavefront.1148 1149 ``agent`` Synchronizes with, and participates in modification1150 and seq_cst total orderings with, other operations1151 (except image operations) for all address spaces1152 (except private, or generic that accesses private)1153 provided the other operation's sync scope is:1154 1155 - ``system`` or ``agent`` and executed by a thread1156 on the same agent.1157 - ``workgroup`` and executed by a thread in the1158 same work-group.1159 - ``wavefront`` and executed by a thread in the1160 same wavefront.1161 1162 ``cluster`` Synchronizes with, and participates in modification1163 and seq_cst total orderings with, other operations1164 (except image operations) for all address spaces1165 (except private, or generic that accesses private)1166 provided the other operation's sync scope is:1167 1168 - ``system``, ``agent`` or ``cluster`` and1169 executed by a thread on the same cluster.1170 - ``workgroup`` and executed by a thread in the1171 same work-group.1172 - ``wavefront`` and executed by a thread in the1173 same wavefront.1174 1175 On targets that do not support workgroup cluster1176 launch mode, this behaves like ``agent`` scope instead.1177 1178 ``workgroup`` Synchronizes with, and participates in modification1179 and seq_cst total orderings with, other operations1180 (except image operations) for all address spaces1181 (except private, or generic that accesses private)1182 provided the other operation's sync scope is:1183 1184 - ``system``, ``agent`` or ``workgroup`` and1185 executed by a thread in the same work-group.1186 - ``wavefront`` and executed by a thread in the1187 same wavefront.1188 1189 ``wavefront`` Synchronizes with, and participates in modification1190 and seq_cst total orderings with, other operations1191 (except image operations) for all address spaces1192 (except private, or generic that accesses private)1193 provided the other operation's sync scope is:1194 1195 - ``system``, ``agent``, ``workgroup`` or1196 ``wavefront`` and executed by a thread in the1197 same wavefront.1198 1199 ``singlethread`` Only synchronizes with and participates in1200 modification and seq_cst total orderings with,1201 other operations (except image operations) running1202 in the same thread for all address spaces (for1203 example, in signal handlers).1204 1205 ``one-as`` Same as ``system`` but only synchronizes with other1206 operations within the same address space.1207 1208 ``agent-one-as`` Same as ``agent`` but only synchronizes with other1209 operations within the same address space.1210 1211 ``cluster-one-as`` Same as ``cluster`` but only synchronizes with other1212 operations within the same address space.1213 1214 ``workgroup-one-as`` Same as ``workgroup`` but only synchronizes with1215 other operations within the same address space.1216 1217 ``wavefront-one-as`` Same as ``wavefront`` but only synchronizes with1218 other operations within the same address space.1219 1220 ``singlethread-one-as`` Same as ``singlethread`` but only synchronizes with1221 other operations within the same address space.1222 ======================= ===================================================1223 1224Target Types1225------------1226 1227The AMDGPU backend implements some target extension types.1228 1229.. _amdgpu-types-named-barriers:1230 1231Named Barriers1232~~~~~~~~~~~~~~1233 1234Named barriers are fixed function hardware barrier objects that are available1235in gfx12.5+ in addition to the traditional default barriers.1236 1237In LLVM IR, named barriers are represented by global variables of type1238``target("amdgcn.named.barrier", 0)`` in the LDS address space. Named barrier1239global variables do not occupy actual LDS memory, but their lifetime and1240allocation scope matches that of global variables in LDS. Programs in LLVM IR1241refer to named barriers using pointers.1242 1243The following named barrier types are supported in global variables, defined1244recursively:1245 1246* a single, standalone ``target("amdgcn.named.barrier", 0)``1247* an array of supported types1248* a struct containing a single element of supported type1249 1250.. code-block:: llvm1251 1252 @bar = addrspace(3) global target("amdgcn.named.barrier", 0) undef1253 @foo = addrspace(3) global [2 x target("amdgcn.named.barrier", 0)] undef1254 @baz = addrspace(3) global { target("amdgcn.named.barrier", 0) } undef1255 1256 ...1257 1258 %foo.i = getelementptr [2 x target("amdgcn.named.barrier", 0)], ptr addrspace(3) @foo, i32 0, i32 %i1259 call void @llvm.amdgcn.s.barrier.signal.var(ptr addrspace(3) %foo.i, i32 0)1260 1261Named barrier types may not be used in ``alloca``.1262 1263Named barriers do not have an underlying byte representation.1264It is undefined behavior to use a pointer to any part of a named barrier object1265as the pointer operand of a regular memory access instruction or intrinsic.1266Pointers to named barrier objects are intended to be used with dedicated1267intrinsics. Reading from or writing to such pointers is undefined behavior.1268 1269LLVM IR Intrinsics1270------------------1271 1272The AMDGPU backend implements the following LLVM IR intrinsics.1273 1274*This section is WIP.*1275 1276.. table:: AMDGPU LLVM IR Intrinsics1277 :name: amdgpu-llvm-ir-intrinsics-table1278 1279 ============================================== ==========================================================1280 LLVM Intrinsic Description1281 ============================================== ==========================================================1282 llvm.amdgcn.sqrt Provides direct access to v_sqrt_f64, v_sqrt_f32 and v_sqrt_f161283 (on targets with half support). Performs sqrt function.1284 1285 llvm.amdgcn.log Provides direct access to v_log_f32 and v_log_f161286 (on targets with half support). Performs log2 function.1287 1288 llvm.amdgcn.exp2 Provides direct access to v_exp_f32 and v_exp_f161289 (on targets with half support). Performs exp2 function.1290 1291 :ref:`llvm.frexp <int_frexp>` Implemented for half, float and double.1292 1293 :ref:`llvm.log2 <int_log2>` Implemented for float and half (and vectors of float or1294 half). Not implemented for double. Hardware provides1295 1ULP accuracy for float, and 0.51ULP for half. Float1296 instruction does not natively support denormal1297 inputs.1298 1299 :ref:`llvm.sqrt <int_sqrt>` Implemented for double, float and half (and vectors).1300 1301 :ref:`llvm.log <int_log>` Implemented for float and half (and vectors).1302 1303 :ref:`llvm.exp <int_exp>` Implemented for float and half (and vectors).1304 1305 :ref:`llvm.log10 <int_log10>` Implemented for float and half (and vectors).1306 1307 :ref:`llvm.exp2 <int_exp2>` Implemented for float and half (and vectors of float or1308 half). Not implemented for double. Hardware provides1309 1ULP accuracy for float, and 0.51ULP for half. Float1310 instruction does not natively support denormal1311 inputs.1312 1313 :ref:`llvm.stacksave.p5 <int_stacksave>` Implemented, must use the alloca address space.1314 :ref:`llvm.stackrestore.p5 <int_stackrestore>` Implemented, must use the alloca address space.1315 1316 :ref:`llvm.get.fpmode.i32 <int_get_fpmode>` The natural floating-point mode type is i32. This1317 is implemented by extracting relevant bits out of the MODE1318 register with s_getreg_b32. The first 10 bits are the1319 core floating-point mode. Bits 12:18 are the exception1320 mask. On gfx9+, bit 23 is FP16_OVFL. Bitfields not1321 relevant to floating-point instructions are 0s.1322 1323 :ref:`llvm.get.rounding<int_get_rounding>` AMDGPU supports two separately controllable rounding1324 modes depending on the floating-point type. One1325 controls float, and the other controls both double and1326 half operations. If both modes are the same, returns1327 one of the standard return values. If the modes are1328 different, returns one of :ref:`12 extended values1329 <amdgpu-rounding-mode-enumeration-values-table>`1330 describing the two modes.1331 1332 To nearest, ties away from zero is not a supported1333 mode. The raw rounding mode values in the MODE1334 register do not exactly match the FLT_ROUNDS values,1335 so a conversion is performed.1336 1337 :ref:`llvm.set.rounding<int_set_rounding>` Input value expected to be one of the valid results1338 from '``llvm.get.rounding``'. Rounding mode is1339 undefined if not passed a valid input. This should be1340 a wave uniform value. In case of a divergent input1341 value, the first active lane's value will be used.1342 1343 :ref:`llvm.get.fpenv<int_get_fpenv>` Returns the current value of the AMDGPU floating point environment.1344 This stores information related to the current rounding mode,1345 denormalization mode, enabled traps, and floating point exceptions.1346 The format is a 64-bit concatenation of the MODE and TRAPSTS registers.1347 1348 :ref:`llvm.set.fpenv<int_set_fpenv>` Sets the floating point environment to the specified state.1349 llvm.amdgcn.load.to.lds.p<1/7> Loads values from global memory (either in the form of a global1350 a raw fat buffer pointer) to LDS. The size of the data copied can be 1, 2,1351 or 4 bytes (and gfx950 also allows 12 or 16 bytes). The LDS pointer1352 argument should be wavefront-uniform; the global pointer need not be.1353 The LDS pointer is implicitly offset by 4 * lane_id bytes for size <= 4 bytes1354 and 16 * lane_id bytes for larger sizes. This lowers to `global_load_lds`,1355 `buffer_load_* ... lds`, or `global_load__* ... lds` depending on address1356 space and architecture. `amdgcn.global.load.lds` has the same semantics as1357 `amdgcn.load.to.lds.p1`.1358 llvm.amdgcn.readfirstlane Provides direct access to v_readfirstlane_b32. Returns the value in1359 the lowest active lane of the input operand. Currently implemented1360 for i16, i32, float, half, bfloat, <2 x i16>, <2 x half>, <2 x bfloat>,1361 i64, double, pointers, multiples of the 32-bit vectors.1362 1363 llvm.amdgcn.readlane Provides direct access to v_readlane_b32. Returns the value in the1364 specified lane of the first input operand. The second operand specifies1365 the lane to read from. Currently implemented for i16, i32, float, half,1366 bfloat, <2 x i16>, <2 x half>, <2 x bfloat>, i64, double, pointers,1367 multiples of the 32-bit vectors.1368 1369 llvm.amdgcn.writelane Provides direct access to v_writelane_b32. Writes value in the first input1370 operand to the specified lane of divergent output. The second operand1371 specifies the lane to write. Currently implemented for i16, i32, float,1372 half, bfloat, <2 x i16>, <2 x half>, <2 x bfloat>, i64, double, pointers,1373 multiples of the 32-bit vectors.1374 1375 llvm.amdgcn.wave.reduce.umin Performs an arithmetic unsigned min reduction on the unsigned values1376 provided by each lane in the wavefront.1377 Intrinsic takes a hint for reduction strategy using second operand1378 0: Target default preference,1379 1: `Iterative strategy`, and1380 2: `DPP`.1381 If target does not support the DPP operations (e.g. gfx6/7),1382 reduction will be performed using default iterative strategy.1383 Intrinsic is currently only implemented for i32.1384 1385 llvm.amdgcn.wave.reduce.umax Performs an arithmetic unsigned max reduction on the unsigned values1386 provided by each lane in the wavefront.1387 Intrinsic takes a hint for reduction strategy using second operand1388 0: Target default preference,1389 1: `Iterative strategy`, and1390 2: `DPP`.1391 If target does not support the DPP operations (e.g. gfx6/7),1392 reduction will be performed using default iterative strategy.1393 Intrinsic is currently only implemented for i32.1394 1395 llvm.amdgcn.permlane16 Provides direct access to v_permlane16_b32. Performs arbitrary gather-style1396 operation within a row (16 contiguous lanes) of the second input operand.1397 The third and fourth inputs must be scalar values. These are combined into1398 a single 64-bit value representing lane selects used to swizzle within each1399 row. Currently implemented for i16, i32, float, half, bfloat, <2 x i16>,1400 <2 x half>, <2 x bfloat>, i64, double, pointers, multiples of the 32-bit vectors.1401 1402 llvm.amdgcn.permlanex16 Provides direct access to v_permlanex16_b32. Performs arbitrary gather-style1403 operation across two rows of the second input operand (each row is 16 contiguous1404 lanes). The third and fourth inputs must be scalar values. These are combined1405 into a single 64-bit value representing lane selects used to swizzle within each1406 row. Currently implemented for i16, i32, float, half, bfloat, <2 x i16>, <2 x half>,1407 <2 x bfloat>, i64, double, pointers, multiples of the 32-bit vectors.1408 1409 llvm.amdgcn.permlane64 Provides direct access to v_permlane64_b32. Performs a specific permutation across1410 lanes of the input operand where the high half and low half of a wave64 are swapped.1411 Performs no operation in wave32 mode. Currently implemented for i16, i32, float, half,1412 bfloat, <2 x i16>, <2 x half>, <2 x bfloat>, i64, double, pointers, multiples of the1413 32-bit vectors.1414 1415 llvm.amdgcn.udot2 Provides direct access to v_dot2_u32_u16 across targets which1416 support such instructions. This performs an unsigned dot product1417 with two v2i16 operands, summed with the third i32 operand. The1418 i1 fourth operand is used to clamp the output.1419 1420 llvm.amdgcn.udot4 Provides direct access to v_dot4_u32_u8 across targets which1421 support such instructions. This performs an unsigned dot product1422 with two i32 operands (holding a vector of 4 8bit values), summed1423 with the third i32 operand. The i1 fourth operand is used to clamp1424 the output.1425 1426 llvm.amdgcn.udot8 Provides direct access to v_dot8_u32_u4 across targets which1427 support such instructions. This performs an unsigned dot product1428 with two i32 operands (holding a vector of 8 4bit values), summed1429 with the third i32 operand. The i1 fourth operand is used to clamp1430 the output.1431 1432 llvm.amdgcn.sdot2 Provides direct access to v_dot2_i32_i16 across targets which1433 support such instructions. This performs a signed dot product1434 with two v2i16 operands, summed with the third i32 operand. The1435 i1 fourth operand is used to clamp the output.1436 When applicable (e.g. no clamping), this is lowered into1437 v_dot2c_i32_i16 for targets which support it.1438 1439 llvm.amdgcn.sdot4 Provides direct access to v_dot4_i32_i8 across targets which1440 support such instructions. This performs a signed dot product1441 with two i32 operands (holding a vector of 4 8bit values), summed1442 with the third i32 operand. The i1 fourth operand is used to clamp1443 the output.1444 When applicable (i.e. no clamping / operand modifiers), this is lowered1445 into v_dot4c_i32_i8 for targets which support it.1446 RDNA3 does not offer v_dot4_i32_i8, and rather offers1447 v_dot4_i32_iu8 which has operands to hold the signedness of the1448 vector operands. Thus, this intrinsic lowers to the signed version1449 of this instruction for gfx11 targets.1450 1451 llvm.amdgcn.sdot8 Provides direct access to v_dot8_u32_u4 across targets which1452 support such instructions. This performs a signed dot product1453 with two i32 operands (holding a vector of 8 4bit values), summed1454 with the third i32 operand. The i1 fourth operand is used to clamp1455 the output.1456 When applicable (i.e. no clamping / operand modifiers), this is lowered1457 into v_dot8c_i32_i4 for targets which support it.1458 RDNA3 does not offer v_dot8_i32_i4, and rather offers1459 v_dot4_i32_iu4 which has operands to hold the signedness of the1460 vector operands. Thus, this intrinsic lowers to the signed version1461 of this instruction for gfx11 targets.1462 1463 llvm.amdgcn.sudot4 Provides direct access to v_dot4_i32_iu8 on gfx11 targets. This performs1464 dot product with two i32 operands (holding a vector of 4 8bit values), summed1465 with the fifth i32 operand. The i1 sixth operand is used to clamp1466 the output. The i1s preceding the vector operands decide the signedness.1467 1468 llvm.amdgcn.sudot8 Provides direct access to v_dot8_i32_iu4 on gfx11 targets. This performs1469 dot product with two i32 operands (holding a vector of 8 4bit values), summed1470 with the fifth i32 operand. The i1 sixth operand is used to clamp1471 the output. The i1s preceding the vector operands decide the signedness.1472 1473 llvm.amdgcn.sched.barrier Controls the types of instructions that may be allowed to cross the intrinsic1474 during instruction scheduling. The parameter is a mask for the instruction types1475 that can cross the intrinsic.1476 1477 - 0x0000: No instructions may be scheduled across sched_barrier.1478 - 0x0001: All, non-memory, non-side-effect producing instructions may be1479 scheduled across sched_barrier, *i.e.* allow ALU instructions to pass.1480 - 0x0002: VALU instructions may be scheduled across sched_barrier.1481 - 0x0004: SALU instructions may be scheduled across sched_barrier.1482 - 0x0008: MFMA/WMMA instructions may be scheduled across sched_barrier.1483 - 0x0010: All VMEM instructions may be scheduled across sched_barrier.1484 - 0x0020: VMEM read instructions may be scheduled across sched_barrier.1485 - 0x0040: VMEM write instructions may be scheduled across sched_barrier.1486 - 0x0080: All DS instructions may be scheduled across sched_barrier.1487 - 0x0100: All DS read instructions may be scheduled across sched_barrier.1488 - 0x0200: All DS write instructions may be scheduled across sched_barrier.1489 - 0x0400: All Transcendental (e.g. V_EXP) instructions may be scheduled across sched_barrier.1490 1491 llvm.amdgcn.sched.group.barrier Creates schedule groups with specific properties to create custom scheduling1492 pipelines. The ordering between groups is enforced by the instruction scheduler.1493 The intrinsic applies to the code that precedes the intrinsic. The intrinsic1494 takes three values that control the behavior of the schedule groups.1495 1496 - Mask : Classify instruction groups using the llvm.amdgcn.sched_barrier mask values.1497 - Size : The number of instructions that are in the group.1498 - SyncID : Order is enforced between groups with matching values.1499 1500 The mask can include multiple instruction types. It is undefined behavior to set1501 values beyond the range of valid masks.1502 1503 Combining multiple sched_group_barrier intrinsics enables an ordering of specific1504 instruction types during instruction scheduling. For example, the following enforces1505 a sequence of 1 VMEM read, followed by 1 VALU instruction, followed by 5 MFMA1506 instructions.1507 1508 | ``// 1 VMEM read``1509 | ``__builtin_amdgcn_sched_group_barrier(32, 1, 0)``1510 | ``// 1 VALU``1511 | ``__builtin_amdgcn_sched_group_barrier(2, 1, 0)``1512 | ``// 5 MFMA``1513 | ``__builtin_amdgcn_sched_group_barrier(8, 5, 0)``1514 1515 llvm.amdgcn.iglp.opt An **experimental** intrinsic for instruction group level parallelism. The intrinsic1516 implements predefined instruction scheduling orderings. The intrinsic applies to the1517 surrounding scheduling region. The intrinsic takes a value that specifies the1518 strategy. The compiler implements two strategies.1519 1520 0. Interleave DS and MFMA instructions for small GEMM kernels.1521 1. Interleave DS and MFMA instructions for single wave small GEMM kernels.1522 2. Interleave TRANS and MFMA instructions, as well as their VALU and DS predecessors, for attention kernels.1523 3. Interleave TRANS and MFMA instructions, with no predecessor interleaving, for attention kernels.1524 1525 Only one iglp_opt intrinsic may be used in a scheduling region. The iglp_opt intrinsic1526 cannot be combined with sched_barrier or sched_group_barrier.1527 1528 The iglp_opt strategy implementations are subject to change.1529 1530 llvm.amdgcn.atomic.cond.sub.u32 Provides direct access to flat_atomic_cond_sub_u32, global_atomic_cond_sub_u321531 and ds_cond_sub_u32 based on address space on gfx12 targets. This1532 performs a subtraction only if the memory value is greater than or1533 equal to the data value.1534 1535 llvm.amdgcn.s.barrier.signal.isfirst Provides access to the s_barrier_signal_first instruction;1536 additionally ensures that the result value is valid even when the1537 intrinsic is used from a wave that is not running in a workgroup.1538 1539 llvm.amdgcn.s.getpc Provides access to the s_getpc_b64 instruction, but with the return value1540 sign-extended from the width of the underlying PC hardware register even on1541 processors where the s_getpc_b64 instruction returns a zero-extended value.1542 1543 llvm.amdgcn.ballot Returns a bitfield(i32 or i64) containing the result of its i1 argument1544 in all active lanes, and zero in all inactive lanes.1545 Provides a way to convert i1 in LLVM IR to i32 or i64 lane mask - bitfield1546 used by hardware to control active lanes when used in EXEC register.1547 For example, ballot(i1 true) return EXEC mask.1548 1549 llvm.amdgcn.mfma.scale.f32.16x16x128.f8f6f4 Emit `v_mfma_scale_f32_16x16x128_f8f6f4` to set the scale factor. The1550 last 4 operands correspond to the scale inputs.1551 1552 - 2-bit byte index to use for each lane for matrix A1553 - Matrix A scale values1554 - 2-bit byte index to use for each lane for matrix B1555 - Matrix B scale values1556 1557 llvm.amdgcn.mfma.scale.f32.32x32x64.f8f6f4 Emit `v_mfma_scale_f32_32x32x64_f8f6f4`1558 1559 llvm.amdgcn.permlane16.swap Provide direct access to `v_permlane16_swap_b32` instruction on supported targets.1560 Swaps the values across lanes of first 2 operands. Odd rows of the first operand are1561 swapped with even rows of the second operand (one row is 16 lanes).1562 Returns a pair for the swapped registers. The first element of the return corresponds1563 to the swapped element of the first argument.1564 1565 llvm.amdgcn.permlane32.swap Provide direct access to `v_permlane32_swap_b32` instruction on supported targets.1566 Swaps the values across lanes of first 2 operands. Rows 2 and 3 of the first operand are1567 swapped with rows 0 and 1 of the second operand (one row is 16 lanes).1568 Returns a pair for the swapped registers. The first element of the return1569 corresponds to the swapped element of the first argument.1570 1571 llvm.amdgcn.mov.dpp The llvm.amdgcn.mov.dpp.`<type>` intrinsic represents the mov.dpp operation in AMDGPU.1572 This operation is being deprecated and can be replaced with llvm.amdgcn.update.dpp.1573 1574 llvm.amdgcn.update.dpp The llvm.amdgcn.update.dpp.`<type>` intrinsic represents the update.dpp operation in AMDGPU.1575 It takes an old value, a source operand, a DPP control operand, a row mask, a bank mask, and a bound control.1576 Various data types are supported, including, bf16, f16, f32, f64, i16, i32, i64, p0, p3, p5, v2f16, v2f32, v2i16, v2i32, v2p0, v3i32, v4i32, v8f16.1577 This operation is equivalent to a sequence of v_mov_b32 operations.1578 It is preferred over llvm.amdgcn.mov.dpp.`<type>` for future use.1579 `llvm.amdgcn.update.dpp.<type> <old> <src> <dpp_ctrl> <row_mask> <bank_mask> <bound_ctrl>`1580 Should be equivalent to:1581 1582 - `v_mov_b32 <dest> <old>`1583 - `v_mov_b32 <dest> <src> <dpp_ctrl> <row_mask> <bank_mask> <bound_ctrl>`1584 1585 :ref:`llvm.prefetch <int_prefetch>` Implemented on gfx1250, ignored on earlier targets.1586 First argument is flat, global, or constant address space pointer.1587 Any other address space is not supported.1588 On gfx125x generates flat_prefetch_b8 or global_prefetch_b8 and brings data to GL2.1589 Second argument is rw and currently ignored. Can be 0 or 1.1590 Third argument is locality, 0-3. Translates to memory scope:1591 1592 * 0 - SCOPE_SYS1593 * 1 - SCOPE_DEV1594 * 2 - SCOPE_SE1595 * 3 - SCOPE_SE1596 1597 Note that SCOPE_CU is not generated and not safe on an invalid address.1598 Fourth argument is cache type:1599 1600 * 0 - Instruction cache, currently ignored and no code is generated.1601 * 1 - Data cache.1602 1603 Instruction cache prefetches are unsafe on invalid address.1604 ============================================== ==========================================================1605 1606.. TODO::1607 1608 List AMDGPU intrinsics.1609 1610'``llvm.amdgcn.cooperative.atomic``' Intrinsics1611~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~1612 1613The ``llvm.amdgcn.cooperative.atomic`` :ref:`family of intrinsics<amdgpu-cooperative-atomic-intrinsics-table>`1614provide atomic load and store operations to a naturally-aligned contiguous memory regions.1615Memory is accessed cooperatively by a collection of convergent threads, with each thread accessing1616a fraction of the contiguous memory region.1617 1618 .. TODO::1619 1620 The memory model described here is imprecise; see SWDEV-536264.1621 1622This intrinsic has a memory ordering and may be used to synchronize-with another cooperative atomic.1623If the memory ordering is relaxed, it may pair with a fence if that same fence is executed by1624all participating threads with the same synchronization scope and set of address spaces.1625 1626In both cases, a synchronize-with relation can only be established between cooperative atomics with the1627same total access size.1628 1629Each target may have additional restrictions on how the intrinsic may be used; see1630:ref:`the table below<amdgpu-llvm-ir-cooperative-atomic-intrinsics-availability>`.1631Targets not covered in the table do not support these intrinsics.1632 1633 .. table:: AMDGPU Cooperative Atomic Intrinsics Availability1634 :name: amdgpu-llvm-ir-cooperative-atomic-intrinsics-availability1635 1636 =============== =============================================================1637 GFX Version Target Restrictions1638 =============== =============================================================1639 GFX 12.5 :ref:`amdgpu-amdhsa-memory-model-gfx125x-cooperative-atomics`1640 =============== =============================================================1641 1642If the intrinsic is used without meeting all of the above conditions, or the target-specific conditions,1643then this intrinsic causes undefined behavior.1644 1645 .. table:: AMDGPU Cooperative Atomic Intrinsics1646 :name: amdgpu-cooperative-atomic-intrinsics-table1647 1648 ======================================================= =========== ============ ==========1649 LLVM Intrinsic Number of Access Size Total Size1650 Threads Per Thread1651 Used1652 ======================================================= =========== ============ ==========1653 ``llvm.amdgcn.cooperative.atomic.store.32x4B`` 32 4B 128B1654 1655 ``llvm.amdgcn.cooperative.atomic.load.32x4B`` 32 4B 128B1656 1657 ``llvm.amdgcn.cooperative.atomic.store.16x8B`` 16 8B 128B1658 1659 ``llvm.amdgcn.cooperative.atomic.load.16x8B`` 16 8B 128B1660 1661 ``llvm.amdgcn.cooperative.atomic.store.8x16B`` 8 16B 128B1662 1663 ``llvm.amdgcn.cooperative.atomic.load.8x16B`` 8 16B 128B1664 1665 ======================================================= =========== ============ ==========1666 1667The intrinsics are available for the global (``.p1`` suffix) and generic (``.p0`` suffix) address spaces.1668 1669The atomic ordering operand (3rd operand for ``.store``, 2nd for ``.load``) is an integer that follows the1670C ABI encoding of atomic memory orderings. The supported values are in1671:ref:`the table below<amdgpu-cooperative-atomic-intrinsics-atomic-memory-orderings-table>`.1672 1673 .. table:: AMDGPU Cooperative Atomic Intrinsics Atomic Memory Orderings1674 :name: amdgpu-cooperative-atomic-intrinsics-atomic-memory-orderings-table1675 1676 ====== ================ =================================1677 Value Atomic Memory Notes1678 Ordering1679 ====== ================ =================================1680 ``0`` ``relaxed`` The default for unsupported values.1681 1682 ``2`` ``acquire`` Only for ``.load``1683 1684 ``3`` ``release`` Only for ``.store``1685 1686 ``5`` ``seq_cst``1687 ====== ================ =================================1688 1689The last argument of the intrinsic is the synchronization scope1690as a metadata string, which must be one of the supported :ref:`memory scopes<amdgpu-memory-scopes>`.1691 1692.. _amdgpu_metadata:1693 1694LLVM IR Metadata1695================1696 1697The AMDGPU backend implements the following target custom LLVM IR1698metadata.1699 1700.. _amdgpu_last_use:1701 1702'``amdgpu.last.use``' Metadata1703------------------------------1704 1705Sets TH_LOAD_LU temporal hint on load instructions that support it.1706Takes priority over nontemporal hint (TH_LOAD_NT). This takes no1707arguments.1708 1709.. code-block:: llvm1710 1711 %val = load i32, ptr %in, align 4, !amdgpu.last.use !{}1712 1713'``amdgpu.no.remote.memory``' Metadata1714---------------------------------------------1715 1716Asserts a memory operation does not access bytes in host memory, or1717remote connected peer device memory (the address must be device1718local). This is intended for use with :ref:`atomicrmw <i_atomicrmw>`1719and other atomic instructions. This is required to emit a native1720hardware instruction for some :ref:`system scope1721<amdgpu-memory-scopes>` atomic operations on some subtargets. For most1722integer atomic operations, this is a sufficient restriction to emit a1723native atomic instruction.1724 1725An :ref:`atomicrmw <i_atomicrmw>` without metadata will be treated1726conservatively as required to preserve the operation behavior in all1727cases. This will typically be used in conjunction with1728:ref:`\!amdgpu.no.fine.grained.memory<amdgpu_no_fine_grained_memory>`.1729 1730 1731.. code-block:: llvm1732 1733 ; Indicates the atomic does not access fine-grained memory, or1734 ; remote device memory.1735 %old0 = atomicrmw sub ptr %ptr0, i32 1 acquire, !amdgpu.no.fine.grained.memory !0, !amdgpu.no.remote.memory !01736 1737 ; Indicates the atomic does not access peer device memory.1738 %old2 = atomicrmw sub ptr %ptr2, i32 1 acquire, !amdgpu.no.remote.memory !01739 1740 !0 = !{}1741 1742.. _amdgpu_no_fine_grained_memory:1743 1744'``amdgpu.no.fine.grained.memory``' Metadata1745-------------------------------------------------1746 1747Asserts a memory access does not access bytes allocated in1748fine-grained allocated memory. This is intended for use with1749:ref:`atomicrmw <i_atomicrmw>` and other atomic instructions. This is1750required to emit a native hardware instruction for some :ref:`system1751scope <amdgpu-memory-scopes>` atomic operations on some subtargets. An1752:ref:`atomicrmw <i_atomicrmw>` without metadata will be treated1753conservatively as required to preserve the operation behavior in all1754cases. This will typically be used in conjunction with1755:ref:`\!amdgpu.no.remote.memory.access<amdgpu_no_remote_memory_access>`.1756 1757.. code-block:: llvm1758 1759 ; Indicates the access does not access fine-grained memory, or1760 ; remote device memory.1761 %old0 = atomicrmw sub ptr %ptr0, i32 1 acquire, !amdgpu.no.fine.grained.memory !0, !amdgpu.no.remote.memory.access !01762 1763 ; Indicates the access does not access fine-grained memory1764 %old2 = atomicrmw sub ptr %ptr2, i32 1 acquire, !amdgpu.no.fine.grained.memory !01765 1766 !0 = !{}1767 1768.. _amdgpu_no_remote_memory_access:1769 1770'``amdgpu.ignore.denormal.mode``' Metadata1771------------------------------------------1772 1773For use with :ref:`atomicrmw <i_atomicrmw>` floating-point1774operations. Indicates the handling of denormal inputs and results is1775insignificant and may be inconsistent with the expected floating-point1776mode. This is necessary to emit a native atomic instruction on some1777targets for some address spaces where float denormals are1778unconditionally flushed. This is typically used in conjunction with1779:ref:`\!amdgpu.no.remote.memory.access<amdgpu_no_remote_memory_access>`1780and1781:ref:`\!amdgpu.no.fine.grained.memory<amdgpu_no_fine_grained_memory>`1782 1783 1784.. code-block:: llvm1785 1786 %res0 = atomicrmw fadd ptr addrspace(1) %ptr, float %value seq_cst, align 4, !amdgpu.ignore.denormal.mode !01787 %res1 = atomicrmw fadd ptr addrspace(1) %ptr, float %value seq_cst, align 4, !amdgpu.ignore.denormal.mode !0, !amdgpu.no.fine.grained.memory !0, !amdgpu.no.remote.memory.access !01788 1789 !0 = !{}1790 1791 1792LLVM IR Attributes1793==================1794 1795The AMDGPU backend supports the following LLVM IR attributes.1796 1797 .. table:: AMDGPU LLVM IR Attributes1798 :name: amdgpu-llvm-ir-attributes-table1799 1800 ================================================ ==========================================================1801 LLVM Attribute Description1802 ================================================ ==========================================================1803 "amdgpu-flat-work-group-size"="min,max" Specify the minimum and maximum flat work group sizes that1804 will be specified when the kernel is dispatched. Generated1805 by the ``amdgpu_flat_work_group_size`` CLANG attribute [CLANG-ATTR]_.1806 The IR implied default value is 1,1024. Clang may emit this attribute1807 with more restrictive bounds depending on language defaults.1808 If the actual block or workgroup size exceeds the limit at any point during1809 the execution, the behavior is undefined. For example, even if there is1810 only one active thread but the thread local id exceeds the limit, the1811 behavior is undefined.1812 1813 "amdgpu-implicitarg-num-bytes"="n" Number of kernel argument bytes to add to the kernel1814 argument block size for the implicit arguments. This1815 varies by OS and language (for OpenCL see1816 :ref:`opencl-kernel-implicit-arguments-appended-for-amdhsa-os-table`).1817 "amdgpu-num-sgpr"="n" Specifies the number of SGPRs to use. Generated by1818 the ``amdgpu_num_sgpr`` CLANG attribute [CLANG-ATTR]_.1819 "amdgpu-num-vgpr"="n" Specifies the number of VGPRs to use. Generated by the1820 ``amdgpu_num_vgpr`` CLANG attribute [CLANG-ATTR]_.1821 "amdgpu-waves-per-eu"="m,n" Specify the minimum and maximum number of waves per1822 execution unit. Generated by the ``amdgpu_waves_per_eu``1823 CLANG attribute [CLANG-ATTR]_. This is an optimization hint,1824 and the backend may not be able to satisfy the request. If1825 the specified range is incompatible with the function's1826 "amdgpu-flat-work-group-size" value, the implied occupancy1827 bounds by the workgroup size takes precedence.1828 1829 "amdgpu-ieee" true/false. GFX6-GFX11 Only1830 Specify whether the function expects the IEEE field of the1831 mode register to be set on entry. Overrides the default for1832 the calling convention.1833 "amdgpu-dx10-clamp" true/false. GFX6-GFX11 Only1834 Specify whether the function expects the DX10_CLAMP field of1835 the mode register to be set on entry. Overrides the default1836 for the calling convention.1837 1838 "amdgpu-no-workitem-id-x" Indicates the function does not depend on the value of the1839 llvm.amdgcn.workitem.id.x intrinsic. If a function is marked with this1840 attribute, or reached through a call site marked with this attribute,1841 and that intrinsic is called, the behavior of the program is undefined.1842 (Whole-program undefined behavior is used here because, for example,1843 the absence of a required workitem ID in the preloaded register set can1844 mean that all other preloaded registers are earlier than the compilation1845 assumed they would be.) The backend can generally infer this during code1846 generation, so typically there is no benefit to frontends marking1847 functions with this.1848 1849 "amdgpu-no-workitem-id-y" The same as amdgpu-no-workitem-id-x, except for the1850 llvm.amdgcn.workitem.id.y intrinsic.1851 1852 "amdgpu-no-workitem-id-z" The same as amdgpu-no-workitem-id-x, except for the1853 llvm.amdgcn.workitem.id.z intrinsic.1854 1855 "amdgpu-no-workgroup-id-x" The same as amdgpu-no-workitem-id-x, except for the1856 llvm.amdgcn.workgroup.id.x intrinsic.1857 1858 "amdgpu-no-workgroup-id-y" The same as amdgpu-no-workitem-id-x, except for the1859 llvm.amdgcn.workgroup.id.y intrinsic.1860 1861 "amdgpu-no-workgroup-id-z" The same as amdgpu-no-workitem-id-x, except for the1862 llvm.amdgcn.workgroup.id.z intrinsic.1863 1864 "amdgpu-no-cluster-id-x" The same as amdgpu-no-workitem-id-x, except for the1865 llvm.amdgcn.cluster.id.x intrinsic.1866 1867 "amdgpu-no-cluster-id-y" The same as amdgpu-no-workitem-id-x, except for the1868 llvm.amdgcn.cluster.id.y intrinsic.1869 1870 "amdgpu-no-cluster-id-z" The same as amdgpu-no-workitem-id-x, except for the1871 llvm.amdgcn.cluster.id.z intrinsic.1872 1873 "amdgpu-no-dispatch-ptr" The same as amdgpu-no-workitem-id-x, except for the1874 llvm.amdgcn.dispatch.ptr intrinsic.1875 1876 "amdgpu-no-implicitarg-ptr" The same as amdgpu-no-workitem-id-x, except for the1877 llvm.amdgcn.implicitarg.ptr intrinsic.1878 1879 "amdgpu-no-dispatch-id" The same as amdgpu-no-workitem-id-x, except for the1880 llvm.amdgcn.dispatch.id intrinsic.1881 1882 "amdgpu-no-queue-ptr" Similar to amdgpu-no-workitem-id-x, except for the1883 llvm.amdgcn.queue.ptr intrinsic. Note that unlike the other ABI hint1884 attributes, the queue pointer may be required in situations where the1885 intrinsic call does not directly appear in the program. Some subtargets1886 require the queue pointer to handle some addrspacecasts, as well1887 as the llvm.amdgcn.is.shared, llvm.amdgcn.is.private, llvm.trap, and1888 llvm.debug intrinsics.1889 1890 "amdgpu-no-hostcall-ptr" Similar to amdgpu-no-implicitarg-ptr, except specific to the implicit1891 kernel argument that holds the pointer to the hostcall buffer. If this1892 attribute is absent, then the amdgpu-no-implicitarg-ptr is also removed.1893 1894 "amdgpu-no-heap-ptr" Similar to amdgpu-no-implicitarg-ptr, except specific to the implicit1895 kernel argument that holds the pointer to an initialized memory buffer1896 that conforms to the requirements of the malloc/free device library V11897 version implementation. If this attribute is absent, then the1898 amdgpu-no-implicitarg-ptr is also removed.1899 1900 "amdgpu-no-multigrid-sync-arg" Similar to amdgpu-no-implicitarg-ptr, except specific to the implicit1901 kernel argument that holds the multigrid synchronization pointer. If this1902 attribute is absent, then the amdgpu-no-implicitarg-ptr is also removed.1903 1904 "amdgpu-no-default-queue" Similar to amdgpu-no-implicitarg-ptr, except specific to the implicit1905 kernel argument that holds the default queue pointer. If this1906 attribute is absent, then the amdgpu-no-implicitarg-ptr is also removed.1907 1908 "amdgpu-no-completion-action" Similar to amdgpu-no-implicitarg-ptr, except specific to the implicit1909 kernel argument that holds the completion action pointer. If this1910 attribute is absent, then the amdgpu-no-implicitarg-ptr is also removed.1911 1912 "amdgpu-lds-size"="min[,max]" Min is the minimum number of bytes that will be allocated in the Local1913 Data Store at address zero. Variables are allocated within this frame1914 using absolute symbol metadata, primarily by the AMDGPULowerModuleLDS1915 pass. Optional max is the maximum number of bytes that will be allocated.1916 Note that min==max indicates that no further variables can be added to1917 the frame. This is an internal detail of how LDS variables are lowered,1918 language front ends should not set this attribute.1919 1920 "amdgpu-gds-size" Bytes expected to be allocated at the start of GDS memory at entry.1921 1922 "amdgpu-git-ptr-high" The hard-wired high half of the address of the global information table1923 for AMDPAL OS type. 0xffffffff represents no hard-wired high half, since1924 current hardware only allows a 16-bit value.1925 1926 "amdgpu-32bit-address-high-bits" Assumed high 32-bits for 32-bit address spaces which are really truncated1927 64-bit addresses (i.e., addrspace(6))1928 1929 "amdgpu-color-export" Indicates shader exports color information if set to 1.1930 Defaults to 1 for :ref:`amdgpu_ps <amdgpu-cc>`, and 0 for other calling1931 conventions. Determines the necessity and type of null exports when a shader1932 terminates early by killing lanes.1933 1934 "amdgpu-depth-export" Indicates shader exports depth information if set to 1. Determines the1935 necessity and type of null exports when a shader terminates early by killing1936 lanes. A depth-only shader will export to depth channel when no null export1937 target is available (GFX11+).1938 1939 "InitialPSInputAddr" Set the initial value of the `spi_ps_input_addr` register for1940 :ref:`amdgpu_ps <amdgpu-cc>` shaders. Any bits enabled by this value will1941 be enabled in the final register value.1942 1943 "amdgpu-wave-priority-threshold" VALU instruction count threshold for adjusting wave priority. If exceeded,1944 temporarily raise the wave priority at the start of the shader function1945 until its last VMEM instructions to allow younger waves to issue their VMEM1946 instructions as well.1947 1948 "amdgpu-memory-bound" Set internally by backend1949 1950 "amdgpu-wave-limiter" Set internally by backend1951 1952 "amdgpu-unroll-threshold" Set base cost threshold preference for loop unrolling within this function,1953 default is 300. Actual threshold may be varied by per-loop metadata or1954 reduced by heuristics.1955 1956 "amdgpu-max-num-workgroups"="x,y,z" Specify the maximum number of work groups for the kernel dispatch in the1957 X, Y, and Z dimensions. Each number must be >= 1. Generated by the1958 ``amdgpu_max_num_work_groups`` CLANG attribute [CLANG-ATTR]_. Clang only1959 emits this attribute when all the three numbers are >= 1.1960 1961 "amdgpu-hidden-argument" This attribute is used internally by the backend to mark function arguments1962 as hidden. Hidden arguments are managed by the compiler and are not part of1963 the explicit arguments supplied by the user.1964 1965 "amdgpu-agpr-alloc"="min(,max)" Indicates a minimum and maximum range for the number of AGPRs to make1966 available to allocate. The values will be rounded up to the next multiple1967 of the allocation granularity (4). The minimum value is interpreted as the1968 minimum required number of AGPRs for the function to allocate (that is, the1969 function requires no more than min registers). If only one value is specified,1970 it is interpreted as the minimum register budget. The maximum will restrict1971 allocation to use no more than max AGPRs.1972 1973 The values may be ignored if satisfying it would violate other allocation1974 constraints.1975 1976 The behavior is undefined if a function which requires more AGPRs than the1977 lower bound is reached through any function marked with a higher value of this1978 attribute. A minimum value of 0 indicates the function does not require1979 any AGPRs.1980 1981 This is only relevant on targets with AGPRs which support accum_offset (gfx90a+).1982 1983 "amdgpu-sgpr-hazard-wait" Disabled SGPR hazard wait insertion if set to 0.1984 Exists for testing performance impact of SGPR hazard waits only.1985 1986 "amdgpu-sgpr-hazard-boundary-cull" Enable insertion of SGPR hazard cull sequences at function call boundaries.1987 Cull sequence reduces future hazard waits, but has a performance cost.1988 1989 "amdgpu-sgpr-hazard-mem-wait-cull" Enable insertion of SGPR hazard cull sequences before memory waits.1990 Cull sequence reduces future hazard waits, but has a performance cost.1991 Attempt to amortize cost by overlapping with memory accesses.1992 1993 "amdgpu-sgpr-hazard-mem-wait-cull-threshold" Sets the number of active SGPR hazards that must be present before1994 inserting a cull sequence at a memory wait.1995 1996 "amdgpu-promote-alloca-to-vector-max-regs" Maximum vector size (in 32b registers) to create when promoting alloca.1997 1998 "amdgpu-promote-alloca-to-vector-vgpr-ratio" Ratio of VGPRs to budget for promoting alloca to vectors.1999 2000 "amdgpu-dynamic-vgpr-block-size" Represents the size of a VGPR block in the "Dynamic VGPR" hardware mode,2001 introduced in GFX12.2002 A value of 0 (default) means that dynamic VGPRs are not enabled.2003 Valid values for GFX12+ are 16 and 32.2004 Waves launched in this mode may allocate or deallocate the VGPRs2005 using dedicated instructions, but may not send the DEALLOC_VGPRS2006 message. If a shader has this attribute, then all its callees must2007 match its value.2008 An amd_cs_chain CC function with this enabled has an extra symbol2009 prefixed with "_dvgpr$" with the value of the function symbol,2010 offset by one less than the number of dynamic VGPR blocks required2011 by the function encoded in bits 5..3.2012 2013 "amdgpu-cluster-dims"="x,y,z" Specify the cluster workgroup dimensions. A value of "0,0,0" indicates that2014 cluster is disabled. A value of "1024,1024,1024" indicates that cluster is enabled,2015 but the dimensions cannot be determined at compile time. Any other value explicitly2016 specifies the cluster dimensions.2017 2018 This is only relevant on targets with cluster support.2019 2020 2021 ================================================ ==========================================================2022 2023Calling Conventions2024===================2025 2026The AMDGPU backend supports the following calling conventions:2027 2028 .. table:: AMDGPU Calling Conventions2029 :name: amdgpu-cc2030 2031 =============================== ==========================================================2032 Calling Convention Description2033 =============================== ==========================================================2034 ``ccc`` The C calling convention. Used by default.2035 See :ref:`amdgpu-amdhsa-function-call-convention-non-kernel-functions`2036 for more details.2037 2038 ``fastcc`` The fast calling convention. Mostly the same as the ``ccc``.2039 2040 ``coldcc`` The cold calling convention. Mostly the same as the ``ccc``.2041 2042 ``amdgpu_cs`` Used for Mesa/AMDPAL compute shaders.2043 ..TODO::2044 Describe.2045 2046 ``amdgpu_cs_chain`` Similar to ``amdgpu_cs``, with differences described below.2047 2048 Functions with this calling convention cannot be called directly. They must2049 instead be launched via the ``llvm.amdgcn.cs.chain`` intrinsic.2050 2051 Arguments are passed in SGPRs, starting at s0, if they have the ``inreg``2052 attribute, and in VGPRs otherwise, starting at v8. Using more SGPRs or VGPRs2053 than available in the subtarget is not allowed. On subtargets that use2054 a scratch buffer descriptor (as opposed to ``scratch_{load,store}_*`` instructions),2055 the scratch buffer descriptor is passed in s[48:51]. This limits the2056 SGPR / ``inreg`` arguments to the equivalent of 48 dwords; using more2057 than that is not allowed.2058 2059 The return type must be void.2060 Varargs, sret, byval, byref, inalloca, preallocated are not supported.2061 2062 Values in scalar registers as well as v0-v7 are not preserved. Values in2063 VGPRs starting at v8 are not preserved for the active lanes, but must be2064 saved by the callee for inactive lanes when using WWM (a notable exception is2065 when the llvm.amdgcn.init.whole.wave intrinsic is used in the function - in this2066 case the backend assumes that there are no inactive lanes upon entry; any inactive2067 lanes that need to be preserved must be explicitly present in the IR).2068 2069 Wave scratch is "empty" at function boundaries. There is no stack pointer input2070 or output value, but functions are free to use scratch starting from an initial2071 stack pointer. Calls to ``amdgpu_gfx`` functions are allowed and behave like they2072 do in ``amdgpu_cs`` functions.2073 2074 All counters (``lgkmcnt``, ``vmcnt``, ``storecnt``, etc.) are presumed in an2075 unknown state at function entry.2076 2077 A function may have multiple exits (e.g. one chain exit and one plain ``ret void``2078 for when the wave ends), but all ``llvm.amdgcn.cs.chain`` exits must be in2079 uniform control flow.2080 2081 ``amdgpu_cs_chain_preserve`` Same as ``amdgpu_cs_chain``, but active lanes for VGPRs starting at v8 are preserved.2082 Calls to ``amdgpu_gfx`` functions are not allowed, and any calls to ``llvm.amdgcn.cs.chain``2083 must not pass more VGPR arguments than the caller's VGPR function parameters.2084 2085 ``amdgpu_es`` Used for AMDPAL shader stage before geometry shader if geometry is in2086 use. So either the domain (= tessellation evaluation) shader if2087 tessellation is in use, or otherwise the vertex shader.2088 ..TODO::2089 Describe.2090 2091 ``amdgpu_gfx`` Used for AMD graphics targets. Functions with this calling convention2092 cannot be used as entry points.2093 ..TODO::2094 Describe.2095 2096 ``amdgpu_gfx_whole_wave`` Used for AMD graphics targets. Functions with this calling convention2097 cannot be used as entry points. They must have an i1 as the first argument,2098 which will be mapped to the value of EXEC on entry into the function. Other2099 arguments will contain poison in their inactive lanes. Similarly, the return2100 value for the inactive lanes is poison.2101 2102 The function will run with all lanes enabled, i.e. EXEC will be set to -1 in the2103 prologue and restored to its original value in the epilogue. The inactive lanes2104 will be preserved for all the registers used by the function. Active lanes only2105 will only be preserved for the callee saved registers.2106 2107 In all other respects, functions with this calling convention behave like2108 ``amdgpu_gfx`` functions.2109 2110 ``amdgpu_gs`` Used for Mesa/AMDPAL geometry shaders.2111 ..TODO::2112 Describe.2113 2114 ``amdgpu_hs`` Used for Mesa/AMDPAL hull shaders (= tessellation control shaders).2115 ..TODO::2116 Describe.2117 2118 ``amdgpu_kernel`` See :ref:`amdgpu-amdhsa-function-call-convention-kernel-functions`2119 2120 ``amdgpu_ls`` Used for AMDPAL vertex shader if tessellation is in use.2121 ..TODO::2122 Describe.2123 2124 ``amdgpu_ps`` Used for Mesa/AMDPAL pixel shaders.2125 ..TODO::2126 Describe.2127 2128 ``amdgpu_vs`` Used for Mesa/AMDPAL last shader stage before rasterization (vertex2129 shader if tessellation and geometry are not in use, or otherwise2130 copy shader if one is needed).2131 ..TODO::2132 Describe.2133 2134 =============================== ==========================================================2135 2136AMDGPU MCExpr2137-------------2138 2139As part of the AMDGPU MC layer, AMDGPU provides the following target-specific2140``MCExpr``\s.2141 2142 .. table:: AMDGPU MCExpr types:2143 :name: amdgpu-mcexpr-table2144 2145 =================== ================= ========================================================2146 MCExpr Operands Return value2147 =================== ================= ========================================================2148 ``max(arg, ...)`` 1 or more Variadic signed operation that returns the maximum2149 value of all its arguments.2150 2151 ``or(arg, ...)`` 1 or more Variadic signed operation that returns the bitwise-or2152 result of all its arguments.2153 2154 =================== ================= ========================================================2155 2156Function Resource Usage2157-----------------------2158 2159A function's resource usage depends on each of its callees' resource usage. The2160expressions used to denote resource usage reflect this by propagating each2161callees' equivalent expressions. Said expressions are emitted as symbols by the2162compiler when compiling to either assembly or object format and should not be2163overwritten or redefined.2164 2165The following describes all emitted function resource usage symbols:2166 2167 .. table:: Function Resource Usage:2168 :name: function-usage-table2169 2170 ===================================== ========= ========================================= ===============================================================================2171 Symbol Type Description Example2172 ===================================== ========= ========================================= ===============================================================================2173 <function_name>.num_vgpr Integer Number of VGPRs used by <function_name>, .set foo.num_vgpr, max(32, bar.num_vgpr, baz.num_vgpr)2174 worst case of itself and its callees'2175 VGPR use2176 <function_name>.num_agpr Integer Number of AGPRs used by <function_name>, .set foo.num_agpr, max(35, bar.num_agpr)2177 worst case of itself and its callees'2178 AGPR use2179 <function_name>.numbered_sgpr Integer Number of SGPRs used by <function_name>, .set foo.num_sgpr, 212180 worst case of itself and its callees'2181 SGPR use (without any of the implicitly2182 used SGPRs)2183 <function_name>.private_seg_size Integer Total stack size required for .set foo.private_seg_size, 16+max(bar.private_seg_size, baz.private_seg_size)2184 <function_name>, expression is the2185 locally used stack size + the worst case2186 callee2187 <function_name>.uses_vcc Bool Whether <function_name>, or any of its .set foo.uses_vcc, or(0, bar.uses_vcc)2188 callees, uses vcc2189 <function_name>.uses_flat_scratch Bool Whether <function_name>, or any of its .set foo.uses_flat_scratch, 12190 callees, uses flat scratch or not2191 <function_name>.has_dyn_sized_stack Bool Whether <function_name>, or any of its .set foo.has_dyn_sized_stack, 12192 callees, is dynamically sized2193 <function_name>.has_recursion Bool Whether <function_name>, or any of its .set foo.has_recursion, 02194 callees, contains recursion2195 <function_name>.has_indirect_call Bool Whether <function_name>, or any of its .set foo.has_indirect_call, max(0, bar.has_indirect_call)2196 callees, contains an indirect call2197 ===================================== ========= ========================================= ===============================================================================2198 2199Furthermore, three symbols are additionally emitted describing the compilation2200unit's worst case (i.e, maxima) ``num_vgpr``, ``num_agpr``, and2201``numbered_sgpr`` which may be referenced and used by the aforementioned2202symbolic expressions. These three symbols are ``amdgcn.max_num_vgpr``,2203``amdgcn.max_num_agpr``, and ``amdgcn.max_num_sgpr``.2204 2205.. _amdgpu-elf-code-object:2206 2207ELF Code Object2208===============2209 2210The AMDGPU backend generates a standard ELF [ELF]_ relocatable code object that2211can be linked by ``lld`` to produce a standard ELF shared code object which can2212be loaded and executed on an AMDGPU target.2213 2214.. _amdgpu-elf-header:2215 2216Header2217------2218 2219The AMDGPU backend uses the following ELF header:2220 2221 .. table:: AMDGPU ELF Header2222 :name: amdgpu-elf-header-table2223 2224 ========================== ===============================2225 Field Value2226 ========================== ===============================2227 ``e_ident[EI_CLASS]`` ``ELFCLASS64``2228 ``e_ident[EI_DATA]`` ``ELFDATA2LSB``2229 ``e_ident[EI_OSABI]`` - ``ELFOSABI_NONE``2230 - ``ELFOSABI_AMDGPU_HSA``2231 - ``ELFOSABI_AMDGPU_PAL``2232 - ``ELFOSABI_AMDGPU_MESA3D``2233 ``e_ident[EI_ABIVERSION]`` - ``ELFABIVERSION_AMDGPU_HSA_V2``2234 - ``ELFABIVERSION_AMDGPU_HSA_V3``2235 - ``ELFABIVERSION_AMDGPU_HSA_V4``2236 - ``ELFABIVERSION_AMDGPU_HSA_V5``2237 - ``ELFABIVERSION_AMDGPU_HSA_V6``2238 - ``ELFABIVERSION_AMDGPU_PAL``2239 - ``ELFABIVERSION_AMDGPU_MESA3D``2240 ``e_type`` - ``ET_REL``2241 - ``ET_DYN``2242 ``e_machine`` ``EM_AMDGPU``2243 ``e_entry`` 02244 ``e_flags`` See :ref:`amdgpu-elf-header-e_flags-v2-table`,2245 :ref:`amdgpu-elf-header-e_flags-table-v3`,2246 :ref:`amdgpu-elf-header-e_flags-table-v4-v5`,2247 and :ref:`amdgpu-elf-header-e_flags-table-v6-onwards`2248 ========================== ===============================2249 2250..2251 2252 .. table:: AMDGPU ELF Header Enumeration Values2253 :name: amdgpu-elf-header-enumeration-values-table2254 2255 =============================== =====2256 Name Value2257 =============================== =====2258 ``EM_AMDGPU`` 2242259 ``ELFOSABI_NONE`` 02260 ``ELFOSABI_AMDGPU_HSA`` 642261 ``ELFOSABI_AMDGPU_PAL`` 652262 ``ELFOSABI_AMDGPU_MESA3D`` 662263 ``ELFABIVERSION_AMDGPU_HSA_V2`` 02264 ``ELFABIVERSION_AMDGPU_HSA_V3`` 12265 ``ELFABIVERSION_AMDGPU_HSA_V4`` 22266 ``ELFABIVERSION_AMDGPU_HSA_V5`` 32267 ``ELFABIVERSION_AMDGPU_HSA_V6`` 42268 ``ELFABIVERSION_AMDGPU_PAL`` 02269 ``ELFABIVERSION_AMDGPU_MESA3D`` 02270 =============================== =====2271 2272``e_ident[EI_CLASS]``2273 The ELF class is:2274 2275 * ``ELFCLASS32`` for ``r600`` architecture.2276 2277 * ``ELFCLASS64`` for ``amdgcn`` architecture which only supports 64-bit2278 process address space applications.2279 2280``e_ident[EI_DATA]``2281 All AMDGPU targets use ``ELFDATA2LSB`` for little-endian byte ordering.2282 2283``e_ident[EI_OSABI]``2284 One of the following AMDGPU target architecture specific OS ABIs2285 (see :ref:`amdgpu-os`):2286 2287 * ``ELFOSABI_NONE`` for *unknown* OS.2288 2289 * ``ELFOSABI_AMDGPU_HSA`` for ``amdhsa`` OS.2290 2291 * ``ELFOSABI_AMDGPU_PAL`` for ``amdpal`` OS.2292 2293 * ``ELFOSABI_AMDGPU_MESA3D`` for ``mesa3D`` OS.2294 2295``e_ident[EI_ABIVERSION]``2296 The ABI version of the AMDGPU target architecture specific OS ABI to which the code2297 object conforms:2298 2299 * ``ELFABIVERSION_AMDGPU_HSA_V2`` is used to specify the version of AMD HSA2300 runtime ABI for code object V2. Can no longer be emitted by this version of LLVM.2301 2302 * ``ELFABIVERSION_AMDGPU_HSA_V3`` is used to specify the version of AMD HSA2303 runtime ABI for code object V3. Can no longer be emitted by this version of LLVM.2304 2305 * ``ELFABIVERSION_AMDGPU_HSA_V4`` is used to specify the version of AMD HSA2306 runtime ABI for code object V4. Specify using the Clang option2307 ``-mcode-object-version=4``.2308 2309 * ``ELFABIVERSION_AMDGPU_HSA_V5`` is used to specify the version of AMD HSA2310 runtime ABI for code object V5. Specify using the Clang option2311 ``-mcode-object-version=5``. This is the default code object2312 version if not specified.2313 2314 * ``ELFABIVERSION_AMDGPU_HSA_V6`` is used to specify the version of AMD HSA2315 runtime ABI for code object V6. Specify using the Clang option2316 ``-mcode-object-version=6``.2317 2318 * ``ELFABIVERSION_AMDGPU_PAL`` is used to specify the version of AMD PAL2319 runtime ABI.2320 2321 * ``ELFABIVERSION_AMDGPU_MESA3D`` is used to specify the version of AMD MESA2322 3D runtime ABI.2323 2324``e_type``2325 Can be one of the following values:2326 2327 2328 ``ET_REL``2329 The type produced by the AMDGPU backend compiler as it is relocatable code2330 object.2331 2332 ``ET_DYN``2333 The type produced by the linker as it is a shared code object.2334 2335 The AMD HSA runtime loader requires a ``ET_DYN`` code object.2336 2337``e_machine``2338 The value ``EM_AMDGPU`` is used for the machine for all processors supported2339 by the ``r600`` and ``amdgcn`` architectures (see2340 :ref:`amdgpu-processor-table`). The specific processor is specified in the2341 ``NT_AMD_HSA_ISA_VERSION`` note record for code object V2 (see2342 :ref:`amdgpu-note-records-v2`) and in the ``EF_AMDGPU_MACH`` bit field of the2343 ``e_flags`` for code object V3 and above (see2344 :ref:`amdgpu-elf-header-e_flags-table-v3`,2345 :ref:`amdgpu-elf-header-e_flags-table-v4-v5` and2346 :ref:`amdgpu-elf-header-e_flags-table-v6-onwards`).2347 2348``e_entry``2349 The entry point is 0 as the entry points for individual kernels must be2350 selected in order to invoke them through AQL packets.2351 2352``e_flags``2353 The AMDGPU backend uses the following ELF header flags:2354 2355 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V22356 :name: amdgpu-elf-header-e_flags-v2-table2357 2358 ===================================== ===== =============================2359 Name Value Description2360 ===================================== ===== =============================2361 ``EF_AMDGPU_FEATURE_XNACK_V2`` 0x01 Indicates if the ``xnack``2362 target feature is2363 enabled for all code2364 contained in the code object.2365 If the processor2366 does not support the2367 ``xnack`` target2368 feature then must2369 be 0.2370 See2371 :ref:`amdgpu-target-features`.2372 ``EF_AMDGPU_FEATURE_TRAP_HANDLER_V2`` 0x02 Indicates if the trap2373 handler is enabled for all2374 code contained in the code2375 object. If the processor2376 does not support a trap2377 handler then must be 0.2378 See2379 :ref:`amdgpu-target-features`.2380 ===================================== ===== =============================2381 2382 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V32383 :name: amdgpu-elf-header-e_flags-table-v32384 2385 ================================= ===== =============================2386 Name Value Description2387 ================================= ===== =============================2388 ``EF_AMDGPU_MACH`` 0x0ff AMDGPU processor selection2389 mask for2390 ``EF_AMDGPU_MACH_xxx`` values2391 defined in2392 :ref:`amdgpu-ef-amdgpu-mach-table`.2393 ``EF_AMDGPU_FEATURE_XNACK_V3`` 0x100 Indicates if the ``xnack``2394 target feature is2395 enabled for all code2396 contained in the code object.2397 If the processor2398 does not support the2399 ``xnack`` target2400 feature then must2401 be 0.2402 See2403 :ref:`amdgpu-target-features`.2404 ``EF_AMDGPU_FEATURE_SRAMECC_V3`` 0x200 Indicates if the ``sramecc``2405 target feature is2406 enabled for all code2407 contained in the code object.2408 If the processor2409 does not support the2410 ``sramecc`` target2411 feature then must2412 be 0.2413 See2414 :ref:`amdgpu-target-features`.2415 ================================= ===== =============================2416 2417 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V4 and V52418 :name: amdgpu-elf-header-e_flags-table-v4-v52419 2420 ============================================ ===== ===================================2421 Name Value Description2422 ============================================ ===== ===================================2423 ``EF_AMDGPU_MACH`` 0x0ff AMDGPU processor selection2424 mask for2425 ``EF_AMDGPU_MACH_xxx`` values2426 defined in2427 :ref:`amdgpu-ef-amdgpu-mach-table`.2428 ``EF_AMDGPU_FEATURE_XNACK_V4`` 0x300 XNACK selection mask for2429 ``EF_AMDGPU_FEATURE_XNACK_*_V4``2430 values.2431 ``EF_AMDGPU_FEATURE_XNACK_UNSUPPORTED_V4`` 0x000 XNACK unsupported.2432 ``EF_AMDGPU_FEATURE_XNACK_ANY_V4`` 0x100 XNACK can have any value.2433 ``EF_AMDGPU_FEATURE_XNACK_OFF_V4`` 0x200 XNACK disabled.2434 ``EF_AMDGPU_FEATURE_XNACK_ON_V4`` 0x300 XNACK enabled.2435 ``EF_AMDGPU_FEATURE_SRAMECC_V4`` 0xc00 SRAMECC selection mask for2436 ``EF_AMDGPU_FEATURE_SRAMECC_*_V4``2437 values.2438 ``EF_AMDGPU_FEATURE_SRAMECC_UNSUPPORTED_V4`` 0x000 SRAMECC unsupported.2439 ``EF_AMDGPU_FEATURE_SRAMECC_ANY_V4`` 0x400 SRAMECC can have any value.2440 ``EF_AMDGPU_FEATURE_SRAMECC_OFF_V4`` 0x800 SRAMECC disabled,2441 ``EF_AMDGPU_FEATURE_SRAMECC_ON_V4`` 0xc00 SRAMECC enabled.2442 ============================================ ===== ===================================2443 2444 .. table:: AMDGPU ELF Header ``e_flags`` for Code Object V6 and After2445 :name: amdgpu-elf-header-e_flags-table-v6-onwards2446 2447 ============================================ ========== =========================================2448 Name Value Description2449 ============================================ ========== =========================================2450 ``EF_AMDGPU_MACH`` 0x0ff AMDGPU processor selection2451 mask for2452 ``EF_AMDGPU_MACH_xxx`` values2453 defined in2454 :ref:`amdgpu-ef-amdgpu-mach-table`.2455 ``EF_AMDGPU_FEATURE_XNACK_V4`` 0x300 XNACK selection mask for2456 ``EF_AMDGPU_FEATURE_XNACK_*_V4``2457 values.2458 ``EF_AMDGPU_FEATURE_XNACK_UNSUPPORTED_V4`` 0x000 XNACK unsupported.2459 ``EF_AMDGPU_FEATURE_XNACK_ANY_V4`` 0x100 XNACK can have any value.2460 ``EF_AMDGPU_FEATURE_XNACK_OFF_V4`` 0x200 XNACK disabled.2461 ``EF_AMDGPU_FEATURE_XNACK_ON_V4`` 0x300 XNACK enabled.2462 ``EF_AMDGPU_FEATURE_SRAMECC_V4`` 0xc00 SRAMECC selection mask for2463 ``EF_AMDGPU_FEATURE_SRAMECC_*_V4``2464 values.2465 ``EF_AMDGPU_FEATURE_SRAMECC_UNSUPPORTED_V4`` 0x000 SRAMECC unsupported.2466 ``EF_AMDGPU_FEATURE_SRAMECC_ANY_V4`` 0x400 SRAMECC can have any value.2467 ``EF_AMDGPU_FEATURE_SRAMECC_OFF_V4`` 0x800 SRAMECC disabled,2468 ``EF_AMDGPU_FEATURE_SRAMECC_ON_V4`` 0xc00 SRAMECC enabled.2469 ``EF_AMDGPU_GENERIC_VERSION_V`` 0xff000000 Generic code object version selection2470 mask. This is a value between 1 and 255,2471 stored in the most significant byte2472 of EFLAGS.2473 See :ref:`amdgpu-generic-processor-versioning`2474 ============================================ ========== =========================================2475 2476 .. table:: AMDGPU ``EF_AMDGPU_MACH`` Values2477 :name: amdgpu-ef-amdgpu-mach-table2478 2479 ========================================== ========== =============================2480 Name Value Description (see2481 :ref:`amdgpu-processor-table`)2482 ========================================== ========== =============================2483 ``EF_AMDGPU_MACH_NONE`` 0x000 *not specified*2484 ``EF_AMDGPU_MACH_R600_R600`` 0x001 ``r600``2485 ``EF_AMDGPU_MACH_R600_R630`` 0x002 ``r630``2486 ``EF_AMDGPU_MACH_R600_RS880`` 0x003 ``rs880``2487 ``EF_AMDGPU_MACH_R600_RV670`` 0x004 ``rv670``2488 ``EF_AMDGPU_MACH_R600_RV710`` 0x005 ``rv710``2489 ``EF_AMDGPU_MACH_R600_RV730`` 0x006 ``rv730``2490 ``EF_AMDGPU_MACH_R600_RV770`` 0x007 ``rv770``2491 ``EF_AMDGPU_MACH_R600_CEDAR`` 0x008 ``cedar``2492 ``EF_AMDGPU_MACH_R600_CYPRESS`` 0x009 ``cypress``2493 ``EF_AMDGPU_MACH_R600_JUNIPER`` 0x00a ``juniper``2494 ``EF_AMDGPU_MACH_R600_REDWOOD`` 0x00b ``redwood``2495 ``EF_AMDGPU_MACH_R600_SUMO`` 0x00c ``sumo``2496 ``EF_AMDGPU_MACH_R600_BARTS`` 0x00d ``barts``2497 ``EF_AMDGPU_MACH_R600_CAICOS`` 0x00e ``caicos``2498 ``EF_AMDGPU_MACH_R600_CAYMAN`` 0x00f ``cayman``2499 ``EF_AMDGPU_MACH_R600_TURKS`` 0x010 ``turks``2500 *reserved* 0x011 - Reserved for ``r600``2501 0x01f architecture processors.2502 ``EF_AMDGPU_MACH_AMDGCN_GFX600`` 0x020 ``gfx600``2503 ``EF_AMDGPU_MACH_AMDGCN_GFX601`` 0x021 ``gfx601``2504 ``EF_AMDGPU_MACH_AMDGCN_GFX700`` 0x022 ``gfx700``2505 ``EF_AMDGPU_MACH_AMDGCN_GFX701`` 0x023 ``gfx701``2506 ``EF_AMDGPU_MACH_AMDGCN_GFX702`` 0x024 ``gfx702``2507 ``EF_AMDGPU_MACH_AMDGCN_GFX703`` 0x025 ``gfx703``2508 ``EF_AMDGPU_MACH_AMDGCN_GFX704`` 0x026 ``gfx704``2509 *reserved* 0x027 Reserved.2510 ``EF_AMDGPU_MACH_AMDGCN_GFX801`` 0x028 ``gfx801``2511 ``EF_AMDGPU_MACH_AMDGCN_GFX802`` 0x029 ``gfx802``2512 ``EF_AMDGPU_MACH_AMDGCN_GFX803`` 0x02a ``gfx803``2513 ``EF_AMDGPU_MACH_AMDGCN_GFX810`` 0x02b ``gfx810``2514 ``EF_AMDGPU_MACH_AMDGCN_GFX900`` 0x02c ``gfx900``2515 ``EF_AMDGPU_MACH_AMDGCN_GFX902`` 0x02d ``gfx902``2516 ``EF_AMDGPU_MACH_AMDGCN_GFX904`` 0x02e ``gfx904``2517 ``EF_AMDGPU_MACH_AMDGCN_GFX906`` 0x02f ``gfx906``2518 ``EF_AMDGPU_MACH_AMDGCN_GFX908`` 0x030 ``gfx908``2519 ``EF_AMDGPU_MACH_AMDGCN_GFX909`` 0x031 ``gfx909``2520 ``EF_AMDGPU_MACH_AMDGCN_GFX90C`` 0x032 ``gfx90c``2521 ``EF_AMDGPU_MACH_AMDGCN_GFX1010`` 0x033 ``gfx1010``2522 ``EF_AMDGPU_MACH_AMDGCN_GFX1011`` 0x034 ``gfx1011``2523 ``EF_AMDGPU_MACH_AMDGCN_GFX1012`` 0x035 ``gfx1012``2524 ``EF_AMDGPU_MACH_AMDGCN_GFX1030`` 0x036 ``gfx1030``2525 ``EF_AMDGPU_MACH_AMDGCN_GFX1031`` 0x037 ``gfx1031``2526 ``EF_AMDGPU_MACH_AMDGCN_GFX1032`` 0x038 ``gfx1032``2527 ``EF_AMDGPU_MACH_AMDGCN_GFX1033`` 0x039 ``gfx1033``2528 ``EF_AMDGPU_MACH_AMDGCN_GFX602`` 0x03a ``gfx602``2529 ``EF_AMDGPU_MACH_AMDGCN_GFX705`` 0x03b ``gfx705``2530 ``EF_AMDGPU_MACH_AMDGCN_GFX805`` 0x03c ``gfx805``2531 ``EF_AMDGPU_MACH_AMDGCN_GFX1035`` 0x03d ``gfx1035``2532 ``EF_AMDGPU_MACH_AMDGCN_GFX1034`` 0x03e ``gfx1034``2533 ``EF_AMDGPU_MACH_AMDGCN_GFX90A`` 0x03f ``gfx90a``2534 *reserved* 0x040 Reserved.2535 ``EF_AMDGPU_MACH_AMDGCN_GFX1100`` 0x041 ``gfx1100``2536 ``EF_AMDGPU_MACH_AMDGCN_GFX1013`` 0x042 ``gfx1013``2537 ``EF_AMDGPU_MACH_AMDGCN_GFX1150`` 0x043 ``gfx1150``2538 ``EF_AMDGPU_MACH_AMDGCN_GFX1103`` 0x044 ``gfx1103``2539 ``EF_AMDGPU_MACH_AMDGCN_GFX1036`` 0x045 ``gfx1036``2540 ``EF_AMDGPU_MACH_AMDGCN_GFX1101`` 0x046 ``gfx1101``2541 ``EF_AMDGPU_MACH_AMDGCN_GFX1102`` 0x047 ``gfx1102``2542 ``EF_AMDGPU_MACH_AMDGCN_GFX1200`` 0x048 ``gfx1200``2543 ``EF_AMDGPU_MACH_AMDGCN_GFX1250`` 0x049 ``gfx1250``2544 ``EF_AMDGPU_MACH_AMDGCN_GFX1151`` 0x04a ``gfx1151``2545 *reserved* 0x04b Reserved.2546 ``EF_AMDGPU_MACH_AMDGCN_GFX942`` 0x04c ``gfx942``2547 *reserved* 0x04d Reserved.2548 ``EF_AMDGPU_MACH_AMDGCN_GFX1201`` 0x04e ``gfx1201``2549 ``EF_AMDGPU_MACH_AMDGCN_GFX950`` 0x04f ``gfx950``2550 *reserved* 0x050 Reserved.2551 ``EF_AMDGPU_MACH_AMDGCN_GFX9_GENERIC`` 0x051 ``gfx9-generic``2552 ``EF_AMDGPU_MACH_AMDGCN_GFX10_1_GENERIC`` 0x052 ``gfx10-1-generic``2553 ``EF_AMDGPU_MACH_AMDGCN_GFX10_3_GENERIC`` 0x053 ``gfx10-3-generic``2554 ``EF_AMDGPU_MACH_AMDGCN_GFX11_GENERIC`` 0x054 ``gfx11-generic``2555 ``EF_AMDGPU_MACH_AMDGCN_GFX1152`` 0x055 ``gfx1152``.2556 *reserved* 0x056 Reserved.2557 *reserved* 0x057 Reserved.2558 ``EF_AMDGPU_MACH_AMDGCN_GFX1153`` 0x058 ``gfx1153``.2559 ``EF_AMDGPU_MACH_AMDGCN_GFX12_GENERIC`` 0x059 ``gfx12-generic``2560 ``EF_AMDGPU_MACH_AMDGCN_GFX1251`` 0x05a ``gfx1251``2561 ``EF_AMDGPU_MACH_AMDGCN_GFX9_4_GENERIC`` 0x05f ``gfx9-4-generic``2562 ========================================== ========== =============================2563 2564Sections2565--------2566 2567An AMDGPU target ELF code object has the standard ELF sections which include:2568 2569 .. table:: AMDGPU ELF Sections2570 :name: amdgpu-elf-sections-table2571 2572 ================== ================ =================================2573 Name Type Attributes2574 ================== ================ =================================2575 ``.bss`` ``SHT_NOBITS`` ``SHF_ALLOC`` + ``SHF_WRITE``2576 ``.data`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE``2577 ``.debug_``\ *\** ``SHT_PROGBITS`` *none*2578 ``.dynamic`` ``SHT_DYNAMIC`` ``SHF_ALLOC``2579 ``.dynstr`` ``SHT_PROGBITS`` ``SHF_ALLOC``2580 ``.dynsym`` ``SHT_PROGBITS`` ``SHF_ALLOC``2581 ``.got`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_WRITE``2582 ``.hash`` ``SHT_HASH`` ``SHF_ALLOC``2583 ``.note`` ``SHT_NOTE`` *none*2584 ``.rela``\ *name* ``SHT_RELA`` *none*2585 ``.rela.dyn`` ``SHT_RELA`` *none*2586 ``.rodata`` ``SHT_PROGBITS`` ``SHF_ALLOC``2587 ``.shstrtab`` ``SHT_STRTAB`` *none*2588 ``.strtab`` ``SHT_STRTAB`` *none*2589 ``.symtab`` ``SHT_SYMTAB`` *none*2590 ``.text`` ``SHT_PROGBITS`` ``SHF_ALLOC`` + ``SHF_EXECINSTR``2591 ================== ================ =================================2592 2593These sections have their standard meanings (see [ELF]_) and are only generated2594if needed.2595 2596``.debug``\ *\**2597 The standard DWARF sections. See :ref:`amdgpu-dwarf-debug-information` for2598 information on the DWARF produced by the AMDGPU backend.2599 2600``.dynamic``, ``.dynstr``, ``.dynsym``, ``.hash``2601 The standard sections used by a dynamic loader.2602 2603``.note``2604 See :ref:`amdgpu-note-records` for the note records supported by the AMDGPU2605 backend.2606 2607``.rela``\ *name*, ``.rela.dyn``2608 For relocatable code objects, *name* is the name of the section that the2609 relocation records apply. For example, ``.rela.text`` is the section name for2610 relocation records associated with the ``.text`` section.2611 2612 For linked shared code objects, ``.rela.dyn`` contains all the relocation2613 records from each of the relocatable code object's ``.rela``\ *name* sections.2614 2615 See :ref:`amdgpu-relocation-records` for the relocation records supported by2616 the AMDGPU backend.2617 2618``.text``2619 The executable machine code for the kernels and functions they call. Generated2620 as position independent code. See :ref:`amdgpu-code-conventions` for2621 information on conventions used in the isa generation.2622 2623``.amdgpu.kernel.runtime.handle``2624 Symbols used for device enqueue.2625 2626.. _amdgpu-note-records:2627 2628Note Records2629------------2630 2631The AMDGPU backend code object contains ELF note records in the ``.note``2632section. The set of generated notes and their semantics depend on the code2633object version; see :ref:`amdgpu-note-records-v2` and2634:ref:`amdgpu-note-records-v3-onwards`.2635 2636As required by ``ELFCLASS32`` and ``ELFCLASS64``, minimal zero-byte padding2637must be generated after the ``name`` field to ensure the ``desc`` field is 42638byte aligned. In addition, minimal zero-byte padding must be generated to2639ensure the ``desc`` field size is a multiple of 4 bytes. The ``sh_addralign``2640field of the ``.note`` section must be at least 4 to indicate at least 8 byte2641alignment.2642 2643.. _amdgpu-note-records-v2:2644 2645Code Object V2 Note Records2646~~~~~~~~~~~~~~~~~~~~~~~~~~~2647 2648.. warning::2649 Code object V2 generation is no longer supported by this version of LLVM.2650 2651The AMDGPU backend code object uses the following ELF note record in the2652``.note`` section when compiling for code object V2.2653 2654The note record vendor field is "AMD".2655 2656Additional note records may be present, but any which are not documented here2657are deprecated and should not be used.2658 2659 .. table:: AMDGPU Code Object V2 ELF Note Records2660 :name: amdgpu-elf-note-records-v2-table2661 2662 ===== ===================================== ======================================2663 Name Type Description2664 ===== ===================================== ======================================2665 "AMD" ``NT_AMD_HSA_CODE_OBJECT_VERSION`` Code object version.2666 "AMD" ``NT_AMD_HSA_HSAIL`` HSAIL properties generated by the HSAIL2667 Finalizer and not the LLVM compiler.2668 "AMD" ``NT_AMD_HSA_ISA_VERSION`` Target ISA version.2669 "AMD" ``NT_AMD_HSA_METADATA`` Metadata null terminated string in2670 YAML [YAML]_ textual format.2671 "AMD" ``NT_AMD_HSA_ISA_NAME`` Target ISA name.2672 ===== ===================================== ======================================2673 2674..2675 2676 .. table:: AMDGPU Code Object V2 ELF Note Record Enumeration Values2677 :name: amdgpu-elf-note-record-enumeration-values-v2-table2678 2679 ===================================== =====2680 Name Value2681 ===================================== =====2682 ``NT_AMD_HSA_CODE_OBJECT_VERSION`` 12683 ``NT_AMD_HSA_HSAIL`` 22684 ``NT_AMD_HSA_ISA_VERSION`` 32685 *reserved* 4-92686 ``NT_AMD_HSA_METADATA`` 102687 ``NT_AMD_HSA_ISA_NAME`` 112688 ===================================== =====2689 2690``NT_AMD_HSA_CODE_OBJECT_VERSION``2691 Specifies the code object version number. The description field has the2692 following layout:2693 2694 .. code:: c2695 2696 struct amdgpu_hsa_note_code_object_version_s {2697 uint32_t major_version;2698 uint32_t minor_version;2699 };2700 2701 The ``major_version`` has a value less than or equal to 2.2702 2703``NT_AMD_HSA_HSAIL``2704 Specifies the HSAIL properties used by the HSAIL Finalizer. The description2705 field has the following layout:2706 2707 .. code:: c2708 2709 struct amdgpu_hsa_note_hsail_s {2710 uint32_t hsail_major_version;2711 uint32_t hsail_minor_version;2712 uint8_t profile;2713 uint8_t machine_model;2714 uint8_t default_float_round;2715 };2716 2717``NT_AMD_HSA_ISA_VERSION``2718 Specifies the target ISA version. The description field has the following layout:2719 2720 .. code:: c2721 2722 struct amdgpu_hsa_note_isa_s {2723 uint16_t vendor_name_size;2724 uint16_t architecture_name_size;2725 uint32_t major;2726 uint32_t minor;2727 uint32_t stepping;2728 char vendor_and_architecture_name[1];2729 };2730 2731 ``vendor_name_size`` and ``architecture_name_size`` are the length of the2732 vendor and architecture names respectively, including the NUL character.2733 2734 ``vendor_and_architecture_name`` contains the NUL terminated string for the2735 vendor, immediately followed by the NUL terminated string for the2736 architecture.2737 2738 This note record is used by the HSA runtime loader.2739 2740 Code object V2 only supports a limited number of processors and has fixed2741 settings for target features. See2742 :ref:`amdgpu-elf-note-record-supported_processors-v2-table` for a list of2743 processors and the corresponding target ID. In the table the note record ISA2744 name is a concatenation of the vendor name, architecture name, major, minor,2745 and stepping separated by a ":".2746 2747 The target ID column shows the processor name and fixed target features used2748 by the LLVM compiler. The LLVM compiler does not generate a2749 ``NT_AMD_HSA_HSAIL`` note record.2750 2751 A code object generated by the Finalizer also uses code object V2 and always2752 generates a ``NT_AMD_HSA_HSAIL`` note record. The processor name and2753 ``sramecc`` target feature is as shown in2754 :ref:`amdgpu-elf-note-record-supported_processors-v2-table` but the ``xnack``2755 target feature is specified by the ``EF_AMDGPU_FEATURE_XNACK_V2`` ``e_flags``2756 bit.2757 2758``NT_AMD_HSA_ISA_NAME``2759 Specifies the target ISA name as a non-NUL terminated string.2760 2761 This note record is not used by the HSA runtime loader.2762 2763 See the ``NT_AMD_HSA_ISA_VERSION`` note record description of the code object2764 V2's limited support of processors and fixed settings for target features.2765 2766 See :ref:`amdgpu-elf-note-record-supported_processors-v2-table` for a mapping2767 from the string to the corresponding target ID. If the ``xnack`` target2768 feature is supported and enabled, the string produced by the LLVM compiler2769 will may have a ``+xnack`` appended. The Finlizer did not do the appending and2770 instead used the ``EF_AMDGPU_FEATURE_XNACK_V2`` ``e_flags`` bit.2771 2772``NT_AMD_HSA_METADATA``2773 Specifies extensible metadata associated with the code objects executed on HSA2774 [HSA]_ compatible runtimes (see :ref:`amdgpu-os`). It is required when the2775 target triple OS is ``amdhsa`` (see :ref:`amdgpu-target-triples`). See2776 :ref:`amdgpu-amdhsa-code-object-metadata-v2` for the syntax of the code object2777 metadata string.2778 2779 .. table:: AMDGPU Code Object V2 Supported Processors and Fixed Target Feature Settings2780 :name: amdgpu-elf-note-record-supported_processors-v2-table2781 2782 ===================== ==========================2783 Note Record ISA Name Target ID2784 ===================== ==========================2785 ``AMD:AMDGPU:6:0:0`` ``gfx600``2786 ``AMD:AMDGPU:6:0:1`` ``gfx601``2787 ``AMD:AMDGPU:6:0:2`` ``gfx602``2788 ``AMD:AMDGPU:7:0:0`` ``gfx700``2789 ``AMD:AMDGPU:7:0:1`` ``gfx701``2790 ``AMD:AMDGPU:7:0:2`` ``gfx702``2791 ``AMD:AMDGPU:7:0:3`` ``gfx703``2792 ``AMD:AMDGPU:7:0:4`` ``gfx704``2793 ``AMD:AMDGPU:7:0:5`` ``gfx705``2794 ``AMD:AMDGPU:8:0:0`` ``gfx802``2795 ``AMD:AMDGPU:8:0:1`` ``gfx801:xnack+``2796 ``AMD:AMDGPU:8:0:2`` ``gfx802``2797 ``AMD:AMDGPU:8:0:3`` ``gfx803``2798 ``AMD:AMDGPU:8:0:4`` ``gfx803``2799 ``AMD:AMDGPU:8:0:5`` ``gfx805``2800 ``AMD:AMDGPU:8:1:0`` ``gfx810:xnack+``2801 ``AMD:AMDGPU:9:0:0`` ``gfx900:xnack-``2802 ``AMD:AMDGPU:9:0:1`` ``gfx900:xnack+``2803 ``AMD:AMDGPU:9:0:2`` ``gfx902:xnack-``2804 ``AMD:AMDGPU:9:0:3`` ``gfx902:xnack+``2805 ``AMD:AMDGPU:9:0:4`` ``gfx904:xnack-``2806 ``AMD:AMDGPU:9:0:5`` ``gfx904:xnack+``2807 ``AMD:AMDGPU:9:0:6`` ``gfx906:sramecc-:xnack-``2808 ``AMD:AMDGPU:9:0:7`` ``gfx906:sramecc-:xnack+``2809 ``AMD:AMDGPU:9:0:12`` ``gfx90c:xnack-``2810 ===================== ==========================2811 2812.. _amdgpu-note-records-v3-onwards:2813 2814Code Object V3 and Above Note Records2815~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~2816 2817The AMDGPU backend code object uses the following ELF note record in the2818``.note`` section when compiling for code object V3 and above.2819 2820The note record vendor field is "AMDGPU".2821 2822Additional note records may be present, but any which are not documented here2823are deprecated and should not be used.2824 2825 .. table:: AMDGPU Code Object V3 and Above ELF Note Records2826 :name: amdgpu-elf-note-records-table-v3-onwards2827 2828 ======== ============================== ======================================2829 Name Type Description2830 ======== ============================== ======================================2831 "AMDGPU" ``NT_AMDGPU_METADATA`` Metadata in Message Pack [MsgPack]_2832 binary format.2833 "AMDGPU" ``NT_AMDGPU_KFD_CORE_STATE`` Snapshot of runtime, agent and queues2834 state for use in core dump. See2835 :ref:`amdgpu_corefile_note`.2836 ======== ============================== ======================================2837 2838..2839 2840 .. table:: AMDGPU Code Object V3 and Above ELF Note Record Enumeration Values2841 :name: amdgpu-elf-note-record-enumeration-values-table-v3-onwards2842 2843 ============================== =====2844 Name Value2845 ============================== =====2846 *reserved* 0-312847 ``NT_AMDGPU_METADATA`` 322848 ``NT_AMDGPU_KFD_CORE_STATE`` 332849 ============================== =====2850 2851``NT_AMDGPU_METADATA``2852 Specifies extensible metadata associated with an AMDGPU code object. It is2853 encoded as a map in the Message Pack [MsgPack]_ binary data format. See2854 :ref:`amdgpu-amdhsa-code-object-metadata-v3`,2855 :ref:`amdgpu-amdhsa-code-object-metadata-v4` and2856 :ref:`amdgpu-amdhsa-code-object-metadata-v5` for the map keys defined for the2857 ``amdhsa`` OS.2858 2859.. _amdgpu-symbols:2860 2861Symbols2862-------2863 2864Symbols include the following:2865 2866 .. table:: AMDGPU ELF Symbols2867 :name: amdgpu-elf-symbols-table2868 2869 ===================== ================== ================ ==================2870 Name Type Section Description2871 ===================== ================== ================ ==================2872 *link-name* ``STT_OBJECT`` - ``.data`` Global variable2873 - ``.rodata``2874 - ``.bss``2875 *link-name*\ ``.kd`` ``STT_OBJECT`` - ``.rodata`` Kernel descriptor2876 *link-name* ``STT_FUNC`` - ``.text`` Kernel entry point2877 *link-name* ``STT_OBJECT`` - SHN_AMDGPU_LDS Global variable in LDS2878 ===================== ================== ================ ==================2879 2880Global variable2881 Global variables both used and defined by the compilation unit.2882 2883 If the symbol is defined in the compilation unit then it is allocated in the2884 appropriate section according to if it has initialized data or is readonly.2885 2886 If the symbol is external then its section is ``STN_UNDEF`` and the loader2887 will resolve relocations using the definition provided by another code object2888 or explicitly defined by the runtime.2889 2890 If the symbol resides in local/group memory (LDS) then its section is the2891 special processor specific section name ``SHN_AMDGPU_LDS``, and the2892 ``st_value`` field describes alignment requirements as it does for common2893 symbols.2894 2895 .. TODO::2896 2897 Add description of linked shared object symbols. Seems undefined symbols2898 are marked as STT_NOTYPE.2899 2900Kernel descriptor2901 Every HSA kernel has an associated kernel descriptor. It is the address of the2902 kernel descriptor that is used in the AQL dispatch packet used to invoke the2903 kernel, not the kernel entry point. The layout of the HSA kernel descriptor is2904 defined in :ref:`amdgpu-amdhsa-kernel-descriptor`.2905 2906Kernel entry point2907 Every HSA kernel also has a symbol for its machine code entry point.2908 2909.. _amdgpu-relocation-records:2910 2911Relocation Records2912------------------2913 2914The AMDGPU backend generates ``Elf64_Rela`` relocation records for2915AMDHSA or ``Elf64_Rel`` relocation records for Mesa/AMDPAL. Supported2916relocatable fields are:2917 2918``word32``2919 This specifies a 32-bit field occupying 4 bytes with arbitrary byte2920 alignment. These values use the same byte order as other word values in the2921 AMDGPU architecture.2922 2923``word64``2924 This specifies a 64-bit field occupying 8 bytes with arbitrary byte2925 alignment. These values use the same byte order as other word values in the2926 AMDGPU architecture.2927 2928Following notations are used for specifying relocation calculations:2929 2930**A**2931 Represents the addend used to compute the value of the relocatable field. If2932 the addend field is smaller than 64 bits then it is zero-extended to 64 bits2933 for use in the calculations below. (In practice this only affects ``_HI``2934 relocation types on Mesa/AMDPAL, where the addend comes from the 32-bit field2935 but the result of the calculation depends on the high part of the full 64-bit2936 address.)2937 2938**G**2939 Represents the offset into the global offset table at which the relocation2940 entry's symbol will reside during execution.2941 2942**GOT**2943 Represents the address of the global offset table.2944 2945**P**2946 Represents the place (section offset for ``et_rel`` or address for ``et_dyn``)2947 of the storage unit being relocated (computed using ``r_offset``).2948 2949**S**2950 Represents the value of the symbol whose index resides in the relocation2951 entry. Relocations not using this must specify a symbol index of2952 ``STN_UNDEF``.2953 2954**B**2955 Represents the base address of a loaded executable or shared object which is2956 the difference between the ELF address and the actual load address.2957 Relocations using this are only valid in executable or shared objects.2958 2959The following relocation types are supported:2960 2961 .. table:: AMDGPU ELF Relocation Records2962 :name: amdgpu-elf-relocation-records-table2963 2964 ========================== ======= ===== ========== ==============================2965 Relocation Type Kind Value Field Calculation2966 ========================== ======= ===== ========== ==============================2967 ``R_AMDGPU_NONE`` 0 *none* *none*2968 ``R_AMDGPU_ABS32_LO`` Static, 1 ``word32`` (S + A) & 0xFFFFFFFF2969 Dynamic2970 ``R_AMDGPU_ABS32_HI`` Static, 2 ``word32`` (S + A) >> 322971 Dynamic2972 ``R_AMDGPU_ABS64`` Static, 3 ``word64`` S + A2973 Dynamic2974 ``R_AMDGPU_REL32`` Static 4 ``word32`` S + A - P2975 ``R_AMDGPU_REL64`` Static 5 ``word64`` S + A - P2976 ``R_AMDGPU_ABS32`` Static, 6 ``word32`` S + A2977 Dynamic2978 ``R_AMDGPU_GOTPCREL`` Static 7 ``word32`` G + GOT + A - P2979 ``R_AMDGPU_GOTPCREL32_LO`` Static 8 ``word32`` (G + GOT + A - P) & 0xFFFFFFFF2980 ``R_AMDGPU_GOTPCREL32_HI`` Static 9 ``word32`` (G + GOT + A - P) >> 322981 ``R_AMDGPU_REL32_LO`` Static 10 ``word32`` (S + A - P) & 0xFFFFFFFF2982 ``R_AMDGPU_REL32_HI`` Static 11 ``word32`` (S + A - P) >> 322983 *reserved* 122984 ``R_AMDGPU_RELATIVE64`` Dynamic 13 ``word64`` B + A2985 ``R_AMDGPU_REL16`` Static 14 ``word16`` ((S + A - P) - 4) / 42986 ========================== ======= ===== ========== ==============================2987 2988``R_AMDGPU_ABS32_LO`` and ``R_AMDGPU_ABS32_HI`` are only supported by2989the ``mesa3d`` OS, which does not support ``R_AMDGPU_ABS64``.2990 2991There is no current OS loader support for 32-bit programs and so2992``R_AMDGPU_ABS32`` is only generated for static relocations, for example to2993implement some DWARF32 forms.2994 2995.. _amdgpu-loaded-code-object-path-uniform-resource-identifier:2996 2997Loaded Code Object Path Uniform Resource Identifier (URI)2998---------------------------------------------------------2999 3000The AMD GPU code object loader represents the path of the ELF shared object from3001which the code object was loaded as a textual Uniform Resource Identifier (URI).3002Note that the code object is the in memory loaded relocated form of the ELF3003shared object. Multiple code objects may be loaded at different memory3004addresses in the same process from the same ELF shared object.3005 3006The loaded code object path URI syntax is defined by the following BNF syntax:3007 3008.. code::3009 3010 code_object_uri ::== file_uri | memory_uri3011 file_uri ::== "file://" file_path [ range_specifier ]3012 memory_uri ::== "memory://" process_id range_specifier3013 range_specifier ::== [ "#" | "?" ] "offset=" number "&" "size=" number3014 file_path ::== URI_ENCODED_OS_FILE_PATH3015 process_id ::== DECIMAL_NUMBER3016 number ::== HEX_NUMBER | DECIMAL_NUMBER | OCTAL_NUMBER3017 3018**number**3019 Is a C integral literal where hexadecimal values are prefixed by "0x" or "0X",3020 and octal values by "0".3021 3022**file_path**3023 Is the file's path specified as a URI encoded UTF-8 string. In URI encoding,3024 every character that is not in the regular expression ``[a-zA-Z0-9/_.~-]`` is3025 encoded as two uppercase hexadecimal digits proceeded by "%". Directories in3026 the path are separated by "/".3027 3028**offset**3029 Is a 0-based byte offset to the start of the code object. For a file URI, it3030 is from the start of the file specified by the ``file_path``, and if omitted3031 defaults to 0. For a memory URI, it is the memory address and is required.3032 3033**size**3034 Is the number of bytes in the code object. For a file URI, if omitted it3035 defaults to the size of the file. It is required for a memory URI.3036 3037**process_id**3038 Is the identity of the process owning the memory. For Linux it is the C3039 unsigned integral decimal literal for the process ID (PID).3040 3041For example:3042 3043.. code::3044 3045 file:///dir1/dir2/file13046 file:///dir3/dir4/file2#offset=0x2000&size=30003047 memory://1234#offset=0x20000&size=30003048 3049.. _amdgpu-dwarf-debug-information:3050 3051DWARF Debug Information3052=======================3053 3054.. warning::3055 3056 This section describes **provisional support** for AMDGPU DWARF [DWARF]_ that3057 is not currently fully implemented and is subject to change.3058 3059AMDGPU generates DWARF [DWARF]_ debugging information ELF sections (see3060:ref:`amdgpu-elf-code-object`) which contain information that maps the code3061object executable code and data to the source language constructs. It can be3062used by tools such as debuggers and profilers. It uses features defined in3063:doc:`AMDGPUDwarfExtensionsForHeterogeneousDebugging` that are made available in3064DWARF Version 4 and DWARF Version 5 as an LLVM vendor extension.3065 3066This section defines the AMDGPU target architecture specific DWARF mappings.3067 3068.. _amdgpu-dwarf-register-identifier:3069 3070Register Identifier3071-------------------3072 3073This section defines the AMDGPU target architecture register numbers used in3074DWARF operation expressions (see DWARF Version 5 section 2.5 and3075:ref:`amdgpu-dwarf-operation-expressions`) and Call Frame Information3076instructions (see DWARF Version 5 section 6.4 and3077:ref:`amdgpu-dwarf-call-frame-information`).3078 3079A single code object can contain code for kernels that have different wavefront3080sizes. The vector registers and some scalar registers are based on the wavefront3081size. AMDGPU defines distinct DWARF registers for each wavefront size. This3082simplifies the consumer of the DWARF so that each register has a fixed size,3083rather than being dynamic according to the wavefront size mode. Similarly,3084distinct DWARF registers are defined for those registers that vary in size3085according to the process address size. This allows a consumer to treat a3086specific AMDGPU processor as a single architecture regardless of how it is3087configured at run time. The compiler explicitly specifies the DWARF registers3088that match the mode in which the code it is generating will be executed.3089 3090DWARF registers are encoded as numbers, which are mapped to architecture3091registers. The mapping for AMDGPU is defined in3092:ref:`amdgpu-dwarf-register-mapping-table`. All AMDGPU targets use the same3093mapping.3094 3095.. table:: AMDGPU DWARF Register Mapping3096 :name: amdgpu-dwarf-register-mapping-table3097 3098 ============== ================= ======== ==================================3099 DWARF Register AMDGPU Register Bit Size Description3100 ============== ================= ======== ==================================3101 0 PC_32 32 Program Counter (PC) when3102 executing in a 32-bit process3103 address space. Used in the CFI to3104 describe the PC of the calling3105 frame.3106 1 EXEC_MASK_32 32 Execution Mask Register when3107 executing in wavefront 32 mode.3108 2-15 *Reserved* *Reserved for highly accessed3109 registers using DWARF shortcut.*3110 16 PC_64 64 Program Counter (PC) when3111 executing in a 64-bit process3112 address space. Used in the CFI to3113 describe the PC of the calling3114 frame.3115 17 EXEC_MASK_64 64 Execution Mask Register when3116 executing in wavefront 64 mode.3117 18-31 *Reserved* *Reserved for highly accessed3118 registers using DWARF shortcut.*3119 32-95 SGPR0-SGPR63 32 Scalar General Purpose3120 Registers.3121 96-127 *Reserved* *Reserved for frequently accessed3122 registers using DWARF 1-byte ULEB.*3123 128 STATUS 32 Status Register.3124 129-511 *Reserved* *Reserved for future Scalar3125 Architectural Registers.*3126 512 VCC_32 32 Vector Condition Code Register3127 when executing in wavefront 323128 mode.3129 513-767 *Reserved* *Reserved for future Vector3130 Architectural Registers when3131 executing in wavefront 32 mode.*3132 768 VCC_64 64 Vector Condition Code Register3133 when executing in wavefront 643134 mode.3135 769-1023 *Reserved* *Reserved for future Vector3136 Architectural Registers when3137 executing in wavefront 64 mode.*3138 1024-1087 *Reserved* *Reserved for padding.*3139 1088-1129 SGPR64-SGPR105 32 Scalar General Purpose Registers.3140 1130-1535 *Reserved* *Reserved for future Scalar3141 General Purpose Registers.*3142 1536-2047 VGPR0-VGPR511 32*32 Vector General Purpose Registers3143 when executing in wavefront 323144 mode.3145 2048-2303 AGPR0-AGPR255 32*32 Vector Accumulation Registers3146 when executing in wavefront 323147 mode.3148 2304-2559 *Reserved* *Reserved for future Vector3149 Accumulation Registers when3150 executing in wavefront 32 mode.*3151 2560-2815 VGPR0-VGPR255 64*32 Vector General Purpose Registers3152 when executing in wavefront 643153 mode.3154 2816-3071 *Reserved* *Reserved for future Vector3155 General Purpose Registers when3156 executing in wavefront 64 mode.*3157 3072-3327 AGPR0-AGPR255 64*32 Vector Accumulation Registers3158 when executing in wavefront 643159 mode.3160 3328-3583 *Reserved* *Reserved for future Vector3161 Accumulation Registers when3162 executing in wavefront 64 mode.*3163 3584-4095 VGPR512-VGPR1023 32*32 Second Block of Vector General3164 Purpose Registers When executing3165 in wavefront 32 mode3166 ============== ================= ======== ==================================3167 3168The vector registers are represented as the full size for the wavefront. They3169are organized as consecutive dwords (32-bits), one per lane, with the dword at3170the least significant bit position corresponding to lane 0 and so forth. DWARF3171location expressions involving the ``DW_OP_LLVM_offset`` and3172``DW_OP_LLVM_push_lane`` operations are used to select the part of the vector3173register corresponding to the lane that is executing the current thread of3174execution in languages that are implemented using a SIMD or SIMT execution3175model.3176 3177If the wavefront size is 32 lanes then the wavefront 32 mode register3178definitions are used. If the wavefront size is 64 lanes then the wavefront 643179mode register definitions are used. Some AMDGPU targets support executing in3180both wavefront 32 and wavefront 64 mode. The register definitions corresponding3181to the wavefront mode of the generated code will be used.3182 3183If code is generated to execute in a 32-bit process address space, then the318432-bit process address space register definitions are used. If code is generated3185to execute in a 64-bit process address space, then the 64-bit process address3186space register definitions are used. The ``amdgcn`` target only supports the318764-bit process address space.3188 3189.. _amdgpu-dwarf-memory-space-identifier:3190 3191Memory Space Identifier3192-----------------------3193 3194The DWARF memory space represents the source language memory space. See DWARF3195Version 5 section 2.12 which is updated by the *DWARF Extensions For3196Heterogeneous Debugging* section :ref:`amdgpu-dwarf-memory-spaces`.3197 3198The DWARF memory space mapping used for AMDGPU is defined in3199:ref:`amdgpu-dwarf-memory-space-mapping-table`.3200 3201.. table:: AMDGPU DWARF Memory Space Mapping3202 :name: amdgpu-dwarf-memory-space-mapping-table3203 3204 =========================== ====== =================3205 DWARF AMDGPU3206 ---------------------------------- -----------------3207 Memory Space Name Value Memory Space3208 =========================== ====== =================3209 ``DW_MSPACE_LLVM_none`` 0x0000 Generic (Flat)3210 ``DW_MSPACE_LLVM_global`` 0x0001 Global3211 ``DW_MSPACE_LLVM_constant`` 0x0002 Global3212 ``DW_MSPACE_LLVM_group`` 0x0003 Local (group/LDS)3213 ``DW_MSPACE_LLVM_private`` 0x0004 Private (Scratch)3214 ``DW_MSPACE_AMDGPU_region`` 0x8000 Region (GDS)3215 =========================== ====== =================3216 3217The DWARF memory space values defined in the *DWARF Extensions For Heterogeneous3218Debugging* section :ref:`amdgpu-dwarf-memory-spaces` are used.3219 3220In addition, ``DW_ADDR_AMDGPU_region`` is encoded as a vendor extension. This is3221available for use for the AMD extension for access to the hardware GDS memory3222which is scratchpad memory allocated per device.3223 3224For AMDGPU if no ``DW_AT_LLVM_memory_space`` attribute is present, then the3225default memory space of ``DW_MSPACE_LLVM_none`` is used.3226 3227See :ref:`amdgpu-dwarf-address-space-identifier` for information on the AMDGPU3228mapping of DWARF memory spaces to DWARF address spaces, including address size3229and NULL value.3230 3231.. _amdgpu-dwarf-address-space-identifier:3232 3233Address Space Identifier3234------------------------3235 3236DWARF address spaces correspond to target architecture specific linear3237addressable memory areas. See DWARF Version 5 section 2.12 and *DWARF Extensions3238For Heterogeneous Debugging* section :ref:`amdgpu-dwarf-address-spaces`.3239 3240The DWARF address space mapping used for AMDGPU is defined in3241:ref:`amdgpu-dwarf-address-space-mapping-table`.3242 3243.. table:: AMDGPU DWARF Address Space Mapping3244 :name: amdgpu-dwarf-address-space-mapping-table3245 3246 ======================================= ===== ======= ======== ===================== =======================3247 DWARF AMDGPU Notes3248 --------------------------------------- ----- ---------------- --------------------- -----------------------3249 Address Space Name Value Address Bit Size LLVM IR Address Space3250 --------------------------------------- ----- ------- -------- --------------------- -----------------------3251 .. 64-bit 32-bit3252 process process3253 address address3254 space space3255 ======================================= ===== ======= ======== ===================== =======================3256 ``DW_ASPACE_LLVM_none`` 0x00 64 32 Global *default address space*3257 ``DW_ASPACE_AMDGPU_generic`` 0x01 64 32 Generic (Flat)3258 ``DW_ASPACE_AMDGPU_region`` 0x02 32 32 Region (GDS)3259 ``DW_ASPACE_AMDGPU_local`` 0x03 32 32 Local (group/LDS)3260 *Reserved* 0x043261 ``DW_ASPACE_AMDGPU_private_lane`` 0x05 32 32 Private (Scratch) *focused lane*3262 ``DW_ASPACE_AMDGPU_private_wave`` 0x06 32 32 Private (Scratch) *unswizzled wavefront*3263 ======================================= ===== ======= ======== ===================== =======================3264 3265See :ref:`amdgpu-address-spaces` for information on the AMDGPU LLVM IR address3266spaces including address size and NULL value.3267 3268The ``DW_ASPACE_LLVM_none`` address space is the default target architecture3269address space used in DWARF operations that do not specify an address space. It3270therefore has to map to the global address space so that the ``DW_OP_addr*`` and3271related operations can refer to addresses in the program code.3272 3273The ``DW_ASPACE_AMDGPU_generic`` address space allows location expressions to3274specify the flat address space. If the address corresponds to an address in the3275local address space, then it corresponds to the wavefront that is executing the3276focused thread of execution. If the address corresponds to an address in the3277private address space, then it corresponds to the lane that is executing the3278focused thread of execution for languages that are implemented using a SIMD or3279SIMT execution model.3280 3281.. note::3282 3283 CUDA-like languages such as HIP that do not have address spaces in the3284 language type system, but do allow variables to be allocated in different3285 address spaces, need to explicitly specify the ``DW_ASPACE_AMDGPU_generic``3286 address space in the DWARF expression operations as the default address space3287 is the global address space.3288 3289The ``DW_ASPACE_AMDGPU_local`` address space allows location expressions to3290specify the local address space corresponding to the wavefront that is executing3291the focused thread of execution.3292 3293The ``DW_ASPACE_AMDGPU_private_lane`` address space allows location expressions3294to specify the private address space corresponding to the lane that is executing3295the focused thread of execution for languages that are implemented using a SIMD3296or SIMT execution model.3297 3298The ``DW_ASPACE_AMDGPU_private_wave`` address space allows location expressions3299to specify the unswizzled private address space corresponding to the wavefront3300that is executing the focused thread of execution. The wavefront view of private3301memory is the per wavefront unswizzled backing memory layout defined in3302:ref:`amdgpu-address-spaces`, such that address 0 corresponds to the first3303location for the backing memory of the wavefront (namely the address is not3304offset by ``wavefront-scratch-base``). The following formula can be used to3305convert from a ``DW_ASPACE_AMDGPU_private_lane`` address to a3306``DW_ASPACE_AMDGPU_private_wave`` address:3307 3308::3309 3310 private-address-wavefront =3311 ((private-address-lane / 4) * wavefront-size * 4) +3312 (wavefront-lane-id * 4) + (private-address-lane % 4)3313 3314If the ``DW_ASPACE_AMDGPU_private_lane`` address is dword aligned, and the start3315of the dwords for each lane starting with lane 0 is required, then this3316simplifies to:3317 3318::3319 3320 private-address-wavefront =3321 private-address-lane * wavefront-size3322 3323A compiler can use the ``DW_ASPACE_AMDGPU_private_wave`` address space to read a3324complete spilled vector register back into a complete vector register in the3325CFI. The frame pointer can be a private lane address which is dword aligned,3326which can be shifted to multiply by the wavefront size, and then used to form a3327private wavefront address that gives a location for a contiguous set of dwords,3328one per lane, where the vector register dwords are spilled. The compiler knows3329the wavefront size since it generates the code. Note that the type of the3330address may have to be converted as the size of a3331``DW_ASPACE_AMDGPU_private_lane`` address may be smaller than the size of a3332``DW_ASPACE_AMDGPU_private_wave`` address.3333 3334.. _amdgpu-dwarf-lane-identifier:3335 3336Lane identifier3337---------------3338 3339DWARF lane identifies specify a target architecture lane position for hardware3340that executes in a SIMD or SIMT manner, and on which a source language maps its3341threads of execution onto those lanes. The DWARF lane identifier is pushed by3342the ``DW_OP_LLVM_push_lane`` DWARF expression operation. See DWARF Version 53343section 2.5 which is updated by *DWARF Extensions For Heterogeneous Debugging*3344section :ref:`amdgpu-dwarf-operation-expressions`.3345 3346For AMDGPU, the lane identifier corresponds to the hardware lane ID of a3347wavefront. It is numbered from 0 to the wavefront size minus 1.3348 3349Operation Expressions3350---------------------3351 3352DWARF expressions are used to compute program values and the locations of3353program objects. See DWARF Version 5 section 2.5 and3354:ref:`amdgpu-dwarf-operation-expressions`.3355 3356DWARF location descriptions describe how to access storage which includes memory3357and registers. When accessing storage on AMDGPU, bytes are ordered with least3358significant bytes first, and bits are ordered within bytes with least3359significant bits first.3360 3361For AMDGPU CFI expressions, ``DW_OP_LLVM_select_bit_piece`` is used to describe3362unwinding vector registers that are spilled under the execution mask to memory:3363the zero-single location description is the vector register, and the one-single3364location description is the spilled memory location description. The3365``DW_OP_LLVM_form_aspace_address`` is used to specify the address space of the3366memory location description.3367 3368In AMDGPU expressions, ``DW_OP_LLVM_select_bit_piece`` is used by the3369``DW_AT_LLVM_lane_pc`` attribute expression where divergent control flow is3370controlled by the execution mask. An undefined location description together3371with ``DW_OP_LLVM_extend`` is used to indicate the lane was not active on entry3372to the subprogram. See :ref:`amdgpu-dwarf-dw-at-llvm-lane-pc` for an example.3373 3374.. _amdgpu-dwarf-base-type-conversions:3375 3376Base Type Conversions3377---------------------3378 3379For AMDGPU expressions, ``DW_OP_convert`` may be used to convert between3380``DW_ATE_address``-encoded base types in different address spaces.3381 3382Conversions are defined as in :ref:`amdgpu-address-spaces` when all relevant3383conditions described there are met, and otherwise result in an evaluation3384error.3385 3386.. note::3387 3388 For a target which does not support a particular address space, converting to3389 or from that address space is always an evaluation error.3390 3391 For targets which support the generic address space, converting from3392 ``DW_ASPACE_AMDGPU_generic`` to ``DW_ASPACE_LLVM_none`` is defined when the3393 generic address is in the global address space. The conversion requires no3394 change to the literal value of the address.3395 3396 Converting from ``DW_ASPACE_AMDGPU_generic`` to any of3397 ``DW_ASPACE_AMDGPU_local``, ``DW_ASPACE_AMDGPU_private_wave`` or3398 ``DW_ASPACE_AMDGPU_private_lane`` is defined when the relevant hardware3399 support is present, any required hardware setup has been completed, and the3400 generic address is in the corresponding address space. Conversion to3401 ``DW_ASPACE_AMDGPU_private_lane`` additionally requires the context to3402 include the active lane.3403 3404Debugger Information Entry Attributes3405-------------------------------------3406 3407This section describes how certain debugger information entry attributes are3408used by AMDGPU. See the sections in DWARF Version 5 section 3.3.5 and 3.1.13409which are updated by *DWARF Extensions For Heterogeneous Debugging* section3410:ref:`amdgpu-dwarf-low-level-information` and3411:ref:`amdgpu-dwarf-full-and-partial-compilation-unit-entries`.3412 3413.. _amdgpu-dwarf-dw-at-llvm-lane-pc:3414 3415``DW_AT_LLVM_lane_pc``3416~~~~~~~~~~~~~~~~~~~~~~3417 3418For AMDGPU, the ``DW_AT_LLVM_lane_pc`` attribute is used to specify the program3419location of the separate lanes of a SIMT thread.3420 3421If the lane is an active lane then this will be the same as the current program3422location.3423 3424If the lane is inactive, but was active on entry to the subprogram, then this is3425the program location in the subprogram at which execution of the lane is3426conceptually positioned.3427 3428If the lane was not active on entry to the subprogram, then this will be the3429undefined location. A client debugger can check if the lane is part of a valid3430work-group by checking that the lane is in the range of the associated3431work-group within the grid, accounting for partial work-groups. If it is not,3432then the debugger can omit any information for the lane. Otherwise, the debugger3433may repeatedly unwind the stack and inspect the ``DW_AT_LLVM_lane_pc`` of the3434calling subprogram until it finds a non-undefined location. Conceptually the3435lane only has the call frames that it has a non-undefined3436``DW_AT_LLVM_lane_pc``.3437 3438The following example illustrates how the AMDGPU backend can generate a DWARF3439location list expression for the nested ``IF/THEN/ELSE`` structures of the3440following subprogram pseudo code for a target with 64 lanes per wavefront.3441 3442.. code::3443 :number-lines:3444 3445 SUBPROGRAM X3446 BEGIN3447 a;3448 IF (c1) THEN3449 b;3450 IF (c2) THEN3451 c;3452 ELSE3453 d;3454 ENDIF3455 e;3456 ELSE3457 f;3458 ENDIF3459 g;3460 END3461 3462The AMDGPU backend may generate the following pseudo LLVM MIR to manipulate the3463execution mask (``EXEC``) to linearize the control flow. The condition is3464evaluated to make a mask of the lanes for which the condition evaluates to true.3465First the ``THEN`` region is executed by setting the ``EXEC`` mask to the3466logical ``AND`` of the current ``EXEC`` mask with the condition mask. Then the3467``ELSE`` region is executed by negating the ``EXEC`` mask and logical ``AND`` of3468the saved ``EXEC`` mask at the start of the region. After the ``IF/THEN/ELSE``3469region the ``EXEC`` mask is restored to the value it had at the beginning of the3470region. This is shown below. Other approaches are possible, but the basic3471concept is the same.3472 3473.. code::3474 :number-lines:3475 3476 $lex_start:3477 a;3478 %1 = EXEC3479 %2 = c13480 $lex_1_start:3481 EXEC = %1 & %23482 $if_1_then:3483 b;3484 %3 = EXEC3485 %4 = c23486 $lex_1_1_start:3487 EXEC = %3 & %43488 $lex_1_1_then:3489 c;3490 EXEC = ~EXEC & %33491 $lex_1_1_else:3492 d;3493 EXEC = %33494 $lex_1_1_end:3495 e;3496 EXEC = ~EXEC & %13497 $lex_1_else:3498 f;3499 EXEC = %13500 $lex_1_end:3501 g;3502 $lex_end:3503 3504To create the DWARF location list expression that defines the location3505description of a vector of lane program locations, the LLVM MIR ``DBG_VALUE``3506pseudo instruction can be used to annotate the linearized control flow. This can3507be done by defining an artificial variable for the lane PC. The DWARF location3508list expression created for it is used as the value of the3509``DW_AT_LLVM_lane_pc`` attribute on the subprogram's debugger information entry.3510 3511A DWARF procedure is defined for each well nested structured control flow region3512which provides the conceptual lane program location for a lane if it is not3513active (namely it is divergent). The DWARF operation expression for each region3514conceptually inherits the value of the immediately enclosing region and modifies3515it according to the semantics of the region.3516 3517For an ``IF/THEN/ELSE`` region the divergent program location is at the start of3518the region for the ``THEN`` region since it is executed first. For the ``ELSE``3519region the divergent program location is at the end of the ``IF/THEN/ELSE``3520region since the ``THEN`` region has completed.3521 3522The lane PC artificial variable is assigned at each region transition. It uses3523the immediately enclosing region's DWARF procedure to compute the program3524location for each lane assuming they are divergent, and then modifies the result3525by inserting the current program location for each lane that the ``EXEC`` mask3526indicates is active.3527 3528By having separate DWARF procedures for each region, they can be reused to3529define the value for any nested region. This reduces the total size of the DWARF3530operation expressions.3531 3532The following provides an example using pseudo LLVM MIR.3533 3534.. code::3535 :number-lines:3536 3537 $lex_start:3538 DEFINE_DWARF %__uint_64 = DW_TAG_base_type[3539 DW_AT_name = "__uint64";3540 DW_AT_byte_size = 8;3541 DW_AT_encoding = DW_ATE_unsigned;3542 ];3543 DEFINE_DWARF %__active_lane_pc = DW_TAG_dwarf_procedure[3544 DW_AT_name = "__active_lane_pc";3545 DW_AT_location = [3546 DW_OP_regx PC;3547 DW_OP_LLVM_extend 64, 64;3548 DW_OP_regval_type EXEC, %uint_64;3549 DW_OP_LLVM_select_bit_piece 64, 64;3550 ];3551 ];3552 DEFINE_DWARF %__divergent_lane_pc = DW_TAG_dwarf_procedure[3553 DW_AT_name = "__divergent_lane_pc";3554 DW_AT_location = [3555 DW_OP_LLVM_undefined;3556 DW_OP_LLVM_extend 64, 64;3557 ];3558 ];3559 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[3560 DW_OP_call_ref %__divergent_lane_pc;3561 DW_OP_call_ref %__active_lane_pc;3562 ];3563 a;3564 %1 = EXEC;3565 DBG_VALUE %1, $noreg, %__lex_1_save_exec;3566 %2 = c1;3567 $lex_1_start:3568 EXEC = %1 & %2;3569 $lex_1_then:3570 DEFINE_DWARF %__divergent_lane_pc_1_then = DW_TAG_dwarf_procedure[3571 DW_AT_name = "__divergent_lane_pc_1_then";3572 DW_AT_location = DIExpression[3573 DW_OP_call_ref %__divergent_lane_pc;3574 DW_OP_addrx &lex_1_start;3575 DW_OP_stack_value;3576 DW_OP_LLVM_extend 64, 64;3577 DW_OP_call_ref %__lex_1_save_exec;3578 DW_OP_deref_type 64, %__uint_64;3579 DW_OP_LLVM_select_bit_piece 64, 64;3580 ];3581 ];3582 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[3583 DW_OP_call_ref %__divergent_lane_pc_1_then;3584 DW_OP_call_ref %__active_lane_pc;3585 ];3586 b;3587 %3 = EXEC;3588 DBG_VALUE %3, %__lex_1_1_save_exec;3589 %4 = c2;3590 $lex_1_1_start:3591 EXEC = %3 & %4;3592 $lex_1_1_then:3593 DEFINE_DWARF %__divergent_lane_pc_1_1_then = DW_TAG_dwarf_procedure[3594 DW_AT_name = "__divergent_lane_pc_1_1_then";3595 DW_AT_location = DIExpression[3596 DW_OP_call_ref %__divergent_lane_pc_1_then;3597 DW_OP_addrx &lex_1_1_start;3598 DW_OP_stack_value;3599 DW_OP_LLVM_extend 64, 64;3600 DW_OP_call_ref %__lex_1_1_save_exec;3601 DW_OP_deref_type 64, %__uint_64;3602 DW_OP_LLVM_select_bit_piece 64, 64;3603 ];3604 ];3605 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[3606 DW_OP_call_ref %__divergent_lane_pc_1_1_then;3607 DW_OP_call_ref %__active_lane_pc;3608 ];3609 c;3610 EXEC = ~EXEC & %3;3611 $lex_1_1_else:3612 DEFINE_DWARF %__divergent_lane_pc_1_1_else = DW_TAG_dwarf_procedure[3613 DW_AT_name = "__divergent_lane_pc_1_1_else";3614 DW_AT_location = DIExpression[3615 DW_OP_call_ref %__divergent_lane_pc_1_then;3616 DW_OP_addrx &lex_1_1_end;3617 DW_OP_stack_value;3618 DW_OP_LLVM_extend 64, 64;3619 DW_OP_call_ref %__lex_1_1_save_exec;3620 DW_OP_deref_type 64, %__uint_64;3621 DW_OP_LLVM_select_bit_piece 64, 64;3622 ];3623 ];3624 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[3625 DW_OP_call_ref %__divergent_lane_pc_1_1_else;3626 DW_OP_call_ref %__active_lane_pc;3627 ];3628 d;3629 EXEC = %3;3630 $lex_1_1_end:3631 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[3632 DW_OP_call_ref %__divergent_lane_pc;3633 DW_OP_call_ref %__active_lane_pc;3634 ];3635 e;3636 EXEC = ~EXEC & %1;3637 $lex_1_else:3638 DEFINE_DWARF %__divergent_lane_pc_1_else = DW_TAG_dwarf_procedure[3639 DW_AT_name = "__divergent_lane_pc_1_else";3640 DW_AT_location = DIExpression[3641 DW_OP_call_ref %__divergent_lane_pc;3642 DW_OP_addrx &lex_1_end;3643 DW_OP_stack_value;3644 DW_OP_LLVM_extend 64, 64;3645 DW_OP_call_ref %__lex_1_save_exec;3646 DW_OP_deref_type 64, %__uint_64;3647 DW_OP_LLVM_select_bit_piece 64, 64;3648 ];3649 ];3650 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc, DIExpression[3651 DW_OP_call_ref %__divergent_lane_pc_1_else;3652 DW_OP_call_ref %__active_lane_pc;3653 ];3654 f;3655 EXEC = %1;3656 $lex_1_end:3657 DBG_VALUE $noreg, $noreg, %DW_AT_LLVM_lane_pc DIExpression[3658 DW_OP_call_ref %__divergent_lane_pc;3659 DW_OP_call_ref %__active_lane_pc;3660 ];3661 g;3662 $lex_end:3663 3664The DWARF procedure ``%__active_lane_pc`` is used to update the lane pc elements3665that are active, with the current program location.3666 3667Artificial variables %__lex_1_save_exec and %__lex_1_1_save_exec are created for3668the execution masks saved on entry to a region. Using the ``DBG_VALUE`` pseudo3669instruction, location list entries will be created that describe where the3670artificial variables are allocated at any given program location. The compiler3671may allocate them to registers or spill them to memory.3672 3673The DWARF procedures for each region use the values of the saved execution mask3674artificial variables to only update the lanes that are active on entry to the3675region. All other lanes retain the value of the enclosing region where they were3676last active. If they were not active on entry to the subprogram, then will have3677the undefined location description.3678 3679Other structured control flow regions can be handled similarly. For example,3680loops would set the divergent program location for the region at the end of the3681loop. Any lanes active will be in the loop, and any lanes not active must have3682exited the loop.3683 3684An ``IF/THEN/ELSEIF/ELSEIF/...`` region can be treated as a nest of3685``IF/THEN/ELSE`` regions.3686 3687The DWARF procedures can use the active lane artificial variable described in3688:ref:`amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane` rather than the actual3689``EXEC`` mask in order to support whole or quad wavefront mode.3690 3691.. _amdgpu-dwarf-amdgpu-dw-at-llvm-active-lane:3692 3693``DW_AT_LLVM_active_lane``3694~~~~~~~~~~~~~~~~~~~~~~~~~~3695 3696The ``DW_AT_LLVM_active_lane`` attribute on a subprogram debugger information3697entry is used to specify the lanes that are conceptually active for a SIMT3698thread.3699 3700The execution mask may be modified to implement whole or quad wavefront mode3701operations. For example, all lanes may need to temporarily be made active to3702execute a whole wavefront operation. Such regions would save the ``EXEC`` mask,3703update it to enable the necessary lanes, perform the operations, and then3704restore the ``EXEC`` mask from the saved value. While executing the whole3705wavefront region, the conceptual execution mask is the saved value, not the3706``EXEC`` value.3707 3708This is handled by defining an artificial variable for the active lane mask. The3709active lane mask artificial variable would be the actual ``EXEC`` mask for3710normal regions, and the saved execution mask for regions where the mask is3711temporarily updated. The location list expression created for this artificial3712variable is used to define the value of the ``DW_AT_LLVM_active_lane``3713attribute.3714 3715``DW_AT_LLVM_augmentation``3716~~~~~~~~~~~~~~~~~~~~~~~~~~~3717 3718For AMDGPU, the ``DW_AT_LLVM_augmentation`` attribute of a compilation unit3719debugger information entry has the following value for the augmentation string:3720 3721::3722 3723 [amdgpu:v0.0]3724 3725The "vX.Y" specifies the major X and minor Y version number of the AMDGPU3726extensions used in the DWARF of the compilation unit. The version number3727conforms to [SEMVER]_.3728 3729Call Frame Information3730----------------------3731 3732DWARF Call Frame Information (CFI) describes how a consumer can virtually3733*unwind* call frames in a running process or core dump. See DWARF Version 53734section 6.4 and :ref:`amdgpu-dwarf-call-frame-information`.3735 3736For AMDGPU, the Common Information Entry (CIE) fields have the following values:3737 37381. ``augmentation`` string contains the following null-terminated UTF-8 string:3739 3740 ::3741 3742 [amd:v0.0]3743 3744 The ``vX.Y`` specifies the major X and minor Y version number of the AMDGPU3745 extensions used in this CIE or to the FDEs that use it. The version number3746 conforms to [SEMVER]_.3747 37482. ``address_size`` for the ``Global`` address space is defined in3749 :ref:`amdgpu-dwarf-address-space-identifier`.3750 37513. ``segment_selector_size`` is 0 as AMDGPU does not use a segment selector.3752 37534. ``code_alignment_factor`` is 4 bytes.3754 3755 .. TODO::3756 3757 Add to :ref:`amdgpu-processor-table` table.3758 37595. ``data_alignment_factor`` is 4 bytes.3760 3761 .. TODO::3762 3763 Add to :ref:`amdgpu-processor-table` table.3764 37656. ``return_address_register`` is ``PC_32`` for 32-bit processes and ``PC_64``3766 for 64-bit processes defined in :ref:`amdgpu-dwarf-register-identifier`.3767 37687. ``initial_instructions`` Since a subprogram X with fewer registers can be3769 called from subprogram Y that has more allocated, X will not change any of3770 the extra registers as it cannot access them. Therefore, the default rule3771 for all columns is ``same value``.3772 3773For AMDGPU the register number follows the numbering defined in3774:ref:`amdgpu-dwarf-register-identifier`.3775 3776For AMDGPU the instructions are variable size. A consumer can subtract 1 from3777the return address to get the address of a byte within the call site3778instructions. See DWARF Version 5 section 6.4.4.3779 3780Accelerated Access3781------------------3782 3783See DWARF Version 5 section 6.1.3784 3785Lookup By Name Section Header3786~~~~~~~~~~~~~~~~~~~~~~~~~~~~~3787 3788See DWARF Version 5 section 6.1.1.4.1 and :ref:`amdgpu-dwarf-lookup-by-name`.3789 3790For AMDGPU the lookup by name section header table:3791 3792``augmentation_string_size`` (uword)3793 3794 Set to the length of the ``augmentation_string`` value which is always a3795 multiple of 4.3796 3797``augmentation_string`` (sequence of UTF-8 characters)3798 3799 Contains the following UTF-8 string null padded to a multiple of 4 bytes:3800 3801 ::3802 3803 [amdgpu:v0.0]3804 3805 The "vX.Y" specifies the major X and minor Y version number of the AMDGPU3806 extensions used in the DWARF of this index. The version number conforms to3807 [SEMVER]_.3808 3809 .. note::3810 3811 This is different to the DWARF Version 5 definition that requires the first3812 4 characters to be the vendor ID. But this is consistent with the other3813 augmentation strings and does allow multiple vendor contributions. However,3814 backwards compatibility may be more desirable.3815 3816Lookup By Address Section Header3817~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~3818 3819See DWARF Version 5 section 6.1.2.3820 3821For AMDGPU the lookup by address section header table:3822 3823``address_size`` (ubyte)3824 3825 Match the address size for the ``Global`` address space defined in3826 :ref:`amdgpu-dwarf-address-space-identifier`.3827 3828``segment_selector_size`` (ubyte)3829 3830 AMDGPU does not use a segment selector so this is 0. The entries in the3831 ``.debug_aranges`` do not have a segment selector.3832 3833Line Number Information3834-----------------------3835 3836See DWARF Version 5 section 6.2 and :ref:`amdgpu-dwarf-line-number-information`.3837 3838AMDGPU does not use the ``isa`` state machine registers and always sets it to 0.3839The instruction set must be obtained from the ELF file header ``e_flags`` field3840in the ``EF_AMDGPU_MACH`` bit position (see :ref:`ELF Header3841<amdgpu-elf-header>`). See DWARF Version 5 section 6.2.2.3842 3843.. TODO::3844 3845 Should the ``isa`` state machine register be used to indicate if the code is3846 in wavefront32 or wavefront64 mode? Or used to specify the architecture ISA?3847 3848For AMDGPU the line number program header fields have the following values (see3849DWARF Version 5 section 6.2.4):3850 3851``address_size`` (ubyte)3852 Matches the address size for the ``Global`` address space defined in3853 :ref:`amdgpu-dwarf-address-space-identifier`.3854 3855``segment_selector_size`` (ubyte)3856 AMDGPU does not use a segment selector so this is 0.3857 3858``minimum_instruction_length`` (ubyte)3859 For GFX9-GFX11 this is 4.3860 3861``maximum_operations_per_instruction`` (ubyte)3862 For GFX9-GFX11 this is 1.3863 3864Source text for online-compiled programs (for example, those compiled by the3865OpenCL language runtime) may be embedded into the DWARF Version 5 line table.3866See DWARF Version 5 section 6.2.4.1 which is updated by *DWARF Extensions For3867Heterogeneous Debugging* section :ref:`DW_LNCT_LLVM_source3868<amdgpu-dwarf-line-number-information-dw-lnct-llvm-source>`.3869 3870The Clang option used to control source embedding in AMDGPU is defined in3871:ref:`amdgpu-clang-debug-options-table`.3872 3873 .. table:: AMDGPU Clang Debug Options3874 :name: amdgpu-clang-debug-options-table3875 3876 ==================== ==================================================3877 Debug Flag Description3878 ==================== ==================================================3879 -g[no-]embed-source Enable/disable embedding source text in DWARF3880 debug sections. Useful for environments where3881 source cannot be written to disk, such as3882 when performing online compilation.3883 ==================== ==================================================3884 3885For example:3886 3887``-gembed-source``3888 Enable the embedded source.3889 3890``-gno-embed-source``3891 Disable the embedded source.3892 389332-Bit and 64-Bit DWARF Formats3894-------------------------------3895 3896See DWARF Version 5 section 7.4 and3897:ref:`amdgpu-dwarf-32-bit-and-64-bit-dwarf-formats`.3898 3899For AMDGPU:3900 3901* For the ``amdgcn`` target architecture only the 64-bit process address space3902 is supported.3903 3904* The producer can generate either 32-bit or 64-bit DWARF format. LLVM generates3905 the 32-bit DWARF format.3906 3907Unit Headers3908------------3909 3910For AMDGPU the following values apply for each of the unit headers described in3911DWARF Version 5 sections 7.5.1.1, 7.5.1.2, and 7.5.1.3:3912 3913``address_size`` (ubyte)3914 Matches the address size for the ``Global`` address space defined in3915 :ref:`amdgpu-dwarf-address-space-identifier`.3916 3917.. _amdgpu-code-conventions:3918 3919Code Conventions3920================3921 3922This section provides code conventions used for each supported target triple OS3923(see :ref:`amdgpu-target-triples`).3924 3925AMDHSA3926------3927 3928This section provides code conventions used when the target triple OS is3929``amdhsa`` (see :ref:`amdgpu-target-triples`).3930 3931.. _amdgpu-amdhsa-code-object-metadata:3932 3933Code Object Metadata3934~~~~~~~~~~~~~~~~~~~~3935 3936The code object metadata specifies extensible metadata associated with the code3937objects executed on HSA [HSA]_ compatible runtimes (see :ref:`amdgpu-os`). The3938encoding and semantics of this metadata depends on the code object version; see3939:ref:`amdgpu-amdhsa-code-object-metadata-v2`,3940:ref:`amdgpu-amdhsa-code-object-metadata-v3`,3941:ref:`amdgpu-amdhsa-code-object-metadata-v4` and3942:ref:`amdgpu-amdhsa-code-object-metadata-v5`.3943 3944Code object metadata is specified in a note record (see3945:ref:`amdgpu-note-records`) and is required when the target triple OS is3946``amdhsa`` (see :ref:`amdgpu-target-triples`). It must contain the minimum3947information necessary to support the HSA compatible runtime kernel queries. For3948example, the segment sizes needed in a dispatch packet. In addition, a3949high-level language runtime may require other information to be included. For3950example, the AMD OpenCL runtime records kernel argument information.3951 3952.. _amdgpu-amdhsa-code-object-metadata-v2:3953 3954Code Object V2 Metadata3955+++++++++++++++++++++++3956 3957.. warning::3958 Code object V2 generation is no longer supported by this version of LLVM.3959 3960Code object V2 metadata is specified by the ``NT_AMD_HSA_METADATA`` note record3961(see :ref:`amdgpu-note-records-v2`).3962 3963The metadata is specified as a YAML formatted string (see [YAML]_ and3964:doc:`YamlIO`).3965 3966.. TODO::3967 3968 Is the string null terminated? It probably should not if YAML allows it to3969 contain null characters, otherwise it should be.3970 3971The metadata is represented as a single YAML document comprised of the mapping3972defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-v2-table` and3973referenced tables.3974 3975For boolean values, the string values of ``false`` and ``true`` are used for3976false and true respectively.3977 3978Additional information can be added to the mappings. To avoid conflicts, any3979non-AMD key names should be prefixed by "*vendor-name*.".3980 3981 .. table:: AMDHSA Code Object V2 Metadata Map3982 :name: amdgpu-amdhsa-code-object-metadata-map-v2-table3983 3984 ========== ============== ========= =======================================3985 String Key Value Type Required? Description3986 ========== ============== ========= =======================================3987 "Version" sequence of Required - The first integer is the major3988 2 integers version. Currently 1.3989 - The second integer is the minor3990 version. Currently 0.3991 "Printf" sequence of Each string is encoded information3992 strings about a printf function call. The3993 encoded information is organized as3994 fields separated by colon (':'):3995 3996 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString``3997 3998 where:3999 4000 ``ID``4001 A 32-bit integer as a unique id for4002 each printf function call4003 4004 ``N``4005 A 32-bit integer equal to the number4006 of arguments of printf function call4007 minus 14008 4009 ``S[i]`` (where i = 0, 1, ... , N-1)4010 32-bit integers for the size in bytes4011 of the i-th FormatString argument of4012 the printf function call4013 4014 FormatString4015 The format string passed to the4016 printf function call.4017 "Kernels" sequence of Required Sequence of the mappings for each4018 mapping kernel in the code object. See4019 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-v2-table`4020 for the definition of the mapping.4021 ========== ============== ========= =======================================4022 4023..4024 4025 .. table:: AMDHSA Code Object V2 Kernel Metadata Map4026 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-v2-table4027 4028 ================= ============== ========= ================================4029 String Key Value Type Required? Description4030 ================= ============== ========= ================================4031 "Name" string Required Source name of the kernel.4032 "SymbolName" string Required Name of the kernel4033 descriptor ELF symbol.4034 "Language" string Source language of the kernel.4035 Values include:4036 4037 - "OpenCL C"4038 - "OpenCL C++"4039 - "HCC"4040 - "OpenMP"4041 4042 "LanguageVersion" sequence of - The first integer is the major4043 2 integers version.4044 - The second integer is the4045 minor version.4046 "Attrs" mapping Mapping of kernel attributes.4047 See4048 :ref:`amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-v2-table`4049 for the mapping definition.4050 "Args" sequence of Sequence of mappings of the4051 mapping kernel arguments. See4052 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-v2-table`4053 for the definition of the mapping.4054 "CodeProps" mapping Mapping of properties related to4055 the kernel code. See4056 :ref:`amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-v2-table`4057 for the mapping definition.4058 ================= ============== ========= ================================4059 4060..4061 4062 .. table:: AMDHSA Code Object V2 Kernel Attribute Metadata Map4063 :name: amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-v2-table4064 4065 =================== ============== ========= ==============================4066 String Key Value Type Required? Description4067 =================== ============== ========= ==============================4068 "ReqdWorkGroupSize" sequence of If not 0, 0, 0 then all values4069 3 integers must be >=1 and the dispatch4070 work-group size X, Y, Z must4071 correspond to the specified4072 values. Defaults to 0, 0, 0.4073 4074 Corresponds to the OpenCL4075 ``reqd_work_group_size``4076 attribute.4077 "WorkGroupSizeHint" sequence of The dispatch work-group size4078 3 integers X, Y, Z is likely to be the4079 specified values.4080 4081 Corresponds to the OpenCL4082 ``work_group_size_hint``4083 attribute.4084 "VecTypeHint" string The name of a scalar or vector4085 type.4086 4087 Corresponds to the OpenCL4088 ``vec_type_hint`` attribute.4089 4090 "RuntimeHandle" string The external symbol name4091 associated with a kernel.4092 OpenCL runtime allocates a4093 global buffer for the symbol4094 and saves the kernel's address4095 to it, which is used for4096 device side enqueueing. Only4097 available for device side4098 enqueued kernels.4099 =================== ============== ========= ==============================4100 4101..4102 4103 .. table:: AMDHSA Code Object V2 Kernel Argument Metadata Map4104 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-v2-table4105 4106 ================= ============== ========= ================================4107 String Key Value Type Required? Description4108 ================= ============== ========= ================================4109 "Name" string Kernel argument name.4110 "TypeName" string Kernel argument type name.4111 "Size" integer Required Kernel argument size in bytes.4112 "Align" integer Required Kernel argument alignment in4113 bytes. Must be a power of two.4114 "ValueKind" string Required Kernel argument kind that4115 specifies how to set up the4116 corresponding argument.4117 Values include:4118 4119 "ByValue"4120 The argument is copied4121 directly into the kernarg.4122 4123 "GlobalBuffer"4124 A global address space pointer4125 to the buffer data is passed4126 in the kernarg.4127 4128 "DynamicSharedPointer"4129 A group address space pointer4130 to dynamically allocated LDS4131 is passed in the kernarg.4132 4133 "Sampler"4134 A global address space4135 pointer to a S# is passed in4136 the kernarg.4137 4138 "Image"4139 A global address space4140 pointer to a T# is passed in4141 the kernarg.4142 4143 "Pipe"4144 A global address space pointer4145 to an OpenCL pipe is passed in4146 the kernarg.4147 4148 "Queue"4149 A global address space pointer4150 to an OpenCL device enqueue4151 queue is passed in the4152 kernarg.4153 4154 "HiddenGlobalOffsetX"4155 The OpenCL grid dispatch4156 global offset for the X4157 dimension is passed in the4158 kernarg.4159 4160 "HiddenGlobalOffsetY"4161 The OpenCL grid dispatch4162 global offset for the Y4163 dimension is passed in the4164 kernarg.4165 4166 "HiddenGlobalOffsetZ"4167 The OpenCL grid dispatch4168 global offset for the Z4169 dimension is passed in the4170 kernarg.4171 4172 "HiddenNone"4173 An argument that is not used4174 by the kernel. Space needs to4175 be left for it, but it does4176 not need to be set up.4177 4178 "HiddenPrintfBuffer"4179 A global address space pointer4180 to the runtime printf buffer4181 is passed in kernarg. Mutually4182 exclusive with4183 "HiddenHostcallBuffer".4184 4185 "HiddenHostcallBuffer"4186 A global address space pointer4187 to the runtime hostcall buffer4188 is passed in kernarg. Mutually4189 exclusive with4190 "HiddenPrintfBuffer".4191 4192 "HiddenDefaultQueue"4193 A global address space pointer4194 to the OpenCL device enqueue4195 queue that should be used by4196 the kernel by default is4197 passed in the kernarg.4198 4199 "HiddenCompletionAction"4200 A global address space pointer4201 to help link enqueued kernels into4202 the ancestor tree for determining4203 when the parent kernel has finished.4204 4205 "HiddenMultiGridSyncArg"4206 A global address space pointer for4207 multi-grid synchronization is4208 passed in the kernarg.4209 4210 "ValueType" string Unused and deprecated. This should no longer4211 be emitted, but is accepted for compatibility.4212 4213 4214 "PointeeAlign" integer Alignment in bytes of pointee4215 type for pointer type kernel4216 argument. Must be a power4217 of 2. Only present if4218 "ValueKind" is4219 "DynamicSharedPointer".4220 "AddrSpaceQual" string Kernel argument address space4221 qualifier. Only present if4222 "ValueKind" is "GlobalBuffer" or4223 "DynamicSharedPointer". Values4224 are:4225 4226 - "Private"4227 - "Global"4228 - "Constant"4229 - "Local"4230 - "Generic"4231 - "Region"4232 4233 .. TODO::4234 4235 Is GlobalBuffer only Global4236 or Constant? Is4237 DynamicSharedPointer always4238 Local? Can HCC allow Generic?4239 How can Private or Region4240 ever happen?4241 4242 "AccQual" string Kernel argument access4243 qualifier. Only present if4244 "ValueKind" is "Image" or4245 "Pipe". Values4246 are:4247 4248 - "ReadOnly"4249 - "WriteOnly"4250 - "ReadWrite"4251 4252 .. TODO::4253 4254 Does this apply to4255 GlobalBuffer?4256 4257 "ActualAccQual" string The actual memory accesses4258 performed by the kernel on the4259 kernel argument. Only present if4260 "ValueKind" is "GlobalBuffer",4261 "Image", or "Pipe". This may be4262 more restrictive than indicated4263 by "AccQual" to reflect what the4264 kernel actually does. If not4265 present then the runtime must4266 assume what is implied by4267 "AccQual" and "IsConst". Values4268 are:4269 4270 - "ReadOnly"4271 - "WriteOnly"4272 - "ReadWrite"4273 4274 "IsConst" boolean Indicates if the kernel argument4275 is const qualified. Only present4276 if "ValueKind" is4277 "GlobalBuffer".4278 4279 "IsRestrict" boolean Indicates if the kernel argument4280 is restrict qualified. Only4281 present if "ValueKind" is4282 "GlobalBuffer".4283 4284 "IsVolatile" boolean Indicates if the kernel argument4285 is volatile qualified. Only4286 present if "ValueKind" is4287 "GlobalBuffer".4288 4289 "IsPipe" boolean Indicates if the kernel argument4290 is pipe qualified. Only present4291 if "ValueKind" is "Pipe".4292 4293 .. TODO::4294 4295 Can GlobalBuffer be pipe4296 qualified?4297 4298 ================= ============== ========= ================================4299 4300..4301 4302 .. table:: AMDHSA Code Object V2 Kernel Code Properties Metadata Map4303 :name: amdgpu-amdhsa-code-object-kernel-code-properties-metadata-map-v2-table4304 4305 ============================ ============== ========= =====================4306 String Key Value Type Required? Description4307 ============================ ============== ========= =====================4308 "KernargSegmentSize" integer Required The size in bytes of4309 the kernarg segment4310 that holds the values4311 of the arguments to4312 the kernel.4313 "GroupSegmentFixedSize" integer Required The amount of group4314 segment memory4315 required by a4316 work-group in4317 bytes. This does not4318 include any4319 dynamically allocated4320 group segment memory4321 that may be added4322 when the kernel is4323 dispatched.4324 "PrivateSegmentFixedSize" integer Required The amount of fixed4325 private address space4326 memory required for a4327 work-item in4328 bytes. If the kernel4329 uses a dynamic call4330 stack then additional4331 space must be added4332 to this value for the4333 call stack.4334 "KernargSegmentAlign" integer Required The maximum byte4335 alignment of4336 arguments in the4337 kernarg segment. Must4338 be a power of 2.4339 "WavefrontSize" integer Required Wavefront size. Must4340 be a power of 2.4341 "NumSGPRs" integer Required Number of scalar4342 registers used by a4343 wavefront for4344 GFX6-GFX11. This4345 includes the special4346 SGPRs for VCC, Flat4347 Scratch (GFX7-GFX10)4348 and XNACK (for4349 GFX8-GFX10). It does4350 not include the 164351 SGPR added if a trap4352 handler is4353 enabled. It is not4354 rounded up to the4355 allocation4356 granularity.4357 "NumVGPRs" integer Required Number of vector4358 registers used by4359 each work-item for4360 GFX6-GFX114361 "MaxFlatWorkGroupSize" integer Required Maximum flat4362 work-group size4363 supported by the4364 kernel in work-items.4365 Must be >=1 and4366 consistent with4367 ReqdWorkGroupSize if4368 not 0, 0, 0.4369 "NumSpilledSGPRs" integer Number of stores from4370 a scalar register to4371 a register allocator4372 created spill4373 location.4374 "NumSpilledVGPRs" integer Number of stores from4375 a vector register to4376 a register allocator4377 created spill4378 location.4379 ============================ ============== ========= =====================4380 4381.. _amdgpu-amdhsa-code-object-metadata-v3:4382 4383Code Object V3 Metadata4384+++++++++++++++++++++++4385 4386.. warning::4387 Code object V3 generation is no longer supported by this version of LLVM.4388 4389Code object V3 and above metadata is specified by the ``NT_AMDGPU_METADATA`` note4390record (see :ref:`amdgpu-note-records-v3-onwards`).4391 4392The metadata is represented as Message Pack formatted binary data (see4393[MsgPack]_). The top level is a Message Pack map that includes the4394keys defined in table4395:ref:`amdgpu-amdhsa-code-object-metadata-map-table-v3` and referenced4396tables.4397 4398Additional information can be added to the maps. To avoid conflicts,4399any key names should be prefixed by "*vendor-name*." where4400``vendor-name`` can be the name of the vendor and specific vendor4401tool that generates the information. The prefix is abbreviated to4402simply "." when it appears within a map that has been added by the4403same *vendor-name*.4404 4405 .. table:: AMDHSA Code Object V3 Metadata Map4406 :name: amdgpu-amdhsa-code-object-metadata-map-table-v34407 4408 ================= ============== ========= =======================================4409 String Key Value Type Required? Description4410 ================= ============== ========= =======================================4411 "amdhsa.version" sequence of Required - The first integer is the major4412 2 integers version. Currently 1.4413 - The second integer is the minor4414 version. Currently 0.4415 "amdhsa.printf" sequence of Each string is encoded information4416 strings about a printf function call. The4417 encoded information is organized as4418 fields separated by colon (':'):4419 4420 ``ID:N:S[0]:S[1]:...:S[N-1]:FormatString``4421 4422 where:4423 4424 ``ID``4425 A 32-bit integer as a unique id for4426 each printf function call4427 4428 ``N``4429 A 32-bit integer equal to the number4430 of arguments of printf function call4431 minus 14432 4433 ``S[i]`` (where i = 0, 1, ... , N-1)4434 32-bit integers for the size in bytes4435 of the i-th FormatString argument of4436 the printf function call4437 4438 FormatString4439 The format string passed to the4440 printf function call.4441 "amdhsa.kernels" sequence of Required Sequence of the maps for each4442 map kernel in the code object. See4443 :ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-table-v3`4444 for the definition of the keys included4445 in that map.4446 ================= ============== ========= =======================================4447 4448..4449 4450 .. table:: AMDHSA Code Object V3 Kernel Metadata Map4451 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-table-v34452 4453 =================================== ============== ========= ================================4454 String Key Value Type Required? Description4455 =================================== ============== ========= ================================4456 ".name" string Required Source name of the kernel.4457 ".symbol" string Required Name of the kernel4458 descriptor ELF symbol.4459 ".language" string Source language of the kernel.4460 Values include:4461 4462 - "OpenCL C"4463 - "OpenCL C++"4464 - "HCC"4465 - "HIP"4466 - "OpenMP"4467 - "Assembler"4468 4469 ".language_version" sequence of - The first integer is the major4470 2 integers version.4471 - The second integer is the4472 minor version.4473 ".args" sequence of Sequence of maps of the4474 map kernel arguments. See4475 :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3`4476 for the definition of the keys4477 included in that map.4478 ".reqd_workgroup_size" sequence of If not 0, 0, 0 then all values4479 3 integers must be >=1 and the dispatch4480 work-group size X, Y, Z must4481 correspond to the specified4482 values. Defaults to 0, 0, 0.4483 4484 Corresponds to the OpenCL4485 ``reqd_work_group_size``4486 attribute.4487 ".workgroup_size_hint" sequence of The dispatch work-group size4488 3 integers X, Y, Z is likely to be the4489 specified values.4490 4491 Corresponds to the OpenCL4492 ``work_group_size_hint``4493 attribute.4494 ".vec_type_hint" string The name of a scalar or vector4495 type.4496 4497 Corresponds to the OpenCL4498 ``vec_type_hint`` attribute.4499 4500 ".device_enqueue_symbol" string The external symbol name4501 associated with a kernel.4502 OpenCL runtime allocates a4503 global buffer for the symbol4504 and saves the kernel's address4505 to it, which is used for4506 device side enqueueing. Only4507 available for device side4508 enqueued kernels.4509 ".kernarg_segment_size" integer Required The size in bytes of4510 the kernarg segment4511 that holds the values4512 of the arguments to4513 the kernel.4514 ".group_segment_fixed_size" integer Required The amount of group4515 segment memory4516 required by a4517 work-group in4518 bytes. This does not4519 include any4520 dynamically allocated4521 group segment memory4522 that may be added4523 when the kernel is4524 dispatched.4525 ".private_segment_fixed_size" integer Required The amount of fixed4526 private address space4527 memory required for a4528 work-item in4529 bytes. If the kernel4530 uses a dynamic call4531 stack then additional4532 space must be added4533 to this value for the4534 call stack.4535 ".kernarg_segment_align" integer Required The maximum byte4536 alignment of4537 arguments in the4538 kernarg segment. Must4539 be a power of 2.4540 ".wavefront_size" integer Required Wavefront size. Must4541 be a power of 2.4542 ".sgpr_count" integer Required Number of scalar4543 registers required by a4544 wavefront for4545 GFX6-GFX9. A register4546 is required if it is4547 used explicitly, or4548 if a higher numbered4549 register is used4550 explicitly. This4551 includes the special4552 SGPRs for VCC, Flat4553 Scratch (GFX7-GFX9)4554 and XNACK (for4555 GFX8-GFX9). It does4556 not include the 164557 SGPR added if a trap4558 handler is4559 enabled. It is not4560 rounded up to the4561 allocation4562 granularity.4563 ".vgpr_count" integer Required Number of vector4564 registers required by4565 each work-item for4566 GFX6-GFX9. A register4567 is required if it is4568 used explicitly, or4569 if a higher numbered4570 register is used4571 explicitly.4572 ".agpr_count" integer Required Number of accumulator4573 registers required by4574 each work-item for4575 GFX90A, GFX908.4576 ".max_flat_workgroup_size" integer Required Maximum flat4577 work-group size4578 supported by the4579 kernel in work-items.4580 Must be >=1 and4581 consistent with4582 ReqdWorkGroupSize if4583 not 0, 0, 0.4584 ".sgpr_spill_count" integer Number of stores from4585 a scalar register to4586 a register allocator4587 created spill4588 location.4589 ".vgpr_spill_count" integer Number of stores from4590 a vector register to4591 a register allocator4592 created spill4593 location.4594 ".kind" string The kind of the kernel4595 with the following4596 values:4597 4598 "normal"4599 Regular kernels.4600 4601 "init"4602 These kernels must be4603 invoked after loading4604 the containing code4605 object and must4606 complete before any4607 normal and fini4608 kernels in the same4609 code object are4610 invoked.4611 4612 "fini"4613 These kernels must be4614 invoked before4615 unloading the4616 containing code object4617 and after all init and4618 normal kernels in the4619 same code object have4620 been invoked and4621 completed.4622 4623 If omitted, "normal" is4624 assumed.4625 ".max_num_work_groups_{x,y,z}" integer The max number of4626 launched work-groups4627 in the X, Y, and Z4628 dimensions. Each number4629 must be >=1.4630 =================================== ============== ========= ================================4631 4632..4633 4634 .. table:: AMDHSA Code Object V3 Kernel Argument Metadata Map4635 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v34636 4637 ====================== ============== ========= ================================4638 String Key Value Type Required? Description4639 ====================== ============== ========= ================================4640 ".name" string Kernel argument name.4641 ".type_name" string Kernel argument type name.4642 ".size" integer Required Kernel argument size in bytes.4643 ".offset" integer Required Kernel argument offset in4644 bytes. The offset must be a4645 multiple of the alignment4646 required by the argument.4647 ".value_kind" string Required Kernel argument kind that4648 specifies how to set up the4649 corresponding argument.4650 Values include:4651 4652 "by_value"4653 The argument is copied4654 directly into the kernarg.4655 4656 "global_buffer"4657 A global address space pointer4658 to the buffer data is passed4659 in the kernarg.4660 4661 "dynamic_shared_pointer"4662 A group address space pointer4663 to dynamically allocated LDS4664 is passed in the kernarg.4665 4666 "sampler"4667 A global address space4668 pointer to a S# is passed in4669 the kernarg.4670 4671 "image"4672 A global address space4673 pointer to a T# is passed in4674 the kernarg.4675 4676 "pipe"4677 A global address space pointer4678 to an OpenCL pipe is passed in4679 the kernarg.4680 4681 "queue"4682 A global address space pointer4683 to an OpenCL device enqueue4684 queue is passed in the4685 kernarg.4686 4687 "hidden_global_offset_x"4688 The OpenCL grid dispatch4689 global offset for the X4690 dimension is passed in the4691 kernarg.4692 4693 "hidden_global_offset_y"4694 The OpenCL grid dispatch4695 global offset for the Y4696 dimension is passed in the4697 kernarg.4698 4699 "hidden_global_offset_z"4700 The OpenCL grid dispatch4701 global offset for the Z4702 dimension is passed in the4703 kernarg.4704 4705 "hidden_none"4706 An argument that is not used4707 by the kernel. Space needs to4708 be left for it, but it does4709 not need to be set up.4710 4711 "hidden_printf_buffer"4712 A global address space pointer4713 to the runtime printf buffer4714 is passed in kernarg. Mutually4715 exclusive with4716 "hidden_hostcall_buffer"4717 before Code Object V5.4718 4719 "hidden_hostcall_buffer"4720 A global address space pointer4721 to the runtime hostcall buffer4722 is passed in kernarg. Mutually4723 exclusive with4724 "hidden_printf_buffer"4725 before Code Object V5.4726 4727 "hidden_default_queue"4728 A global address space pointer4729 to the OpenCL device enqueue4730 queue that should be used by4731 the kernel by default is4732 passed in the kernarg.4733 4734 "hidden_completion_action"4735 A global address space pointer4736 to help link enqueued kernels into4737 the ancestor tree for determining4738 when the parent kernel has finished.4739 4740 "hidden_multigrid_sync_arg"4741 A global address space pointer for4742 multi-grid synchronization is4743 passed in the kernarg.4744 4745 ".value_type" string Unused and deprecated. This should no longer4746 be emitted, but is accepted for compatibility.4747 4748 ".pointee_align" integer Alignment in bytes of pointee4749 type for pointer type kernel4750 argument. Must be a power4751 of 2. Only present if4752 ".value_kind" is4753 "dynamic_shared_pointer".4754 ".address_space" string Kernel argument address space4755 qualifier. Only present if4756 ".value_kind" is "global_buffer" or4757 "dynamic_shared_pointer". Values4758 are:4759 4760 - "private"4761 - "global"4762 - "constant"4763 - "local"4764 - "generic"4765 - "region"4766 4767 .. TODO::4768 4769 Is "global_buffer" only "global"4770 or "constant"? Is4771 "dynamic_shared_pointer" always4772 "local"? Can HCC allow "generic"?4773 How can "private" or "region"4774 ever happen?4775 4776 ".access" string Kernel argument access4777 qualifier. Only present if4778 ".value_kind" is "image" or4779 "pipe". Values4780 are:4781 4782 - "read_only"4783 - "write_only"4784 - "read_write"4785 4786 .. TODO::4787 4788 Does this apply to4789 "global_buffer"?4790 4791 ".actual_access" string The actual memory accesses4792 performed by the kernel on the4793 kernel argument. Only present if4794 ".value_kind" is "global_buffer",4795 "image", or "pipe". This may be4796 more restrictive than indicated4797 by ".access" to reflect what the4798 kernel actually does. If not4799 present then the runtime must4800 assume what is implied by4801 ".access" and ".is_const" . Values4802 are:4803 4804 - "read_only"4805 - "write_only"4806 - "read_write"4807 4808 ".is_const" boolean Indicates if the kernel argument4809 is const qualified. Only present4810 if ".value_kind" is4811 "global_buffer".4812 4813 ".is_restrict" boolean Indicates if the kernel argument4814 is restrict qualified. Only4815 present if ".value_kind" is4816 "global_buffer".4817 4818 ".is_volatile" boolean Indicates if the kernel argument4819 is volatile qualified. Only4820 present if ".value_kind" is4821 "global_buffer".4822 4823 ".is_pipe" boolean Indicates if the kernel argument4824 is pipe qualified. Only present4825 if ".value_kind" is "pipe".4826 4827 .. TODO::4828 4829 Can "global_buffer" be pipe4830 qualified?4831 4832 ====================== ============== ========= ================================4833 4834.. _amdgpu-amdhsa-code-object-metadata-v4:4835 4836Code Object V4 Metadata4837+++++++++++++++++++++++4838 4839.. warning::4840 Code object V4 is not the default code object version emitted by this version4841 of LLVM.4842 4843Code object V4 metadata is the same as4844:ref:`amdgpu-amdhsa-code-object-metadata-v3` with the changes and additions4845defined in table :ref:`amdgpu-amdhsa-code-object-metadata-map-table-v4`.4846 4847 .. table:: AMDHSA Code Object V4 Metadata Map Changes4848 :name: amdgpu-amdhsa-code-object-metadata-map-table-v44849 4850 ================= ============== ========= =======================================4851 String Key Value Type Required? Description4852 ================= ============== ========= =======================================4853 "amdhsa.version" sequence of Required - The first integer is the major4854 2 integers version. Currently 1.4855 - The second integer is the minor4856 version. Currently 1.4857 "amdhsa.target" string Required The target name of the code using the syntax:4858 4859 .. code::4860 4861 <target-triple> [ "-" <target-id> ]4862 4863 A canonical target ID must be4864 used. See :ref:`amdgpu-target-triples`4865 and :ref:`amdgpu-target-id`.4866 ================= ============== ========= =======================================4867 4868.. _amdgpu-amdhsa-code-object-metadata-v5:4869 4870Code Object V5 Metadata4871+++++++++++++++++++++++4872 4873Code object V5 metadata is the same as4874:ref:`amdgpu-amdhsa-code-object-metadata-v4` with the changes defined in table4875:ref:`amdgpu-amdhsa-code-object-metadata-map-table-v5`, table4876:ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-table-v5` and table4877:ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v5`.4878 4879 .. table:: AMDHSA Code Object V5 Metadata Map Changes4880 :name: amdgpu-amdhsa-code-object-metadata-map-table-v54881 4882 ================= ============== ========= =======================================4883 String Key Value Type Required? Description4884 ================= ============== ========= =======================================4885 "amdhsa.version" sequence of Required - The first integer is the major4886 2 integers version. Currently 1.4887 - The second integer is the minor4888 version. Currently 2.4889 ================= ============== ========= =======================================4890 4891..4892 4893 .. table:: AMDHSA Code Object V5 Kernel Metadata Map Additions4894 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-table-v54895 4896 ============================= ============= ========== =======================================4897 String Key Value Type Required? Description4898 ============================= ============= ========== =======================================4899 ".uses_dynamic_stack" boolean Indicates if the generated machine code4900 is using a dynamically sized stack.4901 ".workgroup_processor_mode" boolean (GFX10+) Controls ENABLE_WGP_MODE in4902 :ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`.4903 ============================= ============= ========== =======================================4904 4905..4906 4907 .. table:: AMDHSA Code Object V5 Kernel Attribute Metadata Map4908 :name: amdgpu-amdhsa-code-object-kernel-attribute-metadata-map-v5-table4909 4910 =========================== ============== ========= ==============================4911 String Key Value Type Required? Description4912 =========================== ============== ========= ==============================4913 ".uniform_work_group_size" integer Indicates if the kernel4914 requires that each dimension4915 of global size is a multiple4916 of corresponding dimension of4917 work-group size. Value of 14918 implies true and value of 04919 implies false. Metadata is4920 only emitted when value is 1.4921 =========================== ============== ========= ==============================4922 4923..4924 4925..4926 4927 .. table:: AMDHSA Code Object V5 Kernel Argument Metadata Map Additions and Changes4928 :name: amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v54929 4930 ====================== ============== ========= ================================4931 String Key Value Type Required? Description4932 ====================== ============== ========= ================================4933 ".value_kind" string Required Kernel argument kind that4934 specifies how to set up the4935 corresponding argument.4936 Values include:4937 the same as code object V3 metadata4938 (see :ref:`amdgpu-amdhsa-code-object-kernel-argument-metadata-map-table-v3`)4939 with the following additions:4940 4941 "hidden_block_count_x"4942 The grid dispatch work-group count for the X dimension4943 is passed in the kernarg. Some languages, such as OpenCL,4944 support a last work-group in each dimension being partial.4945 This count only includes the non-partial work-group count.4946 This is not the same as the value in the AQL dispatch packet,4947 which has the grid size in work-items.4948 4949 "hidden_block_count_y"4950 The grid dispatch work-group count for the Y dimension4951 is passed in the kernarg. Some languages, such as OpenCL,4952 support a last work-group in each dimension being partial.4953 This count only includes the non-partial work-group count.4954 This is not the same as the value in the AQL dispatch packet,4955 which has the grid size in work-items. If the grid dimensionality4956 is 1, then must be 1.4957 4958 "hidden_block_count_z"4959 The grid dispatch work-group count for the Z dimension4960 is passed in the kernarg. Some languages, such as OpenCL,4961 support a last work-group in each dimension being partial.4962 This count only includes the non-partial work-group count.4963 This is not the same as the value in the AQL dispatch packet,4964 which has the grid size in work-items. If the grid dimensionality4965 is 1 or 2, then must be 1.4966 4967 "hidden_group_size_x"4968 The grid dispatch work-group size for the X dimension is4969 passed in the kernarg. This size only applies to the4970 non-partial work-groups. This is the same value as the AQL4971 dispatch packet work-group size.4972 4973 "hidden_group_size_y"4974 The grid dispatch work-group size for the Y dimension is4975 passed in the kernarg. This size only applies to the4976 non-partial work-groups. This is the same value as the AQL4977 dispatch packet work-group size. If the grid dimensionality4978 is 1, then must be 1.4979 4980 "hidden_group_size_z"4981 The grid dispatch work-group size for the Z dimension is4982 passed in the kernarg. This size only applies to the4983 non-partial work-groups. This is the same value as the AQL4984 dispatch packet work-group size. If the grid dimensionality4985 is 1 or 2, then must be 1.4986 4987 "hidden_remainder_x"4988 The grid dispatch work group size of the partial work group4989 of the X dimension, if it exists. Must be zero if a partial4990 work group does not exist in the X dimension.4991 4992 "hidden_remainder_y"4993 The grid dispatch work group size of the partial work group4994 of the Y dimension, if it exists. Must be zero if a partial4995 work group does not exist in the Y dimension.4996 4997 "hidden_remainder_z"4998 The grid dispatch work group size of the partial work group4999 of the Z dimension, if it exists. Must be zero if a partial5000 work group does not exist in the Z dimension.5001 5002 "hidden_grid_dims"5003 The grid dispatch dimensionality. This is the same value5004 as the AQL dispatch packet dimensionality. Must be a value5005 between 1 and 3.5006 5007 "hidden_heap_v1"5008 A global address space pointer to an initialized memory5009 buffer that conforms to the requirements of the malloc/free5010 device library V1 version implementation.5011 5012 "hidden_dynamic_lds_size"5013 Size of the dynamically allocated LDS memory is passed in the kernarg.5014 5015 "hidden_private_base"5016 The high 32 bits of the flat addressing private aperture base.5017 Only used by GFX8 to allow conversion between private segment5018 and flat addresses. See :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`.5019 5020 "hidden_shared_base"5021 The high 32 bits of the flat addressing shared aperture base.5022 Only used by GFX8 to allow conversion between shared segment5023 and flat addresses. See :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`.5024 5025 "hidden_queue_ptr"5026 A global memory address space pointer to the ROCm runtime5027 ``struct amd_queue_t`` structure for the HSA queue of the5028 associated dispatch AQL packet. It is only required for pre-GFX95029 devices for the trap handler ABI (see :ref:`amdgpu-amdhsa-trap-handler-abi`).5030 5031 ====================== ============== ========= ================================5032 5033.. _amdgpu-amdhsa-code-object-metadata-v6:5034 5035Code Object V6 Metadata5036+++++++++++++++++++++++5037 5038Code object V6 metadata is the same as5039:ref:`amdgpu-amdhsa-code-object-metadata-v5` with the changes defined in table5040:ref:`amdgpu-amdhsa-code-object-kernel-metadata-map-table-v6`.5041 5042 .. table:: AMDHSA Code Object V6 Kernel Metadata Map Additions5043 :name: amdgpu-amdhsa-code-object-kernel-metadata-map-table-v65044 5045 ============================= ============= ========== =======================================5046 String Key Value Type Required? Description5047 ============================= ============= ========== =======================================5048 ".cluster_dims" sequence of The dimension of the cluster.5049 3 integers5050 ============================= ============= ========== =======================================5051 5052Kernel Dispatch5053~~~~~~~~~~~~~~~5054 5055The HSA architected queuing language (AQL) defines a user space memory interface5056that can be used to control the dispatch of kernels, in an agent independent5057way. An agent can have zero or more AQL queues created for it using an HSA5058compatible runtime (see :ref:`amdgpu-os`), in which AQL packets (all of which5059are 64 bytes) can be placed. See the *HSA Platform System Architecture5060Specification* [HSA]_ for the AQL queue mechanics and packet layouts.5061 5062The packet processor of a kernel agent is responsible for detecting and5063dispatching HSA kernels from the AQL queues associated with it. For AMD GPUs the5064packet processor is implemented by the hardware command processor (CP),5065asynchronous dispatch controller (ADC) and shader processor input controller5066(SPI).5067 5068An HSA compatible runtime can be used to allocate an AQL queue object. It uses5069the kernel mode driver to initialize and register the AQL queue with CP.5070 5071To dispatch a kernel the following actions are performed. This can occur in the5072CPU host program, or from an HSA kernel executing on a GPU.5073 50741. A pointer to an AQL queue for the kernel agent on which the kernel is to be5075 executed is obtained.50762. A pointer to the kernel descriptor (see5077 :ref:`amdgpu-amdhsa-kernel-descriptor`) of the kernel to execute is obtained.5078 It must be for a kernel that is contained in a code object that was loaded5079 by an HSA compatible runtime on the kernel agent with which the AQL queue is5080 associated.50813. Space is allocated for the kernel arguments using the HSA compatible runtime5082 allocator for a memory region with the kernarg property for the kernel agent5083 that will execute the kernel. It must be at least 16-byte aligned.50844. Kernel argument values are assigned to the kernel argument memory5085 allocation. The layout is defined in the *HSA Programmer's Language5086 Reference* [HSA]_. For AMDGPU the kernel execution directly accesses the5087 kernel argument memory in the same way constant memory is accessed. (Note5088 that the HSA specification allows an implementation to copy the kernel5089 argument contents to another location that is accessed by the kernel.)50905. An AQL kernel dispatch packet is created on the AQL queue. The HSA compatible5091 runtime api uses 64-bit atomic operations to reserve space in the AQL queue5092 for the packet. The packet must be set up, and the final write must use an5093 atomic store release to set the packet kind to ensure the packet contents are5094 visible to the kernel agent. AQL defines a doorbell signal mechanism to5095 notify the kernel agent that the AQL queue has been updated. These rules, and5096 the layout of the AQL queue and kernel dispatch packet is defined in the *HSA5097 System Architecture Specification* [HSA]_.50986. A kernel dispatch packet includes information about the actual dispatch,5099 such as grid and work-group size, together with information from the code5100 object about the kernel, such as segment sizes. The HSA compatible runtime5101 queries on the kernel symbol can be used to obtain the code object values5102 which are recorded in the :ref:`amdgpu-amdhsa-code-object-metadata`.51037. CP executes micro-code and is responsible for detecting and setting up the5104 GPU to execute the wavefronts of a kernel dispatch.51058. CP ensures that when the a wavefront starts executing the kernel machine5106 code, the scalar general purpose registers (SGPR) and vector general purpose5107 registers (VGPR) are set up as required by the machine code. The required5108 setup is defined in the :ref:`amdgpu-amdhsa-kernel-descriptor`. The initial5109 register state is defined in5110 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`.51119. The prolog of the kernel machine code (see5112 :ref:`amdgpu-amdhsa-kernel-prolog`) sets up the machine state as necessary5113 before continuing executing the machine code that corresponds to the kernel.511410. When the kernel dispatch has completed execution, CP signals the completion5115 signal specified in the kernel dispatch packet if not 0.5116 5117.. _amdgpu-amdhsa-memory-spaces:5118 5119Memory Spaces5120~~~~~~~~~~~~~5121 5122The memory space properties are:5123 5124 .. table:: AMDHSA Memory Spaces5125 :name: amdgpu-amdhsa-memory-spaces-table5126 5127 ================= =========== ======== ======= ==================5128 Memory Space Name HSA Segment Hardware Address NULL Value5129 Name Name Size5130 ================= =========== ======== ======= ==================5131 Private private scratch 32 0x000000005132 Local group LDS 32 0xFFFFFFFF5133 Global global global 64 0x00000000000000005134 Constant constant *same as 64 0x00000000000000005135 global*5136 Generic flat flat 64 0x00000000000000005137 Region N/A GDS 32 *not implemented5138 for AMDHSA*5139 ================= =========== ======== ======= ==================5140 5141The global and constant memory spaces both use global virtual addresses, which5142are the same virtual address space used by the CPU. However, some virtual5143addresses may only be accessible to the CPU, some only accessible by the GPU,5144and some by both.5145 5146Using the constant memory space indicates that the data will not change during5147the execution of the kernel. This allows scalar read instructions to be5148used. The vector and scalar L1 caches are invalidated of volatile data before5149each kernel dispatch execution to allow constant memory to change values between5150kernel dispatches.5151 5152The local memory space uses the hardware Local Data Store (LDS) which is5153automatically allocated when the hardware creates work-groups of wavefronts, and5154freed when all the wavefronts of a work-group have terminated. The data store5155(DS) instructions can be used to access it.5156 5157The private memory space uses the hardware scratch memory support. If the kernel5158uses scratch, then the hardware allocates memory that is accessed using5159wavefront lane dword (4 byte) interleaving. The mapping used from private5160address to physical address is:5161 5162 ``wavefront-scratch-base +5163 (private-address * wavefront-size * 4) +5164 (wavefront-lane-id * 4)``5165 5166There are different ways that the wavefront scratch base address is determined5167by a wavefront (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). This5168memory can be accessed in an interleaved manner using buffer instruction with5169the scratch buffer descriptor and per wavefront scratch offset, by the scratch5170instructions, or by flat instructions. If each lane of a wavefront accesses the5171same private address, the interleaving results in adjacent dwords being accessed5172and hence requires fewer cache lines to be fetched. Multi-dword access is not5173supported except by flat and scratch instructions in GFX9-GFX11.5174 5175The generic address space uses the hardware flat address support available in5176GFX7-GFX11. This uses two fixed ranges of virtual addresses (the private and5177local apertures), that are outside the range of addressable global memory, to5178map from a flat address to a private or local address.5179 5180FLAT instructions can take a flat address and access global, private (scratch)5181and group (LDS) memory depending on if the address is within one of the5182aperture ranges. Flat access to scratch requires hardware aperture setup and5183setup in the kernel prologue (see5184:ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`). Flat access to LDS requires5185hardware aperture setup and M0 (GFX7-GFX8) register setup (see5186:ref:`amdgpu-amdhsa-kernel-prolog-m0`).5187 5188To convert between a segment address and a flat address the base address of the5189apertures address can be used. For GFX7-GFX8 these are available in the5190:ref:`amdgpu-amdhsa-hsa-aql-queue` the address of which can be obtained with5191Queue Ptr SGPR (see :ref:`amdgpu-amdhsa-initial-kernel-execution-state`). For5192GFX9-GFX11 the aperture base addresses are directly available as inline constant5193registers ``SRC_SHARED_BASE/LIMIT`` and ``SRC_PRIVATE_BASE/LIMIT``. In 64-bit5194address mode the aperture sizes are 2^32 bytes and the base is aligned to 2^325195which makes it easier to convert from flat to segment or segment to flat.5196 5197Image and Samplers5198~~~~~~~~~~~~~~~~~~5199 5200Image and sample handles created by an HSA compatible runtime (see5201:ref:`amdgpu-os`) are 64-bit addresses of a hardware 32-byte V# and 48 byte S#5202object respectively. In order to support the HSA ``query_sampler`` operations5203two extra dwords are used to store the HSA BRIG enumeration values for the5204queries that are not trivially deducible from the S# representation.5205 5206HSA Signals5207~~~~~~~~~~~5208 5209HSA signal handles created by an HSA compatible runtime (see :ref:`amdgpu-os`)5210are 64-bit addresses of a structure allocated in memory accessible from both the5211CPU and GPU. The structure is defined by the runtime and subject to change5212between releases. For example, see [AMD-ROCm-github]_.5213 5214.. _amdgpu-amdhsa-hsa-aql-queue:5215 5216HSA AQL Queue5217~~~~~~~~~~~~~5218 5219The HSA AQL queue structure is defined by an HSA compatible runtime (see5220:ref:`amdgpu-os`) and subject to change between releases. For example, see5221[AMD-ROCm-github]_. For some processors it contains fields needed to implement5222certain language features such as the flat address aperture bases. It also5223contains fields used by CP such as managing the allocation of scratch memory.5224 5225.. _amdgpu-amdhsa-kernel-descriptor:5226 5227Kernel Descriptor5228~~~~~~~~~~~~~~~~~5229 5230A kernel descriptor consists of the information needed by CP to initiate the5231execution of a kernel, including the entry point address of the machine code5232that implements the kernel.5233 5234Code Object V3 Kernel Descriptor5235++++++++++++++++++++++++++++++++5236 5237CP microcode requires the Kernel descriptor to be allocated on 64-byte5238alignment.5239 5240The fields used by CP for code objects before V3 also match those specified in5241:ref:`amdgpu-amdhsa-kernel-descriptor-v3-table`.5242 5243 .. table:: Code Object V3 Kernel Descriptor5244 :name: amdgpu-amdhsa-kernel-descriptor-v3-table5245 5246 ======= ======= =============================== ============================5247 Bits Size Field Name Description5248 ======= ======= =============================== ============================5249 31:0 4 bytes GROUP_SEGMENT_FIXED_SIZE The amount of fixed local5250 address space memory5251 required for a work-group5252 in bytes. This does not5253 include any dynamically5254 allocated local address5255 space memory that may be5256 added when the kernel is5257 dispatched.5258 63:32 4 bytes PRIVATE_SEGMENT_FIXED_SIZE The amount of fixed5259 private address space5260 memory required for a5261 work-item in bytes. When5262 this cannot be predicted,5263 code object v4 and older5264 sets this value to be5265 higher than the minimum5266 requirement.5267 95:64 4 bytes KERNARG_SIZE The size of the kernarg5268 memory pointed to by the5269 AQL dispatch packet. The5270 kernarg memory is used to5271 pass arguments to the5272 kernel.5273 5274 * If the kernarg pointer in5275 the dispatch packet is NULL5276 then there are no kernel5277 arguments.5278 * If the kernarg pointer in5279 the dispatch packet is5280 not NULL and this value5281 is 0 then the kernarg5282 memory size is5283 unspecified.5284 * If the kernarg pointer in5285 the dispatch packet is5286 not NULL and this value5287 is not 0 then the value5288 specifies the kernarg5289 memory size in bytes. It5290 is recommended to provide5291 a value as it may be used5292 by CP to optimize making5293 the kernarg memory5294 visible to the kernel5295 code.5296 5297 127:96 4 bytes Reserved, must be 0.5298 191:128 8 bytes KERNEL_CODE_ENTRY_BYTE_OFFSET Byte offset (possibly5299 negative) from base5300 address of kernel5301 descriptor to kernel's5302 entry point instruction5303 which must be 256 byte5304 aligned.5305 351:192 20 Reserved, must be 0.5306 bytes5307 383:352 4 bytes COMPUTE_PGM_RSRC3 GFX6-GFX95308 Reserved, must be 0.5309 GFX90A, GFX9425310 Compute Shader (CS)5311 program settings used by5312 CP to set up5313 ``COMPUTE_PGM_RSRC3``5314 configuration5315 register. See5316 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table`.5317 GFX10-GFX115318 Compute Shader (CS)5319 program settings used by5320 CP to set up5321 ``COMPUTE_PGM_RSRC3``5322 configuration5323 register. See5324 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-gfx11-table`.5325 GFX125326 Compute Shader (CS)5327 program settings used by5328 CP to set up5329 ``COMPUTE_PGM_RSRC3``5330 configuration5331 register. See5332 :ref:`amdgpu-amdhsa-compute_pgm_rsrc3-gfx12-table`.5333 415:384 4 bytes COMPUTE_PGM_RSRC1 Compute Shader (CS)5334 program settings used by5335 CP to set up5336 ``COMPUTE_PGM_RSRC1``5337 configuration5338 register. See5339 :ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx12-table`.5340 447:416 4 bytes COMPUTE_PGM_RSRC2 Compute Shader (CS)5341 program settings used by5342 CP to set up5343 ``COMPUTE_PGM_RSRC2``5344 configuration5345 register. See5346 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx12-table`.5347 458:448 7 bits *See separate bits below.* Enable the setup of the5348 SGPR user data registers5349 (see5350 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`).5351 5352 The total number of SGPR5353 user data registers5354 requested must not exceed5355 16 and match value in5356 ``compute_pgm_rsrc2.user_sgpr.user_sgpr_count``.5357 Any requests beyond 165358 will be ignored.5359 >448 1 bit ENABLE_SGPR_PRIVATE_SEGMENT If the *Target Properties*5360 _BUFFER column of5361 :ref:`amdgpu-processor-table`5362 specifies *Architected flat5363 scratch* then not supported5364 and must be 0,5365 >449 1 bit ENABLE_SGPR_DISPATCH_PTR5366 >450 1 bit ENABLE_SGPR_QUEUE_PTR5367 >451 1 bit ENABLE_SGPR_KERNARG_SEGMENT_PTR5368 >452 1 bit ENABLE_SGPR_DISPATCH_ID5369 >453 1 bit ENABLE_SGPR_FLAT_SCRATCH_INIT If the *Target Properties*5370 column of5371 :ref:`amdgpu-processor-table`5372 specifies *Architected flat5373 scratch* then not supported5374 and must be 0,5375 >454 1 bit ENABLE_SGPR_PRIVATE_SEGMENT5376 _SIZE5377 457:455 3 bits Reserved, must be 0.5378 458 1 bit ENABLE_WAVEFRONT_SIZE32 GFX6-GFX95379 Reserved, must be 0.5380 GFX10-GFX115381 - If 0 execute in5382 wavefront size 64 mode.5383 - If 1 execute in5384 native wavefront size5385 32 mode.5386 459 1 bit USES_DYNAMIC_STACK Indicates if the generated5387 machine code is using a5388 dynamically sized stack.5389 This is only set in code5390 object v5 and later.5391 463:460 4 bits Reserved, must be 0.5392 470:464 7 bits KERNARG_PRELOAD_SPEC_LENGTH GFX6-GFX95393 - Reserved, must be 0.5394 GFX90A, GFX9425395 - The number of dwords from5396 the kernarg segment to preload5397 into User SGPRs before kernel5398 execution. (see5399 :ref:`amdgpu-amdhsa-kernarg-preload`).5400 479:471 9 bits KERNARG_PRELOAD_SPEC_OFFSET GFX6-GFX95401 - Reserved, must be 0.5402 GFX90A, GFX9425403 - An offset in dwords into the5404 kernarg segment to begin5405 preloading data into User5406 SGPRs. (see5407 :ref:`amdgpu-amdhsa-kernarg-preload`).5408 511:480 4 bytes Reserved, must be 0.5409 512 **Total size 64 bytes.**5410 ======= ====================================================================5411 5412..5413 5414 .. table:: compute_pgm_rsrc1 for GFX6-GFX125415 :name: amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx12-table5416 5417 ======= ======= =============================== ===========================================================================5418 Bits Size Field Name Description5419 ======= ======= =============================== ===========================================================================5420 5:0 6 bits GRANULATED_WORKITEM_VGPR_COUNT Number of vector register5421 blocks used by each work-item;5422 granularity is device5423 specific:5424 5425 GFX6-GFX95426 - vgprs_used 0..2565427 - max(0, ceil(vgprs_used / 4) - 1)5428 GFX90A, GFX9425429 - vgprs_used 0..5125430 - vgprs_used = align(arch_vgprs, 4)5431 + acc_vgprs5432 - max(0, ceil(vgprs_used / 8) - 1)5433 GFX10-GFX12 (wavefront size 64)5434 - max_vgpr 1..2565435 - max(0, ceil(vgprs_used / 4) - 1)5436 GFX10-GFX12 (wavefront size 32)5437 - max_vgpr 1..2565438 - max(0, ceil(vgprs_used / 8) - 1)5439 GFX125X (wavefront size 32)5440 - max_vgpr 1..10245441 - max(0, ceil(vgprs_used / 16) - 1)5442 5443 Where vgprs_used is defined5444 as the highest VGPR number5445 explicitly referenced plus5446 one.5447 5448 Used by CP to set up5449 ``COMPUTE_PGM_RSRC1.VGPRS``.5450 5451 The5452 :ref:`amdgpu-assembler`5453 calculates this5454 automatically for the5455 selected processor from5456 values provided to the5457 `.amdhsa_kernel` directive5458 by the5459 `.amdhsa_next_free_vgpr`5460 nested directive (see5461 :ref:`amdhsa-kernel-directives-table`).5462 9:6 4 bits GRANULATED_WAVEFRONT_SGPR_COUNT Number of scalar register5463 blocks used by a wavefront;5464 granularity is device5465 specific:5466 5467 GFX6-GFX85468 - sgprs_used 0..1125469 - max(0, ceil(sgprs_used / 8) - 1)5470 GFX95471 - sgprs_used 0..1125472 - 2 * max(0, ceil(sgprs_used / 16) - 1)5473 GFX10-GFX125474 Reserved, must be 0.5475 (128 SGPRs always5476 allocated.)5477 5478 Where sgprs_used is5479 defined as the highest5480 SGPR number explicitly5481 referenced plus one, plus5482 a target specific number5483 of additional special5484 SGPRs for VCC,5485 FLAT_SCRATCH (GFX7+) and5486 XNACK_MASK (GFX8+), and5487 any additional5488 target specific5489 limitations. It does not5490 include the 16 SGPRs added5491 if a trap handler is5492 enabled.5493 5494 The target specific5495 limitations and special5496 SGPR layout are defined in5497 the hardware5498 documentation, which can5499 be found in the5500 :ref:`amdgpu-processors`5501 table.5502 5503 Used by CP to set up5504 ``COMPUTE_PGM_RSRC1.SGPRS``.5505 5506 The5507 :ref:`amdgpu-assembler`5508 calculates this5509 automatically for the5510 selected processor from5511 values provided to the5512 `.amdhsa_kernel` directive5513 by the5514 `.amdhsa_next_free_sgpr`5515 and `.amdhsa_reserve_*`5516 nested directives (see5517 :ref:`amdhsa-kernel-directives-table`).5518 11:10 2 bits PRIORITY Must be 0.5519 5520 Start executing wavefront5521 at the specified priority.5522 5523 CP is responsible for5524 filling in5525 ``COMPUTE_PGM_RSRC1.PRIORITY``.5526 13:12 2 bits FLOAT_ROUND_MODE_32 Wavefront starts execution5527 with specified rounding5528 mode for single (32-bit)5529 floating point5530 precision floating point5531 operations.5532 5533 Floating point rounding5534 mode values are defined in5535 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`.5536 5537 Used by CP to set up5538 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``.5539 15:14 2 bits FLOAT_ROUND_MODE_16_64 Wavefront starts execution5540 with specified rounding5541 denorm mode for half/double (165542 and 64-bit) floating point5543 precision floating point5544 operations.5545 5546 Floating point rounding5547 mode values are defined in5548 :ref:`amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table`.5549 5550 Used by CP to set up5551 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``.5552 17:16 2 bits FLOAT_DENORM_MODE_32 Wavefront starts execution5553 with specified denorm mode5554 for single (325555 bit) floating point5556 precision floating point5557 operations.5558 5559 Floating point denorm mode5560 values are defined in5561 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`.5562 5563 Used by CP to set up5564 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``.5565 19:18 2 bits FLOAT_DENORM_MODE_16_64 Wavefront starts execution5566 with specified denorm mode5567 for half/double (165568 and 64-bit) floating point5569 precision floating point5570 operations.5571 5572 Floating point denorm mode5573 values are defined in5574 :ref:`amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table`.5575 5576 Used by CP to set up5577 ``COMPUTE_PGM_RSRC1.FLOAT_MODE``.5578 20 1 bit PRIV Must be 0.5579 5580 Start executing wavefront5581 in privilege trap handler5582 mode.5583 5584 CP is responsible for5585 filling in5586 ``COMPUTE_PGM_RSRC1.PRIV``.5587 21 1 bit ENABLE_DX10_CLAMP GFX9-GFX115588 Wavefront starts execution5589 with DX10 clamp mode5590 enabled. Used by the vector5591 ALU to force DX10 style5592 treatment of NaN's (when5593 set, clamp NaN to zero,5594 otherwise pass NaN5595 through).5596 5597 Used by CP to set up5598 ``COMPUTE_PGM_RSRC1.DX10_CLAMP``.5599 WG_RR_EN GFX125600 If 1, wavefronts are scheduled5601 in a round-robin fashion with5602 respect to the other wavefronts5603 of the SIMD. Otherwise, wavefronts5604 are scheduled in oldest age order.5605 5606 CP is responsible for filling in5607 ``COMPUTE_PGM_RSRC1.WG_RR_EN``.5608 22 1 bit DEBUG_MODE Must be 0.5609 5610 Start executing wavefront5611 in single step mode.5612 5613 CP is responsible for5614 filling in5615 ``COMPUTE_PGM_RSRC1.DEBUG_MODE``.5616 23 1 bit ENABLE_IEEE_MODE GFX9-GFX115617 Wavefront starts execution5618 with IEEE mode5619 enabled. Floating point5620 opcodes that support5621 exception flag gathering5622 will quiet and propagate5623 signaling-NaN inputs per5624 IEEE 754-2008. Min_dx10 and5625 max_dx10 become IEEE5626 754-2008 compliant due to5627 signaling-NaN propagation5628 and quieting.5629 5630 Used by CP to set up5631 ``COMPUTE_PGM_RSRC1.IEEE_MODE``.5632 DISABLE_PERF GFX125633 Reserved. Must be 0.5634 24 1 bit BULKY Must be 0.5635 5636 Only one work-group allowed5637 to execute on a compute5638 unit.5639 5640 CP is responsible for5641 filling in5642 ``COMPUTE_PGM_RSRC1.BULKY``.5643 25 1 bit CDBG_USER Must be 0.5644 5645 Flag that can be used to5646 control debugging code.5647 5648 CP is responsible for5649 filling in5650 ``COMPUTE_PGM_RSRC1.CDBG_USER``.5651 26 1 bit FP16_OVFL GFX6-GFX85652 Reserved, must be 0.5653 GFX9-GFX125654 Wavefront starts execution5655 with specified fp16 overflow5656 mode.5657 5658 - If 0, fp16 overflow generates5659 +/-INF values.5660 - If 1, fp16 overflow that is the5661 result of an +/-INF input value5662 or divide by 0 produces a +/-INF,5663 otherwise clamps computed5664 overflow to +/-MAX_FP16 as5665 appropriate.5666 5667 Used by CP to set up5668 ``COMPUTE_PGM_RSRC1.FP16_OVFL``.5669 27 1 bit RESERVED GFX6-GFX120*5670 Reserved, must be 0.5671 FLAT_SCRATCH_IS_NV GFX125*5672 0 - Use the NV ISA as indication5673 that scratch is NV. 1 - Force5674 scratch to NV = 1, even if5675 ISA.NV == 0 if the address falls5676 into scratch space (not global).5677 This allows global.NV = 0 and5678 scratch.NV = 1 for flat ops. Other5679 threads use the ISA bit value.5680 5681 Used by CP to set up5682 ``COMPUTE_PGM_RSRC1.FLAT_SCRATCH_IS_NV``.5683 28 1 bit RESERVED Reserved, must be 0.5684 29 1 bit WGP_MODE GFX6-GFX95685 Reserved, must be 0.5686 GFX10-GFX125687 - If 0 execute work-groups in5688 CU wavefront execution mode.5689 - If 1 execute work-groups on5690 in WGP wavefront execution mode.5691 5692 See :ref:`amdgpu-amdhsa-memory-model`.5693 5694 Used by CP to set up5695 ``COMPUTE_PGM_RSRC1.WGP_MODE``.5696 30 1 bit MEM_ORDERED GFX6-GFX95697 Reserved, must be 0.5698 GFX10-GFX125699 Controls the behavior of the5700 s_waitcnt's vmcnt and vscnt5701 counters.5702 5703 - If 0 vmcnt reports completion5704 of load and atomic with return5705 out of order with sample5706 instructions, and the vscnt5707 reports the completion of5708 store and atomic without5709 return in order.5710 - If 1 vmcnt reports completion5711 of load, atomic with return5712 and sample instructions in5713 order, and the vscnt reports5714 the completion of store and5715 atomic without return in order.5716 5717 Used by CP to set up5718 ``COMPUTE_PGM_RSRC1.MEM_ORDERED``.5719 31 1 bit FWD_PROGRESS GFX6-GFX95720 Reserved, must be 0.5721 GFX10-GFX125722 - If 0 execute SIMD wavefronts5723 using oldest first policy.5724 - If 1 execute SIMD wavefronts to5725 ensure wavefronts will make some5726 forward progress.5727 5728 Used by CP to set up5729 ``COMPUTE_PGM_RSRC1.FWD_PROGRESS``.5730 32 **Total size 4 bytes**5731 ======= ===================================================================================================================5732 5733..5734 5735 .. table:: compute_pgm_rsrc2 for GFX6-GFX125736 :name: amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx12-table5737 5738 ======= ======= =============================== ===========================================================================5739 Bits Size Field Name Description5740 ======= ======= =============================== ===========================================================================5741 0 1 bit ENABLE_PRIVATE_SEGMENT * Enable the setup of the5742 private segment.5743 * If the *Target Properties*5744 column of5745 :ref:`amdgpu-processor-table`5746 does not specify5747 *Architected flat5748 scratch* then enable the5749 setup of the SGPR5750 wavefront scratch offset5751 system register (see5752 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`).5753 * If the *Target Properties*5754 column of5755 :ref:`amdgpu-processor-table`5756 specifies *Architected5757 flat scratch* then enable5758 the setup of the5759 FLAT_SCRATCH register5760 pair (see5761 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`).5762 5763 Used by CP to set up5764 ``COMPUTE_PGM_RSRC2.SCRATCH_EN``.5765 5:1 5 bits USER_SGPR_COUNT GFX6-GFX120*5766 The total number of SGPR5767 user data5768 registers requested. This5769 number must be greater than5770 or equal to the number of user5771 data registers enabled.5772 5773 Used by CP to set up5774 ``COMPUTE_PGM_RSRC2.USER_SGPR``.5775 6 1 bit ENABLE_TRAP_HANDLER GFX6-GFX115776 Must be 0.5777 5778 This bit represents5779 ``COMPUTE_PGM_RSRC2.TRAP_PRESENT``,5780 which is set by the CP if5781 the runtime has installed a5782 trap handler.5783 ENABLE_DYNAMIC_VGPR GFX120*5784 Enables dynamic VGPR mode, where5785 each wave allocates one VGPR chunk5786 at launch and can request for5787 additional space to use during5788 execution in SQ.5789 5790 Used by CP to set up5791 ``COMPUTE_PGM_RSRC2.DYNAMIC_VGPR``.5792 6:1 6 bits USER_SGPR_COUNT GFX125*5793 The total number of SGPR5794 user data5795 registers requested. This5796 number must be greater than5797 or equal to the number of user5798 data registers enabled.5799 5800 Used by CP to set up5801 ``COMPUTE_PGM_RSRC2.USER_SGPR``.5802 7 1 bit ENABLE_SGPR_WORKGROUP_ID_X Enable the setup of the5803 system SGPR register for5804 the work-group id in the X5805 dimension (see5806 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`).5807 5808 Used by CP to set up5809 ``COMPUTE_PGM_RSRC2.TGID_X_EN``.5810 8 1 bit ENABLE_SGPR_WORKGROUP_ID_Y Enable the setup of the5811 system SGPR register for5812 the work-group id in the Y5813 dimension (see5814 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`).5815 5816 Used by CP to set up5817 ``COMPUTE_PGM_RSRC2.TGID_Y_EN``.5818 9 1 bit ENABLE_SGPR_WORKGROUP_ID_Z Enable the setup of the5819 system SGPR register for5820 the work-group id in the Z5821 dimension (see5822 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`).5823 5824 Used by CP to set up5825 ``COMPUTE_PGM_RSRC2.TGID_Z_EN``.5826 10 1 bit ENABLE_SGPR_WORKGROUP_INFO Enable the setup of the5827 system SGPR register for5828 work-group information (see5829 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`).5830 5831 Used by CP to set up5832 ``COMPUTE_PGM_RSRC2.TGID_SIZE_EN``.5833 12:11 2 bits ENABLE_VGPR_WORKITEM_ID Enable the setup of the5834 VGPR system registers used5835 for the work-item ID.5836 :ref:`amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table`5837 defines the values.5838 5839 Used by CP to set up5840 ``COMPUTE_PGM_RSRC2.TIDIG_CMP_CNT``.5841 13 1 bit ENABLE_EXCEPTION_ADDRESS_WATCH Must be 0.5842 5843 Wavefront starts execution5844 with address watch5845 exceptions enabled which5846 are generated when L1 has5847 witnessed a thread access5848 an *address of5849 interest*.5850 5851 CP is responsible for5852 filling in the address5853 watch bit in5854 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB``5855 according to what the5856 runtime requests.5857 14 1 bit ENABLE_EXCEPTION_MEMORY Must be 0.5858 5859 Wavefront starts execution5860 with memory violation5861 exceptions5862 enabled which are generated5863 when a memory violation has5864 occurred for this wavefront from5865 L1 or LDS5866 (write-to-read-only-memory,5867 mis-aligned atomic, LDS5868 address out of range,5869 illegal address, etc.).5870 5871 CP sets the memory5872 violation bit in5873 ``COMPUTE_PGM_RSRC2.EXCP_EN_MSB``5874 according to what the5875 runtime requests.5876 23:15 9 bits GRANULATED_LDS_SIZE Must be 0.5877 5878 CP uses the rounded value5879 from the dispatch packet,5880 not this value, as the5881 dispatch may contain5882 dynamically allocated group5883 segment memory. CP writes5884 directly to5885 ``COMPUTE_PGM_RSRC2.LDS_SIZE``.5886 5887 Amount of group segment5888 (LDS) to allocate for each5889 work-group. Granularity is5890 device specific:5891 5892 GFX65893 roundup(lds-size / (64 * 4))5894 GFX7-GFX125895 roundup(lds-size / (128 * 4))5896 GFX9505897 roundup(lds-size / (320 * 4))5898 GFX125*5899 roundup(lds-size / (512 * 4))5900 5901 24 1 bit ENABLE_EXCEPTION_IEEE_754_FP Wavefront starts execution5902 _INVALID_OPERATION with specified exceptions5903 enabled.5904 5905 Used by CP to set up5906 ``COMPUTE_PGM_RSRC2.EXCP_EN``5907 (set from bits 0..6).5908 5909 IEEE 754 FP Invalid5910 Operation5911 25 1 bit ENABLE_EXCEPTION_FP_DENORMAL FP Denormal one or more5912 _SOURCE input operands is a5913 denormal number5914 26 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Division by5915 _DIVISION_BY_ZERO Zero5916 27 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP FP Overflow5917 _OVERFLOW5918 28 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Underflow5919 _UNDERFLOW5920 29 1 bit ENABLE_EXCEPTION_IEEE_754_FP IEEE 754 FP Inexact5921 _INEXACT5922 30 1 bit ENABLE_EXCEPTION_INT_DIVIDE_BY Integer Division by Zero5923 _ZERO (rcp_iflag_f32 instruction5924 only)5925 31 1 bit RESERVED Reserved, must be 0.5926 32 **Total size 4 bytes.**5927 ======= ===================================================================================================================5928 5929..5930 5931 .. table:: compute_pgm_rsrc3 for GFX90A, GFX9425932 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx90a-table5933 5934 ======= ======= =============================== ===========================================================================5935 Bits Size Field Name Description5936 ======= ======= =============================== ===========================================================================5937 5:0 6 bits ACCUM_OFFSET Offset of a first AccVGPR in the unified register file. Granularity 4.5938 Value 0-63. 0 - accum-offset = 4, 1 - accum-offset = 8, ...,5939 63 - accum-offset = 256.5940 15:6 10 Reserved, must be 0.5941 bits5942 16 1 bit TG_SPLIT - If 0 the waves of a work-group are5943 launched in the same CU.5944 - If 1 the waves of a work-group can be5945 launched in different CUs. The waves5946 cannot use S_BARRIER or LDS.5947 31:17 15 Reserved, must be 0.5948 bits5949 32 **Total size 4 bytes.**5950 ======= ===================================================================================================================5951 5952..5953 5954 .. table:: compute_pgm_rsrc3 for GFX10-GFX115955 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx10-gfx11-table5956 5957 ======= ======= =============================== ===========================================================================5958 Bits Size Field Name Description5959 ======= ======= =============================== ===========================================================================5960 3:0 4 bits SHARED_VGPR_COUNT Number of shared VGPR blocks when executing in subvector mode. For5961 wavefront size 64 the value is 0-15, representing 0-120 VGPRs (granularity5962 of 8), such that (compute_pgm_rsrc1.vgprs +1)*4 + shared_vgpr_count*8 does5963 not exceed 256. For wavefront size 32 shared_vgpr_count must be 0.5964 9:4 6 bits INST_PREF_SIZE GFX105965 Reserved, must be 0.5966 GFX115967 Number of instruction bytes to prefetch, starting at the kernel's entry5968 point instruction, before wavefront starts execution. The value is 0..635969 with a granularity of 128 bytes.5970 10 1 bit TRAP_ON_START GFX105971 Reserved, must be 0.5972 GFX115973 Must be 0.5974 5975 If 1, wavefront starts execution by trapping into the trap handler.5976 5977 CP is responsible for filling in the trap on start bit in5978 ``COMPUTE_PGM_RSRC3.TRAP_ON_START`` according to what the runtime5979 requests.5980 11 1 bit TRAP_ON_END GFX105981 Reserved, must be 0.5982 GFX115983 Must be 0.5984 5985 If 1, wavefront execution terminates by trapping into the trap handler.5986 5987 CP is responsible for filling in the trap on end bit in5988 ``COMPUTE_PGM_RSRC3.TRAP_ON_END`` according to what the runtime requests.5989 30:12 19 bits Reserved, must be 0.5990 31 1 bit IMAGE_OP GFX105991 Reserved, must be 0.5992 GFX115993 If 1, the kernel execution contains image instructions. If executed as5994 part of a graphics pipeline, image read instructions will stall waiting5995 for any necessary ``WAIT_SYNC`` fence to be performed in order to5996 indicate that earlier pipeline stages have completed writing to the5997 image.5998 5999 Not used for compute kernels that are not part of a graphics pipeline and6000 must be 0.6001 32 **Total size 4 bytes.**6002 ======= ===================================================================================================================6003 6004..6005 6006 .. table:: compute_pgm_rsrc3 for GFX126007 :name: amdgpu-amdhsa-compute_pgm_rsrc3-gfx12-table6008 6009 ======= ======= =============================== ===========================================================================6010 Bits Size Field Name Description6011 ======= ======= =============================== ===========================================================================6012 3:0 4 bits RESERVED Reserved, must be 0.6013 11:4 8 bits INST_PREF_SIZE Number of instruction bytes to prefetch, starting at the kernel's entry6014 point instruction, before wavefront starts execution. The value is 0..2556015 with a granularity of 128 bytes.6016 12 1 bit RESERVED Reserved, must be 0.6017 13 1 bit GLG_EN If 1, group launch guarantee will be enabled for this dispatch6018 16:14 3 bits RESERVED GFX120*6019 Reserved, must be 0.6020 NAMED_BAR_CNT GFX125*6021 Number of named barriers to alloc for each workgroup, in granularity of6022 4. Range is from 0-4 allocating 0, 4, 8, 12, 16.6023 17 1 bit RESERVED GFX120*6024 Reserved, must be 0.6025 ENABLE_DYNAMIC_VGPR GFX125*6026 Enables dynamic VGPR mode, where each wave allocates one VGPR chunk6027 at launch and can request for additional space to use during6028 execution in SQ.6029 6030 Used by CP to set up ``COMPUTE_PGM_RSRC3.DYNAMIC_VGPR``.6031 20:18 3 bits RESERVED GFX120*6032 Reserved, must be 0.6033 TCP_SPLIT GFX125*6034 Desired LDS/VC split of TCP. 0: no preference 1: LDS=0, VC=448kB6035 2: LDS=64kB, VC=384kB 3: LDS=128kB, VC=320kB 4: LDS=192kB, VC=256kB6036 5: LDS=256kB, VC=192kB 6: LDS=320kB, VC=128kB 7: LDS=384kB, VC=64kB6037 21 1 bit RESERVED GFX120*6038 Reserved, must be 0.6039 ENABLE_DIDT_THROTTLE GFX125*6040 Enable DIDT throttling for all ACE pipes6041 30:22 9 bits RESERVED Reserved, must be 0.6042 31 1 bit IMAGE_OP If 1, the kernel execution contains image instructions. If executed as6043 part of a graphics pipeline, image read instructions will stall waiting6044 for any necessary ``WAIT_SYNC`` fence to be performed in order to6045 indicate that earlier pipeline stages have completed writing to the6046 image.6047 6048 Not used for compute kernels that are not part of a graphics pipeline and6049 must be 0.6050 32 **Total size 4 bytes.**6051 ======= ===================================================================================================================6052 6053..6054 6055 .. table:: Floating Point Rounding Mode Enumeration Values6056 :name: amdgpu-amdhsa-floating-point-rounding-mode-enumeration-values-table6057 6058 ====================================== ===== ==============================6059 Enumeration Name Value Description6060 ====================================== ===== ==============================6061 FLOAT_ROUND_MODE_NEAR_EVEN 0 Round Ties To Even6062 FLOAT_ROUND_MODE_PLUS_INFINITY 1 Round Toward +infinity6063 FLOAT_ROUND_MODE_MINUS_INFINITY 2 Round Toward -infinity6064 FLOAT_ROUND_MODE_ZERO 3 Round Toward 06065 ====================================== ===== ==============================6066 6067 6068 .. table:: Extended FLT_ROUNDS Enumeration Values6069 :name: amdgpu-rounding-mode-enumeration-values-table6070 6071 +------------------------+---------------+-------------------+--------------------+----------+6072 | | F32 NEAR_EVEN | F32 PLUS_INFINITY | F32 MINUS_INFINITY | F32 ZERO |6073 +------------------------+---------------+-------------------+--------------------+----------+6074 | F64/F16 NEAR_EVEN | 1 | 11 | 14 | 17 |6075 +------------------------+---------------+-------------------+--------------------+----------+6076 | F64/F16 PLUS_INFINITY | 8 | 2 | 15 | 18 |6077 +------------------------+---------------+-------------------+--------------------+----------+6078 | F64/F16 MINUS_INFINITY | 9 | 12 | 3 | 19 |6079 +------------------------+---------------+-------------------+--------------------+----------+6080 | F64/F16 ZERO | 10 | 13 | 16 | 0 |6081 +------------------------+---------------+-------------------+--------------------+----------+6082 6083..6084 6085 .. table:: Floating Point Denorm Mode Enumeration Values6086 :name: amdgpu-amdhsa-floating-point-denorm-mode-enumeration-values-table6087 6088 ====================================== ===== ====================================6089 Enumeration Name Value Description6090 ====================================== ===== ====================================6091 FLOAT_DENORM_MODE_FLUSH_SRC_DST 0 Flush Source and Destination Denorms6092 FLOAT_DENORM_MODE_FLUSH_DST 1 Flush Output Denorms6093 FLOAT_DENORM_MODE_FLUSH_SRC 2 Flush Source Denorms6094 FLOAT_DENORM_MODE_FLUSH_NONE 3 No Flush6095 ====================================== ===== ====================================6096 6097 Denormal flushing is sign respecting, i.e., the behavior expected by6098 ``"denormal-fp-math"="preserve-sign"``. The behavior is undefined with6099 ``"denormal-fp-math"="positive-zero"``6100 6101..6102 6103 .. table:: System VGPR Work-Item ID Enumeration Values6104 :name: amdgpu-amdhsa-system-vgpr-work-item-id-enumeration-values-table6105 6106 ======================================== ===== ============================6107 Enumeration Name Value Description6108 ======================================== ===== ============================6109 SYSTEM_VGPR_WORKITEM_ID_X 0 Set work-item X dimension6110 ID.6111 SYSTEM_VGPR_WORKITEM_ID_X_Y 1 Set work-item X and Y6112 dimensions ID.6113 SYSTEM_VGPR_WORKITEM_ID_X_Y_Z 2 Set work-item X, Y and Z6114 dimensions ID.6115 SYSTEM_VGPR_WORKITEM_ID_UNDEFINED 3 Undefined.6116 ======================================== ===== ============================6117 6118.. _amdgpu-amdhsa-initial-kernel-execution-state:6119 6120Initial Kernel Execution State6121~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~6122 6123This section defines the register state that will be set up by the packet6124processor prior to the start of execution of every wavefront. This is limited by6125the constraints of the hardware controllers of CP/ADC/SPI.6126 6127The order of the SGPR registers is defined, but the compiler can specify which6128ones are actually setup in the kernel descriptor using the ``enable_sgpr_*`` bit6129fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used6130for enabled registers are dense starting at SGPR0: the first enabled register is6131SGPR0, the next enabled register is SGPR1 etc.; disabled registers do not have6132an SGPR number.6133 6134The initial SGPRs comprise up to 16 User SGPRs that are set by CP and apply to6135all wavefronts of the grid. It is possible to specify more than 16 User SGPRs6136using the ``enable_sgpr_*`` bit fields, in which case only the first 16 are6137actually initialized. These are then immediately followed by the System SGPRs6138that are set up by ADC/SPI and can have different values for each wavefront of6139the grid dispatch.6140 6141SGPR register initial state is defined in6142:ref:`amdgpu-amdhsa-sgpr-register-set-up-order-table`.6143 6144 .. table:: SGPR Register Set Up Order6145 :name: amdgpu-amdhsa-sgpr-register-set-up-order-table6146 6147 ========== ========================== ====== ==============================6148 SGPR Order Name Number Description6149 (kernel descriptor enable of6150 field) SGPRs6151 ========== ========================== ====== ==============================6152 First Private Segment Buffer 4 See6153 (enable_sgpr_private :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`.6154 _segment_buffer)6155 then Dispatch Ptr 2 64-bit address of AQL dispatch6156 (enable_sgpr_dispatch_ptr) packet for kernel dispatch6157 actually executing.6158 then Queue Ptr 2 64-bit address of amd_queue_t6159 (enable_sgpr_queue_ptr) object for AQL queue on which6160 the dispatch packet was6161 queued.6162 then Kernarg Segment Ptr 2 64-bit address of Kernarg6163 (enable_sgpr_kernarg segment. This is directly6164 _segment_ptr) copied from the6165 kernarg_address in the kernel6166 dispatch packet.6167 6168 Having CP load it once avoids6169 loading it at the beginning of6170 every wavefront.6171 then Dispatch Id 2 64-bit Dispatch ID of the6172 (enable_sgpr_dispatch_id) dispatch packet being6173 executed.6174 then Flat Scratch Init 2 See6175 (enable_sgpr_flat_scratch :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`.6176 _init)6177 then Private Segment Size 1 The 32-bit byte size of a6178 (enable_sgpr_private single work-item's memory6179 _segment_size) allocation. This is the6180 value from the kernel6181 dispatch packet Private6182 Segment Byte Size rounded up6183 by CP to a multiple of6184 DWORD.6185 6186 Having CP load it once avoids6187 loading it at the beginning of6188 every wavefront.6189 6190 This is not used for6191 GFX7-GFX8 since it is the same6192 value as the second SGPR of6193 Flat Scratch Init. However, it6194 may be needed for GFX9-GFX11 which6195 changes the meaning of the6196 Flat Scratch Init value.6197 then Preloaded Kernargs N/A See6198 (kernarg_preload_spec :ref:`amdgpu-amdhsa-kernarg-preload`.6199 _length)6200 then Work-Group Id X 1 32-bit work-group id in X6201 (enable_sgpr_workgroup_id dimension of grid for6202 _X) wavefront.6203 then Work-Group Id Y 1 32-bit work-group id in Y6204 (enable_sgpr_workgroup_id dimension of grid for6205 _Y) wavefront.6206 then Work-Group Id Z 1 32-bit work-group id in Z6207 (enable_sgpr_workgroup_id dimension of grid for6208 _Z) wavefront.6209 then Work-Group Info 1 {first_wavefront, 14'b0000,6210 (enable_sgpr_workgroup ordered_append_term[10:0],6211 _info) threadgroup_size_in_wavefronts[5:0]}6212 then Scratch Wavefront Offset 1 See6213 (enable_sgpr_private :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`.6214 _segment_wavefront_offset) and6215 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`.6216 ========== ========================== ====== ==============================6217 6218The order of the VGPR registers is defined, but the compiler can specify which6219ones are actually setup in the kernel descriptor using the ``enable_vgpr*`` bit6220fields (see :ref:`amdgpu-amdhsa-kernel-descriptor`). The register numbers used6221for enabled registers are dense starting at VGPR0: the first enabled register is6222VGPR0, the next enabled register is VGPR1 etc.; disabled registers do not have a6223VGPR number.6224 6225There are different methods used for the VGPR initial state:6226 6227* Unless the *Target Properties* column of :ref:`amdgpu-processor-table`6228 specifies otherwise, a separate VGPR register is used per work-item ID. The6229 VGPR register initial state for this method is defined in6230 :ref:`amdgpu-amdhsa-vgpr-register-set-up-order-for-unpacked-work-item-id-method-table`.6231* If *Target Properties* column of :ref:`amdgpu-processor-table`6232 specifies *Packed work-item IDs*, the initial value of VGPR0 register is used6233 for all work-item IDs. The register layout for this method is defined in6234 :ref:`amdgpu-amdhsa-register-layout-for-packed-work-item-id-method-table`.6235 6236 .. table:: VGPR Register Set Up Order for Unpacked Work-Item ID Method6237 :name: amdgpu-amdhsa-vgpr-register-set-up-order-for-unpacked-work-item-id-method-table6238 6239 ========== ========================== ====== ==============================6240 VGPR Order Name Number Description6241 (kernel descriptor enable of6242 field) VGPRs6243 ========== ========================== ====== ==============================6244 First Work-Item Id X 1 32-bit work-item id in X6245 (Always initialized) dimension of work-group for6246 wavefront lane.6247 then Work-Item Id Y 1 32-bit work-item id in Y6248 (enable_vgpr_workitem_id dimension of work-group for6249 > 0) wavefront lane.6250 then Work-Item Id Z 1 32-bit work-item id in Z6251 (enable_vgpr_workitem_id dimension of work-group for6252 > 1) wavefront lane.6253 ========== ========================== ====== ==============================6254 6255..6256 6257 .. table:: Register Layout for Packed Work-Item ID Method6258 :name: amdgpu-amdhsa-register-layout-for-packed-work-item-id-method-table6259 6260 ======= ======= ================ =========================================6261 Bits Size Field Name Description6262 ======= ======= ================ =========================================6263 0:9 10 bits Work-Item Id X Work-item id in X6264 dimension of work-group for6265 wavefront lane.6266 6267 Always initialized.6268 6269 10:19 10 bits Work-Item Id Y Work-item id in Y6270 dimension of work-group for6271 wavefront lane.6272 6273 Initialized if enable_vgpr_workitem_id >6274 0, otherwise set to 0.6275 20:29 10 bits Work-Item Id Z Work-item id in Z6276 dimension of work-group for6277 wavefront lane.6278 6279 Initialized if enable_vgpr_workitem_id >6280 1, otherwise set to 0.6281 30:31 2 bits Reserved, set to 0.6282 ======= ======= ================ =========================================6283 6284The setting of registers is done by GPU CP/ADC/SPI hardware as follows:6285 62861. SGPRs before the Work-Group Ids are set by CP using the 16 User Data6287 registers.62882. Work-group Id registers X, Y, Z are set by ADC which supports any6289 combination including none.62903. Scratch Wavefront Offset is set by SPI in a per wavefront basis which is why6291 its value cannot be included with the flat scratch init value which is per6292 queue (see :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`).62934. The VGPRs are set by SPI which only supports specifying either (X), (X, Y)6294 or (X, Y, Z).62955. Flat Scratch register pair initialization is described in6296 :ref:`amdgpu-amdhsa-kernel-prolog-flat-scratch`.6297 6298The global segment can be accessed either using buffer instructions (GFX6 which6299has V# 64-bit address support), flat instructions (GFX7-GFX11), or global6300instructions (GFX9-GFX11).6301 6302If buffer operations are used, then the compiler can generate a V# with the6303following properties:6304 6305* base address of 06306* no swizzle6307* ATC: 1 if IOMMU present (such as APU)6308* ptr64: 16309* MTYPE set to support memory coherence that matches the runtime (such as CC for6310 APU and NC for dGPU).6311 6312.. _amdgpu-amdhsa-kernarg-preload:6313 6314Preloaded Kernel Arguments6315++++++++++++++++++++++++++6316 6317On hardware that supports this feature, kernel arguments can be preloaded into6318User SGPRs, up to the maximum number of User SGPRs available. The allocation of6319Preload SGPRs occurs directly after the last enabled non-kernarg preload User6320SGPR. (See :ref:`amdgpu-amdhsa-initial-kernel-execution-state`)6321 6322The data preloaded is copied from the kernarg segment, the amount of data is6323determined by the value specified in the kernarg_preload_spec_length field of6324the kernel descriptor. This data is then loaded into consecutive User SGPRs. The6325number of SGPRs receiving preloaded kernarg data corresponds with the value6326given by kernarg_preload_spec_length. The preloading starts at the dword offset6327within the kernarg segment, which is specified by the6328kernarg_preload_spec_offset field.6329 6330If the kernarg_preload_spec_length is non-zero, the CP firmware will append an6331additional 256 bytes to the kernel_code_entry_byte_offset. This addition6332facilitates the incorporation of a prologue to the kernel entry to handle cases6333where code designed for kernarg preloading is executed on hardware equipped with6334incompatible firmware. If hardware has compatible firmware the 256 bytes at the6335start of the kernel entry will be skipped.6336 6337With code object V5 and later, hidden kernel arguments that are normally6338accessed through the Implicit Argument Ptr, may be preloaded into User SGPRs.6339These arguments are added to the kernel function signature and are marked with6340the attributes "inreg" and "amdgpu-hidden-argument". (See6341:ref:`amdgpu-llvm-ir-attributes-table`).6342 6343.. _amdgpu-amdhsa-kernel-prolog:6344 6345Kernel Prolog6346~~~~~~~~~~~~~6347 6348The compiler performs initialization in the kernel prologue depending on the6349target and information about things like stack usage in the kernel and called6350functions. Some of this initialization requires the compiler to request certain6351User and System SGPRs be present in the6352:ref:`amdgpu-amdhsa-initial-kernel-execution-state` via the6353:ref:`amdgpu-amdhsa-kernel-descriptor`.6354 6355.. _amdgpu-amdhsa-kernel-prolog-cfi:6356 6357CFI6358+++6359 63601. The CFI return address is undefined.6361 63622. The CFI CFA is defined using an expression which evaluates to a location6363 description that comprises one memory location description for the6364 ``DW_ASPACE_AMDGPU_private_lane`` address space address ``0``.6365 6366.. _amdgpu-amdhsa-kernel-prolog-m0:6367 6368M06369++6370 6371GFX6-GFX86372 The M0 register must be initialized with a value at least the total LDS size6373 if the kernel may access LDS via DS or flat operations. Total LDS size is6374 available in dispatch packet. For M0, it is also possible to use maximum6375 possible value of LDS for given target (0x7FFF for GFX6 and 0xFFFF for6376 GFX7-GFX8).6377GFX9 and later6378 The M0 register is not used for range checking LDS accesses and so does not6379 need to be initialized in the prolog.6380 6381.. _amdgpu-amdhsa-kernel-prolog-stack-pointer:6382 6383Stack Pointer6384+++++++++++++6385 6386If the kernel has function calls it must set up the ABI stack pointer described6387in :ref:`amdgpu-amdhsa-function-call-convention-non-kernel-functions` by setting6388SGPR32 to the unswizzled scratch offset of the address past the last local6389allocation.6390 6391.. _amdgpu-amdhsa-kernel-prolog-frame-pointer:6392 6393Frame Pointer6394+++++++++++++6395 6396If the kernel needs a frame pointer for the reasons defined in6397``SIFrameLowering`` then SGPR33 is used and is always set to ``0`` in the6398kernel prolog. On GFX12+, when dynamic VGPRs are enabled, the prologue will6399check if the kernel is running on a compute queue, and if so it will reserve6400some scratch space for any dynamic VGPRs that might need to be saved by the6401CWSR trap handler. In this case, the frame pointer will be initialized to6402a suitably aligned offset above this reserved area. If a frame pointer is not6403required then all uses of the frame pointer are replaced with immediate ``0``6404offsets.6405 6406.. _amdgpu-amdhsa-kernel-prolog-flat-scratch:6407 6408Flat Scratch6409++++++++++++6410 6411There are different methods used for initializing flat scratch:6412 6413* If the *Target Properties* column of :ref:`amdgpu-processor-table`6414 specifies *Does not support generic address space*:6415 6416 Flat scratch is not supported and there is no flat scratch register pair.6417 6418* If the *Target Properties* column of :ref:`amdgpu-processor-table`6419 specifies *Offset flat scratch*:6420 6421 If the kernel or any function it calls may use flat operations to access6422 scratch memory, the prolog code must set up the FLAT_SCRATCH register pair6423 (FLAT_SCRATCH_LO/FLAT_SCRATCH_HI). Initialization uses Flat Scratch Init and6424 Scratch Wavefront Offset SGPR registers (see6425 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`):6426 6427 1. The low word of Flat Scratch Init is the 32-bit byte offset from6428 ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to the base of scratch backing memory6429 being managed by SPI for the queue executing the kernel dispatch. This is6430 the same value used in the Scratch Segment Buffer V# base address.6431 6432 CP obtains this from the runtime. (The Scratch Segment Buffer base address6433 is ``SH_HIDDEN_PRIVATE_BASE_VIMID`` plus this offset.)6434 6435 The prolog must add the value of Scratch Wavefront Offset to get the6436 wavefront's byte scratch backing memory offset from6437 ``SH_HIDDEN_PRIVATE_BASE_VIMID``.6438 6439 The Scratch Wavefront Offset must also be used as an offset with Private6440 segment address when using the Scratch Segment Buffer.6441 6442 Since FLAT_SCRATCH_LO is in units of 256 bytes, the offset must be right6443 shifted by 8 before moving into FLAT_SCRATCH_HI.6444 6445 FLAT_SCRATCH_HI corresponds to SGPRn-4 on GFX7, and SGPRn-6 on GFX8 (where6446 SGPRn is the highest numbered SGPR allocated to the wavefront).6447 FLAT_SCRATCH_HI is multiplied by 256 (as it is in units of 256 bytes) and6448 added to ``SH_HIDDEN_PRIVATE_BASE_VIMID`` to calculate the per wavefront6449 FLAT SCRATCH BASE in flat memory instructions that access the scratch6450 aperture.6451 2. The second word of Flat Scratch Init is 32-bit byte size of a single6452 work-items scratch memory usage.6453 6454 CP obtains this from the runtime, and it is always a multiple of DWORD. CP6455 checks that the value in the kernel dispatch packet Private Segment Byte6456 Size is not larger and requests the runtime to increase the queue's scratch6457 size if necessary.6458 6459 CP directly loads from the kernel dispatch packet Private Segment Byte Size6460 field and rounds up to a multiple of DWORD. Having CP load it once avoids6461 loading it at the beginning of every wavefront.6462 6463 The kernel prolog code must move it to FLAT_SCRATCH_LO which is SGPRn-3 on6464 GFX7 and SGPRn-5 on GFX8. FLAT_SCRATCH_LO is used as the FLAT SCRATCH SIZE6465 in flat memory instructions.6466 6467* If the *Target Properties* column of :ref:`amdgpu-processor-table`6468 specifies *Absolute flat scratch*:6469 6470 If the kernel or any function it calls may use flat operations to access6471 scratch memory, the prolog code must set up the FLAT_SCRATCH register pair6472 (FLAT_SCRATCH_LO/FLAT_SCRATCH_HI which are in SGPRn-4/SGPRn-3). Initialization6473 uses Flat Scratch Init and Scratch Wavefront Offset SGPR registers (see6474 :ref:`amdgpu-amdhsa-initial-kernel-execution-state`):6475 6476 The Flat Scratch Init is the 64-bit address of the base of scratch backing6477 memory being managed by SPI for the queue executing the kernel dispatch.6478 6479 CP obtains this from the runtime.6480 6481 The kernel prolog must add the value of the wave's Scratch Wavefront Offset6482 and move the result as a 64-bit value to the FLAT_SCRATCH SGPR register pair6483 which is SGPRn-6 and SGPRn-5. It is used as the FLAT SCRATCH BASE in flat6484 memory instructions.6485 6486 The Scratch Wavefront Offset must also be used as an offset with Private6487 segment address when using the Scratch Segment Buffer (see6488 :ref:`amdgpu-amdhsa-kernel-prolog-private-segment-buffer`).6489 6490* If the *Target Properties* column of :ref:`amdgpu-processor-table`6491 specifies *Architected flat scratch*:6492 6493 If ENABLE_PRIVATE_SEGMENT is enabled in6494 :ref:`amdgpu-amdhsa-compute_pgm_rsrc2-gfx6-gfx12-table` then the FLAT_SCRATCH6495 register pair will be initialized to the 64-bit address of the base of scratch6496 backing memory being managed by SPI for the queue executing the kernel6497 dispatch plus the value of the wave's Scratch Wavefront Offset for use as the6498 flat scratch base in flat memory instructions.6499 6500.. _amdgpu-amdhsa-kernel-prolog-private-segment-buffer:6501 6502Private Segment Buffer6503++++++++++++++++++++++6504 6505If the *Target Properties* column of :ref:`amdgpu-processor-table` specifies6506*Architected flat scratch* then a Private Segment Buffer is not supported.6507Instead the flat SCRATCH instructions are used.6508 6509Otherwise, Private Segment Buffer SGPR register is used to initialize 4 SGPRs6510that are used as a V# to access scratch. CP uses the value provided by the6511runtime. It is used, together with Scratch Wavefront Offset as an offset, to6512access the private memory space using a segment address. See6513:ref:`amdgpu-amdhsa-initial-kernel-execution-state`.6514 6515The scratch V# is a four-aligned SGPR and always selected for the kernel as6516follows:6517 6518 - If it is known during instruction selection that there is stack usage,6519 SGPR0-3 is reserved for use as the scratch V#. Stack usage is assumed if6520 optimizations are disabled (``-O0``), if stack objects already exist (for6521 locals, etc.), or if there are any function calls.6522 6523 - Otherwise, four high numbered SGPRs beginning at a four-aligned SGPR index6524 are reserved for the tentative scratch V#. These will be used if it is6525 determined that spilling is needed.6526 6527 - If no use is made of the tentative scratch V#, then it is unreserved,6528 and the register count is determined ignoring it.6529 - If use is made of the tentative scratch V#, then its register numbers6530 are shifted to the first four-aligned SGPR index after the highest one6531 allocated by the register allocator, and all uses are updated. The6532 register count includes them in the shifted location.6533 - In either case, if the processor has the SGPR allocation bug, the6534 tentative allocation is not shifted or unreserved in order to ensure6535 the register count is higher to workaround the bug.6536 6537 .. note::6538 6539 This approach of using a tentative scratch V# and shifting the register6540 numbers if used avoids having to perform register allocation a second6541 time if the tentative V# is eliminated. This is more efficient and6542 avoids the problem that the second register allocation may perform6543 spilling which will fail as there is no longer a scratch V#.6544 6545When the kernel prolog code is being emitted it is known whether the scratch V#6546described above is actually used. If it is, the prolog code must set it up by6547copying the Private Segment Buffer to the scratch V# registers and then adding6548the Private Segment Wavefront Offset to the queue base address in the V#. The6549result is a V# with a base address pointing to the beginning of the wavefront6550scratch backing memory.6551 6552The Private Segment Buffer is always requested, but the Private Segment6553Wavefront Offset is only requested if it is used (see6554:ref:`amdgpu-amdhsa-initial-kernel-execution-state`).6555 6556.. _amdgpu-amdhsa-memory-model:6557 6558Memory Model6559~~~~~~~~~~~~6560 6561This section describes the mapping of the LLVM memory model onto AMDGPU machine6562code (see :ref:`memmodel`).6563 6564The AMDGPU backend supports the memory synchronization scopes specified in6565:ref:`amdgpu-memory-scopes`.6566 6567The code sequences used to implement the memory model specify the order of6568instructions that a single thread must execute. The ``s_waitcnt`` and cache6569management instructions such as ``buffer_wbinvl1_vol`` are defined with respect6570to other memory instructions executed by the same thread. This allows them to be6571moved earlier or later which can allow them to be combined with other instances6572of the same instruction, or hoisted/sunk out of loops to improve performance.6573Only the instructions related to the memory model are given; additional6574``s_waitcnt`` instructions are required to ensure registers are defined before6575being used. These may be able to be combined with the memory model ``s_waitcnt``6576instructions as described above.6577 6578The AMDGPU backend supports the following memory models:6579 6580 HSA Memory Model [HSA]_6581 The HSA memory model uses a single happens-before relation for all address6582 spaces (see :ref:`amdgpu-address-spaces`).6583 OpenCL Memory Model [OpenCL]_6584 The OpenCL memory model which has separate happens-before relations for the6585 global and local address spaces. Only a fence specifying both global and6586 local address space, and seq_cst instructions join the relationships. Since6587 the LLVM ``memfence`` instruction does not allow an address space to be6588 specified the OpenCL fence has to conservatively assume both local and6589 global address space was specified. However, optimizations can often be6590 done to eliminate the additional ``s_waitcnt`` instructions when there are6591 no intervening memory instructions which access the corresponding address6592 space. The code sequences in the table indicate what can be omitted for the6593 OpenCL memory. The target triple environment is used to determine if the6594 source language is OpenCL (see :ref:`amdgpu-opencl`).6595 6596``ds/flat_load/store/atomic`` instructions to local memory are termed LDS6597operations.6598 6599``buffer/global/flat_load/store/atomic`` instructions to global memory are6600termed vector memory operations.6601 6602``global_load_lds`` or ``buffer/global_load`` instructions with the `lds` flag6603are LDS DMA loads. They interact with caches as if the loaded data were6604being loaded to registers and not to LDS, and so therefore support the same6605cache modifiers. They cannot be performed atomically. They implement volatile6606(via aux/cpol bit 31) and nontemporal (via metadata) as if they were loads6607from the global address space.6608 6609Private address space uses ``buffer_load/store`` using the scratch V#6610(GFX6-GFX8), or ``scratch_load/store`` (GFX9-GFX11). Since only a single thread6611is accessing the memory, atomic memory orderings are not meaningful, and all6612accesses are treated as non-atomic.6613 6614Constant address space uses ``buffer/global_load`` instructions (or equivalent6615scalar memory instructions). Since the constant address space contents do not6616change during the execution of a kernel dispatch it is not legal to perform6617stores, and atomic memory orderings are not meaningful, and all accesses are6618treated as non-atomic.6619 6620A memory synchronization scope wider than work-group is not meaningful for the6621group (LDS) address space and is treated as work-group.6622 6623The memory model does not support the region address space which is treated as6624non-atomic.6625 6626Acquire memory ordering is not meaningful on store atomic instructions and is6627treated as non-atomic.6628 6629Release memory ordering is not meaningful on load atomic instructions and is6630treated as non-atomic.6631 6632Acquire-release memory ordering is not meaningful on load or store atomic6633instructions and is treated as acquire and release respectively.6634 6635The memory order also adds the single thread optimization constraints defined in6636table6637:ref:`amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table`.6638 6639 .. table:: AMDHSA Memory Model Single Thread Optimization Constraints6640 :name: amdgpu-amdhsa-memory-model-single-thread-optimization-constraints-table6641 6642 ============ ==============================================================6643 LLVM Memory Optimization Constraints6644 Ordering6645 ============ ==============================================================6646 unordered *none*6647 monotonic *none*6648 acquire - If a load atomic/atomicrmw then no following load/load6649 atomic/store/store atomic/atomicrmw/fence instruction can be6650 moved before the acquire.6651 - If a fence then same as load atomic, plus no preceding6652 associated fence-paired-atomic can be moved after the fence.6653 release - If a store atomic/atomicrmw then no preceding load/load6654 atomic/store/store atomic/atomicrmw/fence instruction can be6655 moved after the release.6656 - If a fence then same as store atomic, plus no following6657 associated fence-paired-atomic can be moved before the6658 fence.6659 acq_rel Same constraints as both acquire and release.6660 seq_cst - If a load atomic then same constraints as acquire, plus no6661 preceding sequentially consistent load atomic/store6662 atomic/atomicrmw/fence instruction can be moved after the6663 seq_cst.6664 - If a store atomic then the same constraints as release, plus6665 no following sequentially consistent load atomic/store6666 atomic/atomicrmw/fence instruction can be moved before the6667 seq_cst.6668 - If an atomicrmw/fence then same constraints as acq_rel.6669 ============ ==============================================================6670 6671The code sequences used to implement the memory model are defined in the6672following sections:6673 6674* :ref:`amdgpu-amdhsa-memory-model-gfx6-gfx9`6675* :ref:`amdgpu-amdhsa-memory-model-gfx90a`6676* :ref:`amdgpu-amdhsa-memory-model-gfx942`6677* :ref:`amdgpu-amdhsa-memory-model-gfx10-gfx11`6678* :ref:`amdgpu-amdhsa-memory-model-gfx12`6679* :ref:`amdgpu-amdhsa-memory-model-gfx125x`6680 6681.. _amdgpu-fence-as:6682 6683Fence and Address Spaces6684++++++++++++++++++++++++++++++6685 6686LLVM fences do not have address space information, thus, fence6687codegen usually needs to conservatively synchronize all address spaces.6688 6689In the case of OpenCL, where fences only need to synchronize6690user-specified address spaces, this can result in extra unnecessary waits.6691For instance, a fence that is supposed to only synchronize local memory will6692also have to wait on all global memory operations, which is unnecessary.6693 6694:doc:`Memory Model Relaxation Annotations <MemoryModelRelaxationAnnotations>` can6695be used as an optimization hint for fences to solve this problem.6696The AMDGPU backend recognizes the following tags on fences to control which address6697space a fence can synchronize:6698 6699- ``amdgpu-synchronize-as:local`` - for the local address space6700- ``amdgpu-synchronize-as:global``- for the global address space6701 6702Multiple tags can be used at the same time to synchronize with more than one address space.6703 6704.. note::6705 6706 As an optimization hint, those tags are not guaranteed to survive until6707 code generation. Optimizations are free to drop the tags to allow for6708 better code optimization, at the cost of synchronizing additional address6709 spaces.6710 6711.. _amdgpu-amdhsa-memory-model-gfx6-gfx9:6712 6713Memory Model GFX6-GFX96714++++++++++++++++++++++6715 6716For GFX6-GFX9:6717 6718* Each agent has multiple shader arrays (SA).6719* Each SA has multiple compute units (CU).6720* Each CU has multiple SIMDs that execute wavefronts.6721* The wavefronts for a single work-group are executed in the same CU but may be6722 executed by different SIMDs.6723* Each CU has a single LDS memory shared by the wavefronts of the work-groups6724 executing on it.6725* All LDS operations of a CU are performed as wavefront wide operations in a6726 global order and involve no caching. Completion is reported to a wavefront in6727 execution order.6728* The LDS memory has multiple request queues shared by the SIMDs of a6729 CU. Therefore, the LDS operations performed by different wavefronts of a6730 work-group can be reordered relative to each other, which can result in6731 reordering the visibility of vector memory operations with respect to LDS6732 operations of other wavefronts in the same work-group. A ``s_waitcnt6733 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and6734 vector memory operations between wavefronts of a work-group, but not between6735 operations performed by the same wavefront.6736* The vector memory operations are performed as wavefront wide operations and6737 completion is reported to a wavefront in execution order. The exception is6738 that for GFX7-GFX9 ``flat_load/store/atomic`` instructions can report out of6739 vector memory order if they access LDS memory, and out of LDS operation order6740 if they access global memory.6741* The vector memory operations access a single vector L1 cache shared by all6742 SIMDs a CU. Therefore, no special action is required for coherence between the6743 lanes of a single wavefront, or for coherence between wavefronts in the same6744 work-group. A ``buffer_wbinvl1_vol`` is required for coherence between6745 wavefronts executing in different work-groups as they may be executing on6746 different CUs.6747* The scalar memory operations access a scalar L1 cache shared by all wavefronts6748 on a group of CUs. The scalar and vector L1 caches are not coherent. However,6749 scalar operations are used in a restricted way so do not impact the memory6750 model. See :ref:`amdgpu-amdhsa-memory-spaces`.6751* The vector and scalar memory operations use an L2 cache shared by all CUs on6752 the same agent.6753* The L2 cache has independent channels to service disjoint ranges of virtual6754 addresses.6755* Each CU has a separate request queue per channel. Therefore, the vector and6756 scalar memory operations performed by wavefronts executing in different6757 work-groups (which may be executing on different CUs) of an agent can be6758 reordered relative to each other. A ``s_waitcnt vmcnt(0)`` is required to6759 ensure synchronization between vector memory operations of different CUs. It6760 ensures a previous vector memory operation has completed before executing a6761 subsequent vector memory or LDS operation and so can be used to meet the6762 requirements of acquire and release.6763* The L2 cache can be kept coherent with other agents on some targets, or ranges6764 of virtual addresses can be set up to bypass it to ensure system coherence.6765 6766Scalar memory operations are only used to access memory that is proven to not6767change during the execution of the kernel dispatch. This includes constant6768address space and global address space for program scope ``const`` variables.6769Therefore, the kernel machine code does not have to maintain the scalar cache to6770ensure it is coherent with the vector caches. The scalar and vector caches are6771invalidated between kernel dispatches by CP since constant address space data6772may change between kernel dispatch executions. See6773:ref:`amdgpu-amdhsa-memory-spaces`.6774 6775The one exception is if scalar writes are used to spill SGPR registers. In this6776case the AMDGPU backend ensures the memory location used to spill is never6777accessed by vector memory operations at the same time. If scalar writes are used6778then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function6779return since the locations may be used for vector memory instructions by a6780future wavefront that uses the same scratch area, or a function call that6781creates a frame at the same address, respectively. There is no need for a6782``s_dcache_inv`` as all scalar writes are write-before-read in the same thread.6783 6784For kernarg backing memory:6785 6786* CP invalidates the L1 cache at the start of each kernel dispatch.6787* On dGPU the kernarg backing memory is allocated in host memory accessed as6788 MTYPE UC (uncached) to avoid needing to invalidate the L2 cache. This also6789 causes it to be treated as non-volatile and so is not invalidated by6790 ``*_vol``.6791* On APU the kernarg backing memory it is accessed as MTYPE CC (cache coherent)6792 and so the L2 cache will be coherent with the CPU and other agents.6793 6794Scratch backing memory (which is used for the private address space) is accessed6795with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is6796only accessed by a single thread, and is always write-before-read, there is6797never a need to invalidate these entries from the L1 cache. Hence all cache6798invalidates are done as ``*_vol`` to only invalidate the volatile cache lines.6799 6800The code sequences used to implement the memory model for GFX6-GFX9 are defined6801in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table`.6802 6803 .. table:: AMDHSA Memory Model Code Sequences GFX6-GFX96804 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx6-gfx9-table6805 6806 ============ ============ ============== ========== ================================6807 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code6808 Ordering Sync Scope Address GFX6-GFX96809 Space6810 ============ ============ ============== ========== ================================6811 **Non-Atomic**6812 ------------------------------------------------------------------------------------6813 load *none* *none* - global - !volatile & !nontemporal6814 - generic6815 - private 1. buffer/global/flat_load6816 - constant6817 - !volatile & nontemporal6818 6819 1. buffer/global/flat_load6820 glc=1 slc=16821 6822 - volatile6823 6824 1. buffer/global/flat_load6825 glc=16826 2. s_waitcnt vmcnt(0)6827 6828 - Must happen before6829 any following volatile6830 global/generic6831 load/store.6832 - Ensures that6833 volatile6834 operations to6835 different6836 addresses will not6837 be reordered by6838 hardware.6839 6840 load *none* *none* - local 1. ds_load6841 store *none* *none* - global - !volatile & !nontemporal6842 - generic6843 - private 1. buffer/global/flat_store6844 - constant6845 - !volatile & nontemporal6846 6847 1. buffer/global/flat_store6848 glc=1 slc=16849 6850 - volatile6851 6852 1. buffer/global/flat_store6853 2. s_waitcnt vmcnt(0)6854 6855 - Must happen before6856 any following volatile6857 global/generic6858 load/store.6859 - Ensures that6860 volatile6861 operations to6862 different6863 addresses will not6864 be reordered by6865 hardware.6866 6867 store *none* *none* - local 1. ds_store6868 **Unordered Atomic**6869 ------------------------------------------------------------------------------------6870 load atomic unordered *any* *any* *Same as non-atomic*.6871 store atomic unordered *any* *any* *Same as non-atomic*.6872 atomicrmw unordered *any* *any* *Same as monotonic atomic*.6873 **Monotonic Atomic**6874 ------------------------------------------------------------------------------------6875 load atomic monotonic - singlethread - global 1. buffer/global/ds/flat_load6876 - wavefront - local6877 - workgroup - generic6878 load atomic monotonic - agent - global 1. buffer/global/flat_load6879 - system - generic glc=16880 store atomic monotonic - singlethread - global 1. buffer/global/flat_store6881 - wavefront - generic6882 - workgroup6883 - agent6884 - system6885 store atomic monotonic - singlethread - local 1. ds_store6886 - wavefront6887 - workgroup6888 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic6889 - wavefront - generic6890 - workgroup6891 - agent6892 - system6893 atomicrmw monotonic - singlethread - local 1. ds_atomic6894 - wavefront6895 - workgroup6896 **Acquire Atomic**6897 ------------------------------------------------------------------------------------6898 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load6899 - wavefront - local6900 - generic6901 load atomic acquire - workgroup - global 1. buffer/global_load6902 load atomic acquire - workgroup - local 1. ds/flat_load6903 - generic 2. s_waitcnt lgkmcnt(0)6904 6905 - If OpenCL, omit.6906 - Must happen before6907 any following6908 global/generic6909 load/load6910 atomic/store/store6911 atomic/atomicrmw.6912 - Ensures any6913 following global6914 data read is no6915 older than a local load6916 atomic value being6917 acquired.6918 6919 load atomic acquire - agent - global 1. buffer/global_load6920 - system glc=16921 2. s_waitcnt vmcnt(0)6922 6923 - Must happen before6924 following6925 buffer_wbinvl1_vol.6926 - Ensures the load6927 has completed6928 before invalidating6929 the cache.6930 6931 3. buffer_wbinvl1_vol6932 6933 - Must happen before6934 any following6935 global/generic6936 load/load6937 atomic/atomicrmw.6938 - Ensures that6939 following6940 loads will not see6941 stale global data.6942 6943 load atomic acquire - agent - generic 1. flat_load glc=16944 - system 2. s_waitcnt vmcnt(0) &6945 lgkmcnt(0)6946 6947 - If OpenCL omit6948 lgkmcnt(0).6949 - Must happen before6950 following6951 buffer_wbinvl1_vol.6952 - Ensures the flat_load6953 has completed6954 before invalidating6955 the cache.6956 6957 3. buffer_wbinvl1_vol6958 6959 - Must happen before6960 any following6961 global/generic6962 load/load6963 atomic/atomicrmw.6964 - Ensures that6965 following loads6966 will not see stale6967 global data.6968 6969 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic6970 - wavefront - local6971 - generic6972 atomicrmw acquire - workgroup - global 1. buffer/global_atomic6973 atomicrmw acquire - workgroup - local 1. ds/flat_atomic6974 - generic 2. s_waitcnt lgkmcnt(0)6975 6976 - If OpenCL, omit.6977 - Must happen before6978 any following6979 global/generic6980 load/load6981 atomic/store/store6982 atomic/atomicrmw.6983 - Ensures any6984 following global6985 data read is no6986 older than a local6987 atomicrmw value6988 being acquired.6989 6990 atomicrmw acquire - agent - global 1. buffer/global_atomic6991 - system 2. s_waitcnt vmcnt(0)6992 6993 - Must happen before6994 following6995 buffer_wbinvl1_vol.6996 - Ensures the6997 atomicrmw has6998 completed before6999 invalidating the7000 cache.7001 7002 3. buffer_wbinvl1_vol7003 7004 - Must happen before7005 any following7006 global/generic7007 load/load7008 atomic/atomicrmw.7009 - Ensures that7010 following loads7011 will not see stale7012 global data.7013 7014 atomicrmw acquire - agent - generic 1. flat_atomic7015 - system 2. s_waitcnt vmcnt(0) &7016 lgkmcnt(0)7017 7018 - If OpenCL, omit7019 lgkmcnt(0).7020 - Must happen before7021 following7022 buffer_wbinvl1_vol.7023 - Ensures the7024 atomicrmw has7025 completed before7026 invalidating the7027 cache.7028 7029 3. buffer_wbinvl1_vol7030 7031 - Must happen before7032 any following7033 global/generic7034 load/load7035 atomic/atomicrmw.7036 - Ensures that7037 following loads7038 will not see stale7039 global data.7040 7041 fence acquire - singlethread *none* *none*7042 - wavefront7043 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0)7044 7045 - If OpenCL and7046 address space is7047 not generic, omit.7048 - See :ref:`amdgpu-fence-as` for7049 more details on fencing specific7050 address spaces.7051 - Must happen after7052 any preceding7053 local/generic load7054 atomic/atomicrmw7055 with an equal or7056 wider sync scope7057 and memory ordering7058 stronger than7059 unordered (this is7060 termed the7061 fence-paired-atomic).7062 - Must happen before7063 any following7064 global/generic7065 load/load7066 atomic/store/store7067 atomic/atomicrmw.7068 - Ensures any7069 following global7070 data read is no7071 older than the7072 value read by the7073 fence-paired-atomic.7074 7075 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) &7076 - system vmcnt(0)7077 7078 - If OpenCL and7079 address space is7080 not generic, omit7081 lgkmcnt(0).7082 - See :ref:`amdgpu-fence-as` for7083 more details on fencing specific7084 address spaces.7085 - Could be split into7086 separate s_waitcnt7087 vmcnt(0) and7088 s_waitcnt7089 lgkmcnt(0) to allow7090 them to be7091 independently moved7092 according to the7093 following rules.7094 - s_waitcnt vmcnt(0)7095 must happen after7096 any preceding7097 global/generic load7098 atomic/atomicrmw7099 with an equal or7100 wider sync scope7101 and memory ordering7102 stronger than7103 unordered (this is7104 termed the7105 fence-paired-atomic).7106 - s_waitcnt lgkmcnt(0)7107 must happen after7108 any preceding7109 local/generic load7110 atomic/atomicrmw7111 with an equal or7112 wider sync scope7113 and memory ordering7114 stronger than7115 unordered (this is7116 termed the7117 fence-paired-atomic).7118 - Must happen before7119 the following7120 buffer_wbinvl1_vol.7121 - Ensures that the7122 fence-paired atomic7123 has completed7124 before invalidating7125 the7126 cache. Therefore7127 any following7128 locations read must7129 be no older than7130 the value read by7131 the7132 fence-paired-atomic.7133 7134 2. buffer_wbinvl1_vol7135 7136 - Must happen before any7137 following global/generic7138 load/load7139 atomic/store/store7140 atomic/atomicrmw.7141 - Ensures that7142 following loads7143 will not see stale7144 global data.7145 7146 **Release Atomic**7147 ------------------------------------------------------------------------------------7148 store atomic release - singlethread - global 1. buffer/global/ds/flat_store7149 - wavefront - local7150 - generic7151 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0)7152 - generic7153 - If OpenCL, omit.7154 - Must happen after7155 any preceding7156 local/generic7157 load/store/load7158 atomic/store7159 atomic/atomicrmw.7160 - Must happen before7161 the following7162 store.7163 - Ensures that all7164 memory operations7165 to local have7166 completed before7167 performing the7168 store that is being7169 released.7170 7171 2. buffer/global/flat_store7172 store atomic release - workgroup - local 1. ds_store7173 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) &7174 - system - generic vmcnt(0)7175 7176 - If OpenCL and7177 address space is7178 not generic, omit7179 lgkmcnt(0).7180 - Could be split into7181 separate s_waitcnt7182 vmcnt(0) and7183 s_waitcnt7184 lgkmcnt(0) to allow7185 them to be7186 independently moved7187 according to the7188 following rules.7189 - s_waitcnt vmcnt(0)7190 must happen after7191 any preceding7192 global/generic7193 load/store/load7194 atomic/store7195 atomic/atomicrmw.7196 - s_waitcnt lgkmcnt(0)7197 must happen after7198 any preceding7199 local/generic7200 load/store/load7201 atomic/store7202 atomic/atomicrmw.7203 - Must happen before7204 the following7205 store.7206 - Ensures that all7207 memory operations7208 to memory have7209 completed before7210 performing the7211 store that is being7212 released.7213 7214 2. buffer/global/flat_store7215 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic7216 - wavefront - local7217 - generic7218 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0)7219 - generic7220 - If OpenCL, omit.7221 - Must happen after7222 any preceding7223 local/generic7224 load/store/load7225 atomic/store7226 atomic/atomicrmw.7227 - Must happen before7228 the following7229 atomicrmw.7230 - Ensures that all7231 memory operations7232 to local have7233 completed before7234 performing the7235 atomicrmw that is7236 being released.7237 7238 2. buffer/global/flat_atomic7239 atomicrmw release - workgroup - local 1. ds_atomic7240 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) &7241 - system - generic vmcnt(0)7242 7243 - If OpenCL, omit7244 lgkmcnt(0).7245 - Could be split into7246 separate s_waitcnt7247 vmcnt(0) and7248 s_waitcnt7249 lgkmcnt(0) to allow7250 them to be7251 independently moved7252 according to the7253 following rules.7254 - s_waitcnt vmcnt(0)7255 must happen after7256 any preceding7257 global/generic7258 load/store/load7259 atomic/store7260 atomic/atomicrmw.7261 - s_waitcnt lgkmcnt(0)7262 must happen after7263 any preceding7264 local/generic7265 load/store/load7266 atomic/store7267 atomic/atomicrmw.7268 - Must happen before7269 the following7270 atomicrmw.7271 - Ensures that all7272 memory operations7273 to global and local7274 have completed7275 before performing7276 the atomicrmw that7277 is being released.7278 7279 2. buffer/global/flat_atomic7280 fence release - singlethread *none* *none*7281 - wavefront7282 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0)7283 7284 - If OpenCL and7285 address space is7286 not generic, omit.7287 - See :ref:`amdgpu-fence-as` for7288 more details on fencing specific7289 address spaces.7290 - Must happen after7291 any preceding7292 local/generic7293 load/load7294 atomic/store/store7295 atomic/atomicrmw.7296 - Must happen before7297 any following store7298 atomic/atomicrmw7299 with an equal or7300 wider sync scope7301 and memory ordering7302 stronger than7303 unordered (this is7304 termed the7305 fence-paired-atomic).7306 - Ensures that all7307 memory operations7308 to local have7309 completed before7310 performing the7311 following7312 fence-paired-atomic.7313 7314 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) &7315 - system vmcnt(0)7316 7317 - If OpenCL and7318 address space is7319 not generic, omit7320 lgkmcnt(0).7321 - If OpenCL and7322 address space is7323 local, omit7324 vmcnt(0).7325 - See :ref:`amdgpu-fence-as` for7326 more details on fencing specific7327 address spaces.7328 - Could be split into7329 separate s_waitcnt7330 vmcnt(0) and7331 s_waitcnt7332 lgkmcnt(0) to allow7333 them to be7334 independently moved7335 according to the7336 following rules.7337 - s_waitcnt vmcnt(0)7338 must happen after7339 any preceding7340 global/generic7341 load/store/load7342 atomic/store7343 atomic/atomicrmw.7344 - s_waitcnt lgkmcnt(0)7345 must happen after7346 any preceding7347 local/generic7348 load/store/load7349 atomic/store7350 atomic/atomicrmw.7351 - Must happen before7352 any following store7353 atomic/atomicrmw7354 with an equal or7355 wider sync scope7356 and memory ordering7357 stronger than7358 unordered (this is7359 termed the7360 fence-paired-atomic).7361 - Ensures that all7362 memory operations7363 have7364 completed before7365 performing the7366 following7367 fence-paired-atomic.7368 7369 **Acquire-Release Atomic**7370 ------------------------------------------------------------------------------------7371 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic7372 - wavefront - local7373 - generic7374 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0)7375 7376 - If OpenCL, omit.7377 - Must happen after7378 any preceding7379 local/generic7380 load/store/load7381 atomic/store7382 atomic/atomicrmw.7383 - Must happen before7384 the following7385 atomicrmw.7386 - Ensures that all7387 memory operations7388 to local have7389 completed before7390 performing the7391 atomicrmw that is7392 being released.7393 7394 2. buffer/global_atomic7395 7396 atomicrmw acq_rel - workgroup - local 1. ds_atomic7397 2. s_waitcnt lgkmcnt(0)7398 7399 - If OpenCL, omit.7400 - Must happen before7401 any following7402 global/generic7403 load/load7404 atomic/store/store7405 atomic/atomicrmw.7406 - Ensures any7407 following global7408 data read is no7409 older than the local load7410 atomic value being7411 acquired.7412 7413 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0)7414 7415 - If OpenCL, omit.7416 - Must happen after7417 any preceding7418 local/generic7419 load/store/load7420 atomic/store7421 atomic/atomicrmw.7422 - Must happen before7423 the following7424 atomicrmw.7425 - Ensures that all7426 memory operations7427 to local have7428 completed before7429 performing the7430 atomicrmw that is7431 being released.7432 7433 2. flat_atomic7434 3. s_waitcnt lgkmcnt(0)7435 7436 - If OpenCL, omit.7437 - Must happen before7438 any following7439 global/generic7440 load/load7441 atomic/store/store7442 atomic/atomicrmw.7443 - Ensures any7444 following global7445 data read is no7446 older than a local load7447 atomic value being7448 acquired.7449 7450 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) &7451 - system vmcnt(0)7452 7453 - If OpenCL, omit7454 lgkmcnt(0).7455 - Could be split into7456 separate s_waitcnt7457 vmcnt(0) and7458 s_waitcnt7459 lgkmcnt(0) to allow7460 them to be7461 independently moved7462 according to the7463 following rules.7464 - s_waitcnt vmcnt(0)7465 must happen after7466 any preceding7467 global/generic7468 load/store/load7469 atomic/store7470 atomic/atomicrmw.7471 - s_waitcnt lgkmcnt(0)7472 must happen after7473 any preceding7474 local/generic7475 load/store/load7476 atomic/store7477 atomic/atomicrmw.7478 - Must happen before7479 the following7480 atomicrmw.7481 - Ensures that all7482 memory operations7483 to global have7484 completed before7485 performing the7486 atomicrmw that is7487 being released.7488 7489 2. buffer/global_atomic7490 3. s_waitcnt vmcnt(0)7491 7492 - Must happen before7493 following7494 buffer_wbinvl1_vol.7495 - Ensures the7496 atomicrmw has7497 completed before7498 invalidating the7499 cache.7500 7501 4. buffer_wbinvl1_vol7502 7503 - Must happen before7504 any following7505 global/generic7506 load/load7507 atomic/atomicrmw.7508 - Ensures that7509 following loads7510 will not see stale7511 global data.7512 7513 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) &7514 - system vmcnt(0)7515 7516 - If OpenCL, omit7517 lgkmcnt(0).7518 - Could be split into7519 separate s_waitcnt7520 vmcnt(0) and7521 s_waitcnt7522 lgkmcnt(0) to allow7523 them to be7524 independently moved7525 according to the7526 following rules.7527 - s_waitcnt vmcnt(0)7528 must happen after7529 any preceding7530 global/generic7531 load/store/load7532 atomic/store7533 atomic/atomicrmw.7534 - s_waitcnt lgkmcnt(0)7535 must happen after7536 any preceding7537 local/generic7538 load/store/load7539 atomic/store7540 atomic/atomicrmw.7541 - Must happen before7542 the following7543 atomicrmw.7544 - Ensures that all7545 memory operations7546 to global have7547 completed before7548 performing the7549 atomicrmw that is7550 being released.7551 7552 2. flat_atomic7553 3. s_waitcnt vmcnt(0) &7554 lgkmcnt(0)7555 7556 - If OpenCL, omit7557 lgkmcnt(0).7558 - Must happen before7559 following7560 buffer_wbinvl1_vol.7561 - Ensures the7562 atomicrmw has7563 completed before7564 invalidating the7565 cache.7566 7567 4. buffer_wbinvl1_vol7568 7569 - Must happen before7570 any following7571 global/generic7572 load/load7573 atomic/atomicrmw.7574 - Ensures that7575 following loads7576 will not see stale7577 global data.7578 7579 fence acq_rel - singlethread *none* *none*7580 - wavefront7581 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0)7582 7583 - If OpenCL and7584 address space is7585 not generic, omit.7586 - However,7587 since LLVM7588 currently has no7589 address space on7590 the fence need to7591 conservatively7592 always generate7593 (see comment for7594 previous fence).7595 - Must happen after7596 any preceding7597 local/generic7598 load/load7599 atomic/store/store7600 atomic/atomicrmw.7601 - Must happen before7602 any following7603 global/generic7604 load/load7605 atomic/store/store7606 atomic/atomicrmw.7607 - Ensures that all7608 memory operations7609 to local have7610 completed before7611 performing any7612 following global7613 memory operations.7614 - Ensures that the7615 preceding7616 local/generic load7617 atomic/atomicrmw7618 with an equal or7619 wider sync scope7620 and memory ordering7621 stronger than7622 unordered (this is7623 termed the7624 acquire-fence-paired-atomic)7625 has completed7626 before following7627 global memory7628 operations. This7629 satisfies the7630 requirements of7631 acquire.7632 - Ensures that all7633 previous memory7634 operations have7635 completed before a7636 following7637 local/generic store7638 atomic/atomicrmw7639 with an equal or7640 wider sync scope7641 and memory ordering7642 stronger than7643 unordered (this is7644 termed the7645 release-fence-paired-atomic).7646 This satisfies the7647 requirements of7648 release.7649 7650 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) &7651 - system vmcnt(0)7652 7653 - If OpenCL and7654 address space is7655 not generic, omit7656 lgkmcnt(0).7657 - See :ref:`amdgpu-fence-as` for7658 more details on fencing specific7659 address spaces.7660 - Could be split into7661 separate s_waitcnt7662 vmcnt(0) and7663 s_waitcnt7664 lgkmcnt(0) to allow7665 them to be7666 independently moved7667 according to the7668 following rules.7669 - s_waitcnt vmcnt(0)7670 must happen after7671 any preceding7672 global/generic7673 load/store/load7674 atomic/store7675 atomic/atomicrmw.7676 - s_waitcnt lgkmcnt(0)7677 must happen after7678 any preceding7679 local/generic7680 load/store/load7681 atomic/store7682 atomic/atomicrmw.7683 - Must happen before7684 the following7685 buffer_wbinvl1_vol.7686 - Ensures that the7687 preceding7688 global/local/generic7689 load7690 atomic/atomicrmw7691 with an equal or7692 wider sync scope7693 and memory ordering7694 stronger than7695 unordered (this is7696 termed the7697 acquire-fence-paired-atomic)7698 has completed7699 before invalidating7700 the cache. This7701 satisfies the7702 requirements of7703 acquire.7704 - Ensures that all7705 previous memory7706 operations have7707 completed before a7708 following7709 global/local/generic7710 store7711 atomic/atomicrmw7712 with an equal or7713 wider sync scope7714 and memory ordering7715 stronger than7716 unordered (this is7717 termed the7718 release-fence-paired-atomic).7719 This satisfies the7720 requirements of7721 release.7722 7723 2. buffer_wbinvl1_vol7724 7725 - Must happen before7726 any following7727 global/generic7728 load/load7729 atomic/store/store7730 atomic/atomicrmw.7731 - Ensures that7732 following loads7733 will not see stale7734 global data. This7735 satisfies the7736 requirements of7737 acquire.7738 7739 **Sequential Consistent Atomic**7740 ------------------------------------------------------------------------------------7741 load atomic seq_cst - singlethread - global *Same as corresponding7742 - wavefront - local load atomic acquire,7743 - generic except must generate7744 all instructions even7745 for OpenCL.*7746 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0)7747 - generic7748 7749 - Must7750 happen after7751 preceding7752 local/generic load7753 atomic/store7754 atomic/atomicrmw7755 with memory7756 ordering of seq_cst7757 and with equal or7758 wider sync scope.7759 (Note that seq_cst7760 fences have their7761 own s_waitcnt7762 lgkmcnt(0) and so do7763 not need to be7764 considered.)7765 - Ensures any7766 preceding7767 sequential7768 consistent local7769 memory instructions7770 have completed7771 before executing7772 this sequentially7773 consistent7774 instruction. This7775 prevents reordering7776 a seq_cst store7777 followed by a7778 seq_cst load. (Note7779 that seq_cst is7780 stronger than7781 acquire/release as7782 the reordering of7783 load acquire7784 followed by a store7785 release is7786 prevented by the7787 s_waitcnt of7788 the release, but7789 there is nothing7790 preventing a store7791 release followed by7792 load acquire from7793 completing out of7794 order. The s_waitcnt7795 could be placed after7796 seq_store or before7797 the seq_load. We7798 choose the load to7799 make the s_waitcnt be7800 as late as possible7801 so that the store7802 may have already7803 completed.)7804 7805 2. *Following7806 instructions same as7807 corresponding load7808 atomic acquire,7809 except must generate7810 all instructions even7811 for OpenCL.*7812 load atomic seq_cst - workgroup - local *Same as corresponding7813 load atomic acquire,7814 except must generate7815 all instructions even7816 for OpenCL.*7817 7818 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) &7819 - system - generic vmcnt(0)7820 7821 - Could be split into7822 separate s_waitcnt7823 vmcnt(0)7824 and s_waitcnt7825 lgkmcnt(0) to allow7826 them to be7827 independently moved7828 according to the7829 following rules.7830 - s_waitcnt lgkmcnt(0)7831 must happen after7832 preceding7833 global/generic load7834 atomic/store7835 atomic/atomicrmw7836 with memory7837 ordering of seq_cst7838 and with equal or7839 wider sync scope.7840 (Note that seq_cst7841 fences have their7842 own s_waitcnt7843 lgkmcnt(0) and so do7844 not need to be7845 considered.)7846 - s_waitcnt vmcnt(0)7847 must happen after7848 preceding7849 global/generic load7850 atomic/store7851 atomic/atomicrmw7852 with memory7853 ordering of seq_cst7854 and with equal or7855 wider sync scope.7856 (Note that seq_cst7857 fences have their7858 own s_waitcnt7859 vmcnt(0) and so do7860 not need to be7861 considered.)7862 - Ensures any7863 preceding7864 sequential7865 consistent global7866 memory instructions7867 have completed7868 before executing7869 this sequentially7870 consistent7871 instruction. This7872 prevents reordering7873 a seq_cst store7874 followed by a7875 seq_cst load. (Note7876 that seq_cst is7877 stronger than7878 acquire/release as7879 the reordering of7880 load acquire7881 followed by a store7882 release is7883 prevented by the7884 s_waitcnt of7885 the release, but7886 there is nothing7887 preventing a store7888 release followed by7889 load acquire from7890 completing out of7891 order. The s_waitcnt7892 could be placed after7893 seq_store or before7894 the seq_load. We7895 choose the load to7896 make the s_waitcnt be7897 as late as possible7898 so that the store7899 may have already7900 completed.)7901 7902 2. *Following7903 instructions same as7904 corresponding load7905 atomic acquire,7906 except must generate7907 all instructions even7908 for OpenCL.*7909 store atomic seq_cst - singlethread - global *Same as corresponding7910 - wavefront - local store atomic release,7911 - workgroup - generic except must generate7912 - agent all instructions even7913 - system for OpenCL.*7914 atomicrmw seq_cst - singlethread - global *Same as corresponding7915 - wavefront - local atomicrmw acq_rel,7916 - workgroup - generic except must generate7917 - agent all instructions even7918 - system for OpenCL.*7919 fence seq_cst - singlethread *none* *Same as corresponding7920 - wavefront fence acq_rel,7921 - workgroup except must generate7922 - agent all instructions even7923 - system for OpenCL.*7924 ============ ============ ============== ========== ================================7925 7926.. _amdgpu-amdhsa-memory-model-gfx90a:7927 7928Memory Model GFX90A7929+++++++++++++++++++7930 7931For GFX90A:7932 7933* Each agent has multiple shader arrays (SA).7934* Each SA has multiple compute units (CU).7935* Each CU has multiple SIMDs that execute wavefronts.7936* The wavefronts for a single work-group are executed in the same CU but may be7937 executed by different SIMDs. The exception is when in tgsplit execution mode7938 when the wavefronts may be executed by different SIMDs in different CUs.7939* Each CU has a single LDS memory shared by the wavefronts of the work-groups7940 executing on it. The exception is when in tgsplit execution mode when no LDS7941 is allocated as wavefronts of the same work-group can be in different CUs.7942* All LDS operations of a CU are performed as wavefront wide operations in a7943 global order and involve no caching. Completion is reported to a wavefront in7944 execution order.7945* The LDS memory has multiple request queues shared by the SIMDs of a7946 CU. Therefore, the LDS operations performed by different wavefronts of a7947 work-group can be reordered relative to each other, which can result in7948 reordering the visibility of vector memory operations with respect to LDS7949 operations of other wavefronts in the same work-group. A ``s_waitcnt7950 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and7951 vector memory operations between wavefronts of a work-group, but not between7952 operations performed by the same wavefront.7953* The vector memory operations are performed as wavefront wide operations and7954 completion is reported to a wavefront in execution order. The exception is7955 that ``flat_load/store/atomic`` instructions can report out of vector memory7956 order if they access LDS memory, and out of LDS operation order if they access7957 global memory.7958* The vector memory operations access a single vector L1 cache shared by all7959 SIMDs a CU. Therefore:7960 7961 * No special action is required for coherence between the lanes of a single7962 wavefront.7963 7964 * No special action is required for coherence between wavefronts in the same7965 work-group since they execute on the same CU. The exception is when in7966 tgsplit execution mode as wavefronts of the same work-group can be in7967 different CUs and so a ``buffer_wbinvl1_vol`` is required as described in7968 the following item.7969 7970 * A ``buffer_wbinvl1_vol`` is required for coherence between wavefronts7971 executing in different work-groups as they may be executing on different7972 CUs.7973 7974* The scalar memory operations access a scalar L1 cache shared by all wavefronts7975 on a group of CUs. The scalar and vector L1 caches are not coherent. However,7976 scalar operations are used in a restricted way so do not impact the memory7977 model. See :ref:`amdgpu-amdhsa-memory-spaces`.7978* The vector and scalar memory operations use an L2 cache shared by all CUs on7979 the same agent.7980 7981 * The L2 cache has independent channels to service disjoint ranges of virtual7982 addresses.7983 * Each CU has a separate request queue per channel. Therefore, the vector and7984 scalar memory operations performed by wavefronts executing in different7985 work-groups (which may be executing on different CUs), or the same7986 work-group if executing in tgsplit mode, of an agent can be reordered7987 relative to each other. A ``s_waitcnt vmcnt(0)`` is required to ensure7988 synchronization between vector memory operations of different CUs. It7989 ensures a previous vector memory operation has completed before executing a7990 subsequent vector memory or LDS operation and so can be used to meet the7991 requirements of acquire and release.7992 * The L2 cache of one agent can be kept coherent with other agents by:7993 using the MTYPE RW (read-write) or MTYPE CC (cache-coherent) with the PTE7994 C-bit for memory local to the L2; and using the MTYPE NC (non-coherent) with7995 the PTE C-bit set or MTYPE UC (uncached) for memory not local to the L2.7996 7997 * Any local memory cache lines will be automatically invalidated by writes7998 from CUs associated with other L2 caches, or writes from the CPU, due to7999 the cache probe caused by coherent requests. Coherent requests are caused8000 by GPU accesses to pages with the PTE C-bit set, by CPU accesses over8001 XGMI, and by PCIe requests that are configured to be coherent requests.8002 * XGMI accesses from the CPU to local memory may be cached on the CPU.8003 Subsequent access from the GPU will automatically invalidate or writeback8004 the CPU cache due to the L2 probe filter and and the PTE C-bit being set.8005 * Since all work-groups on the same agent share the same L2, no L28006 invalidation or writeback is required for coherence.8007 * To ensure coherence of local and remote memory writes of work-groups in8008 different agents a ``buffer_wbl2`` is required. It will writeback dirty L28009 cache lines of MTYPE RW (used for local coarse grain memory) and MTYPE NC8010 ()used for remote coarse grain memory). Note that MTYPE CC (used for local8011 fine grain memory) causes write through to DRAM, and MTYPE UC (used for8012 remote fine grain memory) bypasses the L2, so both will never result in8013 dirty L2 cache lines.8014 * To ensure coherence of local and remote memory reads of work-groups in8015 different agents a ``buffer_invl2`` is required. It will invalidate L28016 cache lines with MTYPE NC (used for remote coarse grain memory). Note that8017 MTYPE CC (used for local fine grain memory) and MTYPE RW (used for local8018 coarse memory) cause local reads to be invalidated by remote writes with8019 with the PTE C-bit so these cache lines are not invalidated. Note that8020 MTYPE UC (used for remote fine grain memory) bypasses the L2, so will8021 never result in L2 cache lines that need to be invalidated.8022 8023 * PCIe access from the GPU to the CPU memory is kept coherent by using the8024 MTYPE UC (uncached) which bypasses the L2.8025 8026Scalar memory operations are only used to access memory that is proven to not8027change during the execution of the kernel dispatch. This includes constant8028address space and global address space for program scope ``const`` variables.8029Therefore, the kernel machine code does not have to maintain the scalar cache to8030ensure it is coherent with the vector caches. The scalar and vector caches are8031invalidated between kernel dispatches by CP since constant address space data8032may change between kernel dispatch executions. See8033:ref:`amdgpu-amdhsa-memory-spaces`.8034 8035The one exception is if scalar writes are used to spill SGPR registers. In this8036case the AMDGPU backend ensures the memory location used to spill is never8037accessed by vector memory operations at the same time. If scalar writes are used8038then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function8039return since the locations may be used for vector memory instructions by a8040future wavefront that uses the same scratch area, or a function call that8041creates a frame at the same address, respectively. There is no need for a8042``s_dcache_inv`` as all scalar writes are write-before-read in the same thread.8043 8044For kernarg backing memory:8045 8046* CP invalidates the L1 cache at the start of each kernel dispatch.8047* On dGPU over XGMI or PCIe the kernarg backing memory is allocated in host8048 memory accessed as MTYPE UC (uncached) to avoid needing to invalidate the L28049 cache. This also causes it to be treated as non-volatile and so is not8050 invalidated by ``*_vol``.8051* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and8052 so the L2 cache will be coherent with the CPU and other agents.8053 8054Scratch backing memory (which is used for the private address space) is accessed8055with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is8056only accessed by a single thread, and is always write-before-read, there is8057never a need to invalidate these entries from the L1 cache. Hence all cache8058invalidates are done as ``*_vol`` to only invalidate the volatile cache lines.8059 8060The code sequences used to implement the memory model for GFX90A are defined8061in table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx90a-table`.8062 8063 .. table:: AMDHSA Memory Model Code Sequences GFX90A8064 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx90a-table8065 8066 ============ ============ ============== ========== ================================8067 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code8068 Ordering Sync Scope Address GFX90A8069 Space8070 ============ ============ ============== ========== ================================8071 **Non-Atomic**8072 ------------------------------------------------------------------------------------8073 load *none* *none* - global - !volatile & !nontemporal8074 - generic8075 - private 1. buffer/global/flat_load8076 - constant8077 - !volatile & nontemporal8078 8079 1. buffer/global/flat_load8080 glc=1 slc=18081 8082 - volatile8083 8084 1. buffer/global/flat_load8085 glc=18086 2. s_waitcnt vmcnt(0)8087 8088 - Must happen before8089 any following volatile8090 global/generic8091 load/store.8092 - Ensures that8093 volatile8094 operations to8095 different8096 addresses will not8097 be reordered by8098 hardware.8099 8100 load *none* *none* - local 1. ds_load8101 store *none* *none* - global - !volatile & !nontemporal8102 - generic8103 - private 1. buffer/global/flat_store8104 - constant8105 - !volatile & nontemporal8106 8107 1. buffer/global/flat_store8108 glc=1 slc=18109 8110 - volatile8111 8112 1. buffer/global/flat_store8113 2. s_waitcnt vmcnt(0)8114 8115 - Must happen before8116 any following volatile8117 global/generic8118 load/store.8119 - Ensures that8120 volatile8121 operations to8122 different8123 addresses will not8124 be reordered by8125 hardware.8126 8127 store *none* *none* - local 1. ds_store8128 **Unordered Atomic**8129 ------------------------------------------------------------------------------------8130 load atomic unordered *any* *any* *Same as non-atomic*.8131 store atomic unordered *any* *any* *Same as non-atomic*.8132 atomicrmw unordered *any* *any* *Same as monotonic atomic*.8133 **Monotonic Atomic**8134 ------------------------------------------------------------------------------------8135 load atomic monotonic - singlethread - global 1. buffer/global/flat_load8136 - wavefront - generic8137 load atomic monotonic - workgroup - global 1. buffer/global/flat_load8138 - generic glc=18139 8140 - If not TgSplit execution8141 mode, omit glc=1.8142 8143 load atomic monotonic - singlethread - local *If TgSplit execution mode,8144 - wavefront local address space cannot8145 - workgroup be used.*8146 8147 1. ds_load8148 load atomic monotonic - agent - global 1. buffer/global/flat_load8149 - generic glc=18150 load atomic monotonic - system - global 1. buffer/global/flat_load8151 - generic glc=18152 store atomic monotonic - singlethread - global 1. buffer/global/flat_store8153 - wavefront - generic8154 - workgroup8155 - agent8156 store atomic monotonic - system - global 1. buffer/global/flat_store8157 - generic8158 store atomic monotonic - singlethread - local *If TgSplit execution mode,8159 - wavefront local address space cannot8160 - workgroup be used.*8161 8162 1. ds_store8163 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic8164 - wavefront - generic8165 - workgroup8166 - agent8167 atomicrmw monotonic - system - global 1. buffer/global/flat_atomic8168 - generic8169 atomicrmw monotonic - singlethread - local *If TgSplit execution mode,8170 - wavefront local address space cannot8171 - workgroup be used.*8172 8173 1. ds_atomic8174 **Acquire Atomic**8175 ------------------------------------------------------------------------------------8176 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load8177 - wavefront - local8178 - generic8179 load atomic acquire - workgroup - global 1. buffer/global_load glc=18180 8181 - If not TgSplit execution8182 mode, omit glc=1.8183 8184 2. s_waitcnt vmcnt(0)8185 8186 - If not TgSplit execution8187 mode, omit.8188 - Must happen before the8189 following buffer_wbinvl1_vol.8190 8191 3. buffer_wbinvl1_vol8192 8193 - If not TgSplit execution8194 mode, omit.8195 - Must happen before8196 any following8197 global/generic8198 load/load8199 atomic/store/store8200 atomic/atomicrmw.8201 - Ensures that8202 following8203 loads will not see8204 stale data.8205 8206 load atomic acquire - workgroup - local *If TgSplit execution mode,8207 local address space cannot8208 be used.*8209 8210 1. ds_load8211 2. s_waitcnt lgkmcnt(0)8212 8213 - If OpenCL, omit.8214 - Must happen before8215 any following8216 global/generic8217 load/load8218 atomic/store/store8219 atomic/atomicrmw.8220 - Ensures any8221 following global8222 data read is no8223 older than the local load8224 atomic value being8225 acquired.8226 8227 load atomic acquire - workgroup - generic 1. flat_load glc=18228 8229 - If not TgSplit execution8230 mode, omit glc=1.8231 8232 2. s_waitcnt lgkm/vmcnt(0)8233 8234 - Use lgkmcnt(0) if not8235 TgSplit execution mode8236 and vmcnt(0) if TgSplit8237 execution mode.8238 - If OpenCL, omit lgkmcnt(0).8239 - Must happen before8240 the following8241 buffer_wbinvl1_vol and any8242 following global/generic8243 load/load8244 atomic/store/store8245 atomic/atomicrmw.8246 - Ensures any8247 following global8248 data read is no8249 older than a local load8250 atomic value being8251 acquired.8252 8253 3. buffer_wbinvl1_vol8254 8255 - If not TgSplit execution8256 mode, omit.8257 - Ensures that8258 following8259 loads will not see8260 stale data.8261 8262 load atomic acquire - agent - global 1. buffer/global_load8263 glc=18264 2. s_waitcnt vmcnt(0)8265 8266 - Must happen before8267 following8268 buffer_wbinvl1_vol.8269 - Ensures the load8270 has completed8271 before invalidating8272 the cache.8273 8274 3. buffer_wbinvl1_vol8275 8276 - Must happen before8277 any following8278 global/generic8279 load/load8280 atomic/atomicrmw.8281 - Ensures that8282 following8283 loads will not see8284 stale global data.8285 8286 load atomic acquire - system - global 1. buffer/global/flat_load8287 glc=18288 2. s_waitcnt vmcnt(0)8289 8290 - Must happen before8291 following buffer_invl2 and8292 buffer_wbinvl1_vol.8293 - Ensures the load8294 has completed8295 before invalidating8296 the cache.8297 8298 3. buffer_invl2;8299 buffer_wbinvl1_vol8300 8301 - Must happen before8302 any following8303 global/generic8304 load/load8305 atomic/atomicrmw.8306 - Ensures that8307 following8308 loads will not see8309 stale L1 global data,8310 nor see stale L2 MTYPE8311 NC global data.8312 MTYPE RW and CC memory will8313 never be stale in L2 due to8314 the memory probes.8315 8316 load atomic acquire - agent - generic 1. flat_load glc=18317 2. s_waitcnt vmcnt(0) &8318 lgkmcnt(0)8319 8320 - If TgSplit execution mode,8321 omit lgkmcnt(0).8322 - If OpenCL omit8323 lgkmcnt(0).8324 - Must happen before8325 following8326 buffer_wbinvl1_vol.8327 - Ensures the flat_load8328 has completed8329 before invalidating8330 the cache.8331 8332 3. buffer_wbinvl1_vol8333 8334 - Must happen before8335 any following8336 global/generic8337 load/load8338 atomic/atomicrmw.8339 - Ensures that8340 following loads8341 will not see stale8342 global data.8343 8344 load atomic acquire - system - generic 1. flat_load glc=18345 2. s_waitcnt vmcnt(0) &8346 lgkmcnt(0)8347 8348 - If TgSplit execution mode,8349 omit lgkmcnt(0).8350 - If OpenCL omit8351 lgkmcnt(0).8352 - Must happen before8353 following8354 buffer_invl2 and8355 buffer_wbinvl1_vol.8356 - Ensures the flat_load8357 has completed8358 before invalidating8359 the caches.8360 8361 3. buffer_invl2;8362 buffer_wbinvl1_vol8363 8364 - Must happen before8365 any following8366 global/generic8367 load/load8368 atomic/atomicrmw.8369 - Ensures that8370 following8371 loads will not see8372 stale L1 global data,8373 nor see stale L2 MTYPE8374 NC global data.8375 MTYPE RW and CC memory will8376 never be stale in L2 due to8377 the memory probes.8378 8379 atomicrmw acquire - singlethread - global 1. buffer/global/flat_atomic8380 - wavefront - generic8381 atomicrmw acquire - singlethread - local *If TgSplit execution mode,8382 - wavefront local address space cannot8383 be used.*8384 8385 1. ds_atomic8386 atomicrmw acquire - workgroup - global 1. buffer/global_atomic8387 2. s_waitcnt vmcnt(0)8388 8389 - If not TgSplit execution8390 mode, omit.8391 - Must happen before the8392 following buffer_wbinvl1_vol.8393 - Ensures the atomicrmw8394 has completed8395 before invalidating8396 the cache.8397 8398 3. buffer_wbinvl1_vol8399 8400 - If not TgSplit execution8401 mode, omit.8402 - Must happen before8403 any following8404 global/generic8405 load/load8406 atomic/atomicrmw.8407 - Ensures that8408 following loads8409 will not see stale8410 global data.8411 8412 atomicrmw acquire - workgroup - local *If TgSplit execution mode,8413 local address space cannot8414 be used.*8415 8416 1. ds_atomic8417 2. s_waitcnt lgkmcnt(0)8418 8419 - If OpenCL, omit.8420 - Must happen before8421 any following8422 global/generic8423 load/load8424 atomic/store/store8425 atomic/atomicrmw.8426 - Ensures any8427 following global8428 data read is no8429 older than the local8430 atomicrmw value8431 being acquired.8432 8433 atomicrmw acquire - workgroup - generic 1. flat_atomic8434 2. s_waitcnt lgkm/vmcnt(0)8435 8436 - Use lgkmcnt(0) if not8437 TgSplit execution mode8438 and vmcnt(0) if TgSplit8439 execution mode.8440 - If OpenCL, omit lgkmcnt(0).8441 - Must happen before8442 the following8443 buffer_wbinvl1_vol and8444 any following8445 global/generic8446 load/load8447 atomic/store/store8448 atomic/atomicrmw.8449 - Ensures any8450 following global8451 data read is no8452 older than a local8453 atomicrmw value8454 being acquired.8455 8456 3. buffer_wbinvl1_vol8457 8458 - If not TgSplit execution8459 mode, omit.8460 - Ensures that8461 following8462 loads will not see8463 stale data.8464 8465 atomicrmw acquire - agent - global 1. buffer/global_atomic8466 2. s_waitcnt vmcnt(0)8467 8468 - Must happen before8469 following8470 buffer_wbinvl1_vol.8471 - Ensures the8472 atomicrmw has8473 completed before8474 invalidating the8475 cache.8476 8477 3. buffer_wbinvl1_vol8478 8479 - Must happen before8480 any following8481 global/generic8482 load/load8483 atomic/atomicrmw.8484 - Ensures that8485 following loads8486 will not see stale8487 global data.8488 8489 atomicrmw acquire - system - global 1. buffer/global_atomic8490 2. s_waitcnt vmcnt(0)8491 8492 - Must happen before8493 following buffer_invl2 and8494 buffer_wbinvl1_vol.8495 - Ensures the8496 atomicrmw has8497 completed before8498 invalidating the8499 caches.8500 8501 3. buffer_invl2;8502 buffer_wbinvl1_vol8503 8504 - Must happen before8505 any following8506 global/generic8507 load/load8508 atomic/atomicrmw.8509 - Ensures that8510 following8511 loads will not see8512 stale L1 global data,8513 nor see stale L2 MTYPE8514 NC global data.8515 MTYPE RW and CC memory will8516 never be stale in L2 due to8517 the memory probes.8518 8519 atomicrmw acquire - agent - generic 1. flat_atomic8520 2. s_waitcnt vmcnt(0) &8521 lgkmcnt(0)8522 8523 - If TgSplit execution mode,8524 omit lgkmcnt(0).8525 - If OpenCL, omit8526 lgkmcnt(0).8527 - Must happen before8528 following8529 buffer_wbinvl1_vol.8530 - Ensures the8531 atomicrmw has8532 completed before8533 invalidating the8534 cache.8535 8536 3. buffer_wbinvl1_vol8537 8538 - Must happen before8539 any following8540 global/generic8541 load/load8542 atomic/atomicrmw.8543 - Ensures that8544 following loads8545 will not see stale8546 global data.8547 8548 atomicrmw acquire - system - generic 1. flat_atomic8549 2. s_waitcnt vmcnt(0) &8550 lgkmcnt(0)8551 8552 - If TgSplit execution mode,8553 omit lgkmcnt(0).8554 - If OpenCL, omit8555 lgkmcnt(0).8556 - Must happen before8557 following8558 buffer_invl2 and8559 buffer_wbinvl1_vol.8560 - Ensures the8561 atomicrmw has8562 completed before8563 invalidating the8564 caches.8565 8566 3. buffer_invl2;8567 buffer_wbinvl1_vol8568 8569 - Must happen before8570 any following8571 global/generic8572 load/load8573 atomic/atomicrmw.8574 - Ensures that8575 following8576 loads will not see8577 stale L1 global data,8578 nor see stale L2 MTYPE8579 NC global data.8580 MTYPE RW and CC memory will8581 never be stale in L2 due to8582 the memory probes.8583 8584 fence acquire - singlethread *none* *none*8585 - wavefront8586 fence acquire - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0)8587 8588 - Use lgkmcnt(0) if not8589 TgSplit execution mode8590 and vmcnt(0) if TgSplit8591 execution mode.8592 - If OpenCL and8593 address space is8594 not generic, omit8595 lgkmcnt(0).8596 - If OpenCL and8597 address space is8598 local, omit8599 vmcnt(0).8600 - See :ref:`amdgpu-fence-as` for8601 more details on fencing specific8602 address spaces.8603 - s_waitcnt vmcnt(0)8604 must happen after8605 any preceding8606 global/generic load8607 atomic/8608 atomicrmw8609 with an equal or8610 wider sync scope8611 and memory ordering8612 stronger than8613 unordered (this is8614 termed the8615 fence-paired-atomic).8616 - s_waitcnt lgkmcnt(0)8617 must happen after8618 any preceding8619 local/generic load8620 atomic/atomicrmw8621 with an equal or8622 wider sync scope8623 and memory ordering8624 stronger than8625 unordered (this is8626 termed the8627 fence-paired-atomic).8628 - Must happen before8629 the following8630 buffer_wbinvl1_vol and8631 any following8632 global/generic8633 load/load8634 atomic/store/store8635 atomic/atomicrmw.8636 - Ensures any8637 following global8638 data read is no8639 older than the8640 value read by the8641 fence-paired-atomic.8642 8643 2. buffer_wbinvl1_vol8644 8645 - If not TgSplit execution8646 mode, omit.8647 - Ensures that8648 following8649 loads will not see8650 stale data.8651 8652 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) &8653 vmcnt(0)8654 8655 - If TgSplit execution mode,8656 omit lgkmcnt(0).8657 - If OpenCL and8658 address space is8659 not generic, omit8660 lgkmcnt(0).8661 - See :ref:`amdgpu-fence-as` for8662 more details on fencing specific8663 address spaces.8664 - Could be split into8665 separate s_waitcnt8666 vmcnt(0) and8667 s_waitcnt8668 lgkmcnt(0) to allow8669 them to be8670 independently moved8671 according to the8672 following rules.8673 - s_waitcnt vmcnt(0)8674 must happen after8675 any preceding8676 global/generic load8677 atomic/atomicrmw8678 with an equal or8679 wider sync scope8680 and memory ordering8681 stronger than8682 unordered (this is8683 termed the8684 fence-paired-atomic).8685 - s_waitcnt lgkmcnt(0)8686 must happen after8687 any preceding8688 local/generic load8689 atomic/atomicrmw8690 with an equal or8691 wider sync scope8692 and memory ordering8693 stronger than8694 unordered (this is8695 termed the8696 fence-paired-atomic).8697 - Must happen before8698 the following8699 buffer_wbinvl1_vol.8700 - Ensures that the8701 fence-paired atomic8702 has completed8703 before invalidating8704 the8705 cache. Therefore8706 any following8707 locations read must8708 be no older than8709 the value read by8710 the8711 fence-paired-atomic.8712 8713 2. buffer_wbinvl1_vol8714 8715 - Must happen before any8716 following global/generic8717 load/load8718 atomic/store/store8719 atomic/atomicrmw.8720 - Ensures that8721 following loads8722 will not see stale8723 global data.8724 8725 fence acquire - system *none* 1. s_waitcnt lgkmcnt(0) &8726 vmcnt(0)8727 8728 - If TgSplit execution mode,8729 omit lgkmcnt(0).8730 - If OpenCL and8731 address space is8732 not generic, omit8733 lgkmcnt(0).8734 - See :ref:`amdgpu-fence-as` for8735 more details on fencing specific8736 address spaces.8737 - Could be split into8738 separate s_waitcnt8739 vmcnt(0) and8740 s_waitcnt8741 lgkmcnt(0) to allow8742 them to be8743 independently moved8744 according to the8745 following rules.8746 - s_waitcnt vmcnt(0)8747 must happen after8748 any preceding8749 global/generic load8750 atomic/atomicrmw8751 with an equal or8752 wider sync scope8753 and memory ordering8754 stronger than8755 unordered (this is8756 termed the8757 fence-paired-atomic).8758 - s_waitcnt lgkmcnt(0)8759 must happen after8760 any preceding8761 local/generic load8762 atomic/atomicrmw8763 with an equal or8764 wider sync scope8765 and memory ordering8766 stronger than8767 unordered (this is8768 termed the8769 fence-paired-atomic).8770 - Must happen before8771 the following buffer_invl2 and8772 buffer_wbinvl1_vol.8773 - Ensures that the8774 fence-paired atomic8775 has completed8776 before invalidating8777 the8778 cache. Therefore8779 any following8780 locations read must8781 be no older than8782 the value read by8783 the8784 fence-paired-atomic.8785 8786 2. buffer_invl2;8787 buffer_wbinvl1_vol8788 8789 - Must happen before any8790 following global/generic8791 load/load8792 atomic/store/store8793 atomic/atomicrmw.8794 - Ensures that8795 following8796 loads will not see8797 stale L1 global data,8798 nor see stale L2 MTYPE8799 NC global data.8800 MTYPE RW and CC memory will8801 never be stale in L2 due to8802 the memory probes.8803 **Release Atomic**8804 ------------------------------------------------------------------------------------8805 store atomic release - singlethread - global 1. buffer/global/flat_store8806 - wavefront - generic8807 store atomic release - singlethread - local *If TgSplit execution mode,8808 - wavefront local address space cannot8809 be used.*8810 8811 1. ds_store8812 store atomic release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0)8813 - generic8814 - Use lgkmcnt(0) if not8815 TgSplit execution mode8816 and vmcnt(0) if TgSplit8817 execution mode.8818 - If OpenCL, omit lgkmcnt(0).8819 - s_waitcnt vmcnt(0)8820 must happen after8821 any preceding8822 global/generic load/store/8823 load atomic/store atomic/8824 atomicrmw.8825 - s_waitcnt lgkmcnt(0)8826 must happen after8827 any preceding8828 local/generic8829 load/store/load8830 atomic/store8831 atomic/atomicrmw.8832 - Must happen before8833 the following8834 store.8835 - Ensures that all8836 memory operations8837 have8838 completed before8839 performing the8840 store that is being8841 released.8842 8843 2. buffer/global/flat_store8844 store atomic release - workgroup - local *If TgSplit execution mode,8845 local address space cannot8846 be used.*8847 8848 1. ds_store8849 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) &8850 - generic vmcnt(0)8851 8852 - If TgSplit execution mode,8853 omit lgkmcnt(0).8854 - If OpenCL and8855 address space is8856 not generic, omit8857 lgkmcnt(0).8858 - Could be split into8859 separate s_waitcnt8860 vmcnt(0) and8861 s_waitcnt8862 lgkmcnt(0) to allow8863 them to be8864 independently moved8865 according to the8866 following rules.8867 - s_waitcnt vmcnt(0)8868 must happen after8869 any preceding8870 global/generic8871 load/store/load8872 atomic/store8873 atomic/atomicrmw.8874 - s_waitcnt lgkmcnt(0)8875 must happen after8876 any preceding8877 local/generic8878 load/store/load8879 atomic/store8880 atomic/atomicrmw.8881 - Must happen before8882 the following8883 store.8884 - Ensures that all8885 memory operations8886 to memory have8887 completed before8888 performing the8889 store that is being8890 released.8891 8892 2. buffer/global/flat_store8893 store atomic release - system - global 1. buffer_wbl28894 - generic8895 - Must happen before8896 following s_waitcnt.8897 - Performs L2 writeback to8898 ensure previous8899 global/generic8900 store/atomicrmw are8901 visible at system scope.8902 8903 2. s_waitcnt lgkmcnt(0) &8904 vmcnt(0)8905 8906 - If TgSplit execution mode,8907 omit lgkmcnt(0).8908 - If OpenCL and8909 address space is8910 not generic, omit8911 lgkmcnt(0).8912 - Could be split into8913 separate s_waitcnt8914 vmcnt(0) and8915 s_waitcnt8916 lgkmcnt(0) to allow8917 them to be8918 independently moved8919 according to the8920 following rules.8921 - s_waitcnt vmcnt(0)8922 must happen after any8923 preceding8924 global/generic8925 load/store/load8926 atomic/store8927 atomic/atomicrmw.8928 - s_waitcnt lgkmcnt(0)8929 must happen after any8930 preceding8931 local/generic8932 load/store/load8933 atomic/store8934 atomic/atomicrmw.8935 - Must happen before8936 the following8937 store.8938 - Ensures that all8939 memory operations8940 to memory and the L28941 writeback have8942 completed before8943 performing the8944 store that is being8945 released.8946 8947 3. buffer/global/flat_store8948 atomicrmw release - singlethread - global 1. buffer/global/flat_atomic8949 - wavefront - generic8950 atomicrmw release - singlethread - local *If TgSplit execution mode,8951 - wavefront local address space cannot8952 be used.*8953 8954 1. ds_atomic8955 atomicrmw release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0)8956 - generic8957 - Use lgkmcnt(0) if not8958 TgSplit execution mode8959 and vmcnt(0) if TgSplit8960 execution mode.8961 - If OpenCL, omit8962 lgkmcnt(0).8963 - s_waitcnt vmcnt(0)8964 must happen after8965 any preceding8966 global/generic load/store/8967 load atomic/store atomic/8968 atomicrmw.8969 - s_waitcnt lgkmcnt(0)8970 must happen after8971 any preceding8972 local/generic8973 load/store/load8974 atomic/store8975 atomic/atomicrmw.8976 - Must happen before8977 the following8978 atomicrmw.8979 - Ensures that all8980 memory operations8981 have8982 completed before8983 performing the8984 atomicrmw that is8985 being released.8986 8987 2. buffer/global/flat_atomic8988 atomicrmw release - workgroup - local *If TgSplit execution mode,8989 local address space cannot8990 be used.*8991 8992 1. ds_atomic8993 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) &8994 - generic vmcnt(0)8995 8996 - If TgSplit execution mode,8997 omit lgkmcnt(0).8998 - If OpenCL, omit8999 lgkmcnt(0).9000 - Could be split into9001 separate s_waitcnt9002 vmcnt(0) and9003 s_waitcnt9004 lgkmcnt(0) to allow9005 them to be9006 independently moved9007 according to the9008 following rules.9009 - s_waitcnt vmcnt(0)9010 must happen after9011 any preceding9012 global/generic9013 load/store/load9014 atomic/store9015 atomic/atomicrmw.9016 - s_waitcnt lgkmcnt(0)9017 must happen after9018 any preceding9019 local/generic9020 load/store/load9021 atomic/store9022 atomic/atomicrmw.9023 - Must happen before9024 the following9025 atomicrmw.9026 - Ensures that all9027 memory operations9028 to global and local9029 have completed9030 before performing9031 the atomicrmw that9032 is being released.9033 9034 2. buffer/global/flat_atomic9035 atomicrmw release - system - global 1. buffer_wbl29036 - generic9037 - Must happen before9038 following s_waitcnt.9039 - Performs L2 writeback to9040 ensure previous9041 global/generic9042 store/atomicrmw are9043 visible at system scope.9044 9045 2. s_waitcnt lgkmcnt(0) &9046 vmcnt(0)9047 9048 - If TgSplit execution mode,9049 omit lgkmcnt(0).9050 - If OpenCL, omit9051 lgkmcnt(0).9052 - Could be split into9053 separate s_waitcnt9054 vmcnt(0) and9055 s_waitcnt9056 lgkmcnt(0) to allow9057 them to be9058 independently moved9059 according to the9060 following rules.9061 - s_waitcnt vmcnt(0)9062 must happen after9063 any preceding9064 global/generic9065 load/store/load9066 atomic/store9067 atomic/atomicrmw.9068 - s_waitcnt lgkmcnt(0)9069 must happen after9070 any preceding9071 local/generic9072 load/store/load9073 atomic/store9074 atomic/atomicrmw.9075 - Must happen before9076 the following9077 atomicrmw.9078 - Ensures that all9079 memory operations9080 to memory and the L29081 writeback have9082 completed before9083 performing the9084 store that is being9085 released.9086 9087 3. buffer/global/flat_atomic9088 fence release - singlethread *none* *none*9089 - wavefront9090 fence release - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0)9091 9092 - Use lgkmcnt(0) if not9093 TgSplit execution mode9094 and vmcnt(0) if TgSplit9095 execution mode.9096 - If OpenCL and9097 address space is9098 not generic, omit9099 lgkmcnt(0).9100 - If OpenCL and9101 address space is9102 local, omit9103 vmcnt(0).9104 - See :ref:`amdgpu-fence-as` for9105 more details on fencing specific9106 address spaces.9107 - s_waitcnt vmcnt(0)9108 must happen after9109 any preceding9110 global/generic9111 load/store/9112 load atomic/store atomic/9113 atomicrmw.9114 - s_waitcnt lgkmcnt(0)9115 must happen after9116 any preceding9117 local/generic9118 load/load9119 atomic/store/store9120 atomic/atomicrmw.9121 - Must happen before9122 any following store9123 atomic/atomicrmw9124 with an equal or9125 wider sync scope9126 and memory ordering9127 stronger than9128 unordered (this is9129 termed the9130 fence-paired-atomic).9131 - Ensures that all9132 memory operations9133 have9134 completed before9135 performing the9136 following9137 fence-paired-atomic.9138 9139 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) &9140 vmcnt(0)9141 9142 - If TgSplit execution mode,9143 omit lgkmcnt(0).9144 - If OpenCL and9145 address space is9146 not generic, omit9147 lgkmcnt(0).9148 - If OpenCL and9149 address space is9150 local, omit9151 vmcnt(0).9152 - See :ref:`amdgpu-fence-as` for9153 more details on fencing specific9154 address spaces.9155 - Could be split into9156 separate s_waitcnt9157 vmcnt(0) and9158 s_waitcnt9159 lgkmcnt(0) to allow9160 them to be9161 independently moved9162 according to the9163 following rules.9164 - s_waitcnt vmcnt(0)9165 must happen after9166 any preceding9167 global/generic9168 load/store/load9169 atomic/store9170 atomic/atomicrmw.9171 - s_waitcnt lgkmcnt(0)9172 must happen after9173 any preceding9174 local/generic9175 load/store/load9176 atomic/store9177 atomic/atomicrmw.9178 - Must happen before9179 any following store9180 atomic/atomicrmw9181 with an equal or9182 wider sync scope9183 and memory ordering9184 stronger than9185 unordered (this is9186 termed the9187 fence-paired-atomic).9188 - Ensures that all9189 memory operations9190 have9191 completed before9192 performing the9193 following9194 fence-paired-atomic.9195 9196 fence release - system *none* 1. buffer_wbl29197 9198 - If OpenCL and9199 address space is9200 local, omit.9201 - Must happen before9202 following s_waitcnt.9203 - Performs L2 writeback to9204 ensure previous9205 global/generic9206 store/atomicrmw are9207 visible at system scope.9208 9209 2. s_waitcnt lgkmcnt(0) &9210 vmcnt(0)9211 9212 - If TgSplit execution mode,9213 omit lgkmcnt(0).9214 - If OpenCL and9215 address space is9216 not generic, omit9217 lgkmcnt(0).9218 - If OpenCL and9219 address space is9220 local, omit9221 vmcnt(0).9222 - See :ref:`amdgpu-fence-as` for9223 more details on fencing specific9224 address spaces.9225 - Could be split into9226 separate s_waitcnt9227 vmcnt(0) and9228 s_waitcnt9229 lgkmcnt(0) to allow9230 them to be9231 independently moved9232 according to the9233 following rules.9234 - s_waitcnt vmcnt(0)9235 must happen after9236 any preceding9237 global/generic9238 load/store/load9239 atomic/store9240 atomic/atomicrmw.9241 - s_waitcnt lgkmcnt(0)9242 must happen after9243 any preceding9244 local/generic9245 load/store/load9246 atomic/store9247 atomic/atomicrmw.9248 - Must happen before9249 any following store9250 atomic/atomicrmw9251 with an equal or9252 wider sync scope9253 and memory ordering9254 stronger than9255 unordered (this is9256 termed the9257 fence-paired-atomic).9258 - Ensures that all9259 memory operations9260 have9261 completed before9262 performing the9263 following9264 fence-paired-atomic.9265 9266 **Acquire-Release Atomic**9267 ------------------------------------------------------------------------------------9268 atomicrmw acq_rel - singlethread - global 1. buffer/global/flat_atomic9269 - wavefront - generic9270 atomicrmw acq_rel - singlethread - local *If TgSplit execution mode,9271 - wavefront local address space cannot9272 be used.*9273 9274 1. ds_atomic9275 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkm/vmcnt(0)9276 9277 - Use lgkmcnt(0) if not9278 TgSplit execution mode9279 and vmcnt(0) if TgSplit9280 execution mode.9281 - If OpenCL, omit9282 lgkmcnt(0).9283 - Must happen after9284 any preceding9285 local/generic9286 load/store/load9287 atomic/store9288 atomic/atomicrmw.9289 - s_waitcnt vmcnt(0)9290 must happen after9291 any preceding9292 global/generic load/store/9293 load atomic/store atomic/9294 atomicrmw.9295 - s_waitcnt lgkmcnt(0)9296 must happen after9297 any preceding9298 local/generic9299 load/store/load9300 atomic/store9301 atomic/atomicrmw.9302 - Must happen before9303 the following9304 atomicrmw.9305 - Ensures that all9306 memory operations9307 have9308 completed before9309 performing the9310 atomicrmw that is9311 being released.9312 9313 2. buffer/global_atomic9314 3. s_waitcnt vmcnt(0)9315 9316 - If not TgSplit execution9317 mode, omit.9318 - Must happen before9319 the following9320 buffer_wbinvl1_vol.9321 - Ensures any9322 following global9323 data read is no9324 older than the9325 atomicrmw value9326 being acquired.9327 9328 4. buffer_wbinvl1_vol9329 9330 - If not TgSplit execution9331 mode, omit.9332 - Ensures that9333 following9334 loads will not see9335 stale data.9336 9337 atomicrmw acq_rel - workgroup - local *If TgSplit execution mode,9338 local address space cannot9339 be used.*9340 9341 1. ds_atomic9342 2. s_waitcnt lgkmcnt(0)9343 9344 - If OpenCL, omit.9345 - Must happen before9346 any following9347 global/generic9348 load/load9349 atomic/store/store9350 atomic/atomicrmw.9351 - Ensures any9352 following global9353 data read is no9354 older than the local load9355 atomic value being9356 acquired.9357 9358 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkm/vmcnt(0)9359 9360 - Use lgkmcnt(0) if not9361 TgSplit execution mode9362 and vmcnt(0) if TgSplit9363 execution mode.9364 - If OpenCL, omit9365 lgkmcnt(0).9366 - s_waitcnt vmcnt(0)9367 must happen after9368 any preceding9369 global/generic load/store/9370 load atomic/store atomic/9371 atomicrmw.9372 - s_waitcnt lgkmcnt(0)9373 must happen after9374 any preceding9375 local/generic9376 load/store/load9377 atomic/store9378 atomic/atomicrmw.9379 - Must happen before9380 the following9381 atomicrmw.9382 - Ensures that all9383 memory operations9384 have9385 completed before9386 performing the9387 atomicrmw that is9388 being released.9389 9390 2. flat_atomic9391 3. s_waitcnt lgkmcnt(0) &9392 vmcnt(0)9393 9394 - If not TgSplit execution9395 mode, omit vmcnt(0).9396 - If OpenCL, omit9397 lgkmcnt(0).9398 - Must happen before9399 the following9400 buffer_wbinvl1_vol and9401 any following9402 global/generic9403 load/load9404 atomic/store/store9405 atomic/atomicrmw.9406 - Ensures any9407 following global9408 data read is no9409 older than a local load9410 atomic value being9411 acquired.9412 9413 3. buffer_wbinvl1_vol9414 9415 - If not TgSplit execution9416 mode, omit.9417 - Ensures that9418 following9419 loads will not see9420 stale data.9421 9422 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) &9423 vmcnt(0)9424 9425 - If TgSplit execution mode,9426 omit lgkmcnt(0).9427 - If OpenCL, omit9428 lgkmcnt(0).9429 - Could be split into9430 separate s_waitcnt9431 vmcnt(0) and9432 s_waitcnt9433 lgkmcnt(0) to allow9434 them to be9435 independently moved9436 according to the9437 following rules.9438 - s_waitcnt vmcnt(0)9439 must happen after9440 any preceding9441 global/generic9442 load/store/load9443 atomic/store9444 atomic/atomicrmw.9445 - s_waitcnt lgkmcnt(0)9446 must happen after9447 any preceding9448 local/generic9449 load/store/load9450 atomic/store9451 atomic/atomicrmw.9452 - Must happen before9453 the following9454 atomicrmw.9455 - Ensures that all9456 memory operations9457 to global have9458 completed before9459 performing the9460 atomicrmw that is9461 being released.9462 9463 2. buffer/global_atomic9464 3. s_waitcnt vmcnt(0)9465 9466 - Must happen before9467 following9468 buffer_wbinvl1_vol.9469 - Ensures the9470 atomicrmw has9471 completed before9472 invalidating the9473 cache.9474 9475 4. buffer_wbinvl1_vol9476 9477 - Must happen before9478 any following9479 global/generic9480 load/load9481 atomic/atomicrmw.9482 - Ensures that9483 following loads9484 will not see stale9485 global data.9486 9487 atomicrmw acq_rel - system - global 1. buffer_wbl29488 9489 - Must happen before9490 following s_waitcnt.9491 - Performs L2 writeback to9492 ensure previous9493 global/generic9494 store/atomicrmw are9495 visible at system scope.9496 9497 2. s_waitcnt lgkmcnt(0) &9498 vmcnt(0)9499 9500 - If TgSplit execution mode,9501 omit lgkmcnt(0).9502 - If OpenCL, omit9503 lgkmcnt(0).9504 - Could be split into9505 separate s_waitcnt9506 vmcnt(0) and9507 s_waitcnt9508 lgkmcnt(0) to allow9509 them to be9510 independently moved9511 according to the9512 following rules.9513 - s_waitcnt vmcnt(0)9514 must happen after9515 any preceding9516 global/generic9517 load/store/load9518 atomic/store9519 atomic/atomicrmw.9520 - s_waitcnt lgkmcnt(0)9521 must happen after9522 any preceding9523 local/generic9524 load/store/load9525 atomic/store9526 atomic/atomicrmw.9527 - Must happen before9528 the following9529 atomicrmw.9530 - Ensures that all9531 memory operations9532 to global and L2 writeback9533 have completed before9534 performing the9535 atomicrmw that is9536 being released.9537 9538 3. buffer/global_atomic9539 4. s_waitcnt vmcnt(0)9540 9541 - Must happen before9542 following buffer_invl2 and9543 buffer_wbinvl1_vol.9544 - Ensures the9545 atomicrmw has9546 completed before9547 invalidating the9548 caches.9549 9550 5. buffer_invl2;9551 buffer_wbinvl1_vol9552 9553 - Must happen before9554 any following9555 global/generic9556 load/load9557 atomic/atomicrmw.9558 - Ensures that9559 following9560 loads will not see9561 stale L1 global data,9562 nor see stale L2 MTYPE9563 NC global data.9564 MTYPE RW and CC memory will9565 never be stale in L2 due to9566 the memory probes.9567 9568 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) &9569 vmcnt(0)9570 9571 - If TgSplit execution mode,9572 omit lgkmcnt(0).9573 - If OpenCL, omit9574 lgkmcnt(0).9575 - Could be split into9576 separate s_waitcnt9577 vmcnt(0) and9578 s_waitcnt9579 lgkmcnt(0) to allow9580 them to be9581 independently moved9582 according to the9583 following rules.9584 - s_waitcnt vmcnt(0)9585 must happen after9586 any preceding9587 global/generic9588 load/store/load9589 atomic/store9590 atomic/atomicrmw.9591 - s_waitcnt lgkmcnt(0)9592 must happen after9593 any preceding9594 local/generic9595 load/store/load9596 atomic/store9597 atomic/atomicrmw.9598 - Must happen before9599 the following9600 atomicrmw.9601 - Ensures that all9602 memory operations9603 to global have9604 completed before9605 performing the9606 atomicrmw that is9607 being released.9608 9609 2. flat_atomic9610 3. s_waitcnt vmcnt(0) &9611 lgkmcnt(0)9612 9613 - If TgSplit execution mode,9614 omit lgkmcnt(0).9615 - If OpenCL, omit9616 lgkmcnt(0).9617 - Must happen before9618 following9619 buffer_wbinvl1_vol.9620 - Ensures the9621 atomicrmw has9622 completed before9623 invalidating the9624 cache.9625 9626 4. buffer_wbinvl1_vol9627 9628 - Must happen before9629 any following9630 global/generic9631 load/load9632 atomic/atomicrmw.9633 - Ensures that9634 following loads9635 will not see stale9636 global data.9637 9638 atomicrmw acq_rel - system - generic 1. buffer_wbl29639 9640 - Must happen before9641 following s_waitcnt.9642 - Performs L2 writeback to9643 ensure previous9644 global/generic9645 store/atomicrmw are9646 visible at system scope.9647 9648 2. s_waitcnt lgkmcnt(0) &9649 vmcnt(0)9650 9651 - If TgSplit execution mode,9652 omit lgkmcnt(0).9653 - If OpenCL, omit9654 lgkmcnt(0).9655 - Could be split into9656 separate s_waitcnt9657 vmcnt(0) and9658 s_waitcnt9659 lgkmcnt(0) to allow9660 them to be9661 independently moved9662 according to the9663 following rules.9664 - s_waitcnt vmcnt(0)9665 must happen after9666 any preceding9667 global/generic9668 load/store/load9669 atomic/store9670 atomic/atomicrmw.9671 - s_waitcnt lgkmcnt(0)9672 must happen after9673 any preceding9674 local/generic9675 load/store/load9676 atomic/store9677 atomic/atomicrmw.9678 - Must happen before9679 the following9680 atomicrmw.9681 - Ensures that all9682 memory operations9683 to global and L2 writeback9684 have completed before9685 performing the9686 atomicrmw that is9687 being released.9688 9689 3. flat_atomic9690 4. s_waitcnt vmcnt(0) &9691 lgkmcnt(0)9692 9693 - If TgSplit execution mode,9694 omit lgkmcnt(0).9695 - If OpenCL, omit9696 lgkmcnt(0).9697 - Must happen before9698 following buffer_invl2 and9699 buffer_wbinvl1_vol.9700 - Ensures the9701 atomicrmw has9702 completed before9703 invalidating the9704 caches.9705 9706 5. buffer_invl2;9707 buffer_wbinvl1_vol9708 9709 - Must happen before9710 any following9711 global/generic9712 load/load9713 atomic/atomicrmw.9714 - Ensures that9715 following9716 loads will not see9717 stale L1 global data,9718 nor see stale L2 MTYPE9719 NC global data.9720 MTYPE RW and CC memory will9721 never be stale in L2 due to9722 the memory probes.9723 9724 fence acq_rel - singlethread *none* *none*9725 - wavefront9726 fence acq_rel - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0)9727 9728 - Use lgkmcnt(0) if not9729 TgSplit execution mode9730 and vmcnt(0) if TgSplit9731 execution mode.9732 - If OpenCL and9733 address space is9734 not generic, omit9735 lgkmcnt(0).9736 - If OpenCL and9737 address space is9738 local, omit9739 vmcnt(0).9740 - However,9741 since LLVM9742 currently has no9743 address space on9744 the fence need to9745 conservatively9746 always generate9747 (see comment for9748 previous fence).9749 - s_waitcnt vmcnt(0)9750 must happen after9751 any preceding9752 global/generic9753 load/store/9754 load atomic/store atomic/9755 atomicrmw.9756 - s_waitcnt lgkmcnt(0)9757 must happen after9758 any preceding9759 local/generic9760 load/load9761 atomic/store/store9762 atomic/atomicrmw.9763 - Must happen before9764 any following9765 global/generic9766 load/load9767 atomic/store/store9768 atomic/atomicrmw.9769 - Ensures that all9770 memory operations9771 have9772 completed before9773 performing any9774 following global9775 memory operations.9776 - Ensures that the9777 preceding9778 local/generic load9779 atomic/atomicrmw9780 with an equal or9781 wider sync scope9782 and memory ordering9783 stronger than9784 unordered (this is9785 termed the9786 acquire-fence-paired-atomic)9787 has completed9788 before following9789 global memory9790 operations. This9791 satisfies the9792 requirements of9793 acquire.9794 - Ensures that all9795 previous memory9796 operations have9797 completed before a9798 following9799 local/generic store9800 atomic/atomicrmw9801 with an equal or9802 wider sync scope9803 and memory ordering9804 stronger than9805 unordered (this is9806 termed the9807 release-fence-paired-atomic).9808 This satisfies the9809 requirements of9810 release.9811 - Must happen before9812 the following9813 buffer_wbinvl1_vol.9814 - Ensures that the9815 acquire-fence-paired9816 atomic has completed9817 before invalidating9818 the9819 cache. Therefore9820 any following9821 locations read must9822 be no older than9823 the value read by9824 the9825 acquire-fence-paired-atomic.9826 9827 2. buffer_wbinvl1_vol9828 9829 - If not TgSplit execution9830 mode, omit.9831 - Ensures that9832 following9833 loads will not see9834 stale data.9835 9836 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) &9837 vmcnt(0)9838 9839 - If TgSplit execution mode,9840 omit lgkmcnt(0).9841 - If OpenCL and9842 address space is9843 not generic, omit9844 lgkmcnt(0).9845 - See :ref:`amdgpu-fence-as` for9846 more details on fencing specific9847 address spaces.9848 - Could be split into9849 separate s_waitcnt9850 vmcnt(0) and9851 s_waitcnt9852 lgkmcnt(0) to allow9853 them to be9854 independently moved9855 according to the9856 following rules.9857 - s_waitcnt vmcnt(0)9858 must happen after9859 any preceding9860 global/generic9861 load/store/load9862 atomic/store9863 atomic/atomicrmw.9864 - s_waitcnt lgkmcnt(0)9865 must happen after9866 any preceding9867 local/generic9868 load/store/load9869 atomic/store9870 atomic/atomicrmw.9871 - Must happen before9872 the following9873 buffer_wbinvl1_vol.9874 - Ensures that the9875 preceding9876 global/local/generic9877 load9878 atomic/atomicrmw9879 with an equal or9880 wider sync scope9881 and memory ordering9882 stronger than9883 unordered (this is9884 termed the9885 acquire-fence-paired-atomic)9886 has completed9887 before invalidating9888 the cache. This9889 satisfies the9890 requirements of9891 acquire.9892 - Ensures that all9893 previous memory9894 operations have9895 completed before a9896 following9897 global/local/generic9898 store9899 atomic/atomicrmw9900 with an equal or9901 wider sync scope9902 and memory ordering9903 stronger than9904 unordered (this is9905 termed the9906 release-fence-paired-atomic).9907 This satisfies the9908 requirements of9909 release.9910 9911 2. buffer_wbinvl1_vol9912 9913 - Must happen before9914 any following9915 global/generic9916 load/load9917 atomic/store/store9918 atomic/atomicrmw.9919 - Ensures that9920 following loads9921 will not see stale9922 global data. This9923 satisfies the9924 requirements of9925 acquire.9926 9927 fence acq_rel - system *none* 1. buffer_wbl29928 9929 - If OpenCL and9930 address space is9931 local, omit.9932 - Must happen before9933 following s_waitcnt.9934 - Performs L2 writeback to9935 ensure previous9936 global/generic9937 store/atomicrmw are9938 visible at system scope.9939 9940 2. s_waitcnt lgkmcnt(0) &9941 vmcnt(0)9942 9943 - If TgSplit execution mode,9944 omit lgkmcnt(0).9945 - If OpenCL and9946 address space is9947 not generic, omit9948 lgkmcnt(0).9949 - See :ref:`amdgpu-fence-as` for9950 more details on fencing specific9951 address spaces.9952 - Could be split into9953 separate s_waitcnt9954 vmcnt(0) and9955 s_waitcnt9956 lgkmcnt(0) to allow9957 them to be9958 independently moved9959 according to the9960 following rules.9961 - s_waitcnt vmcnt(0)9962 must happen after9963 any preceding9964 global/generic9965 load/store/load9966 atomic/store9967 atomic/atomicrmw.9968 - s_waitcnt lgkmcnt(0)9969 must happen after9970 any preceding9971 local/generic9972 load/store/load9973 atomic/store9974 atomic/atomicrmw.9975 - Must happen before9976 the following buffer_invl2 and9977 buffer_wbinvl1_vol.9978 - Ensures that the9979 preceding9980 global/local/generic9981 load9982 atomic/atomicrmw9983 with an equal or9984 wider sync scope9985 and memory ordering9986 stronger than9987 unordered (this is9988 termed the9989 acquire-fence-paired-atomic)9990 has completed9991 before invalidating9992 the cache. This9993 satisfies the9994 requirements of9995 acquire.9996 - Ensures that all9997 previous memory9998 operations have9999 completed before a10000 following10001 global/local/generic10002 store10003 atomic/atomicrmw10004 with an equal or10005 wider sync scope10006 and memory ordering10007 stronger than10008 unordered (this is10009 termed the10010 release-fence-paired-atomic).10011 This satisfies the10012 requirements of10013 release.10014 10015 3. buffer_invl2;10016 buffer_wbinvl1_vol10017 10018 - Must happen before10019 any following10020 global/generic10021 load/load10022 atomic/store/store10023 atomic/atomicrmw.10024 - Ensures that10025 following10026 loads will not see10027 stale L1 global data,10028 nor see stale L2 MTYPE10029 NC global data.10030 MTYPE RW and CC memory will10031 never be stale in L2 due to10032 the memory probes.10033 10034 **Sequential Consistent Atomic**10035 ------------------------------------------------------------------------------------10036 load atomic seq_cst - singlethread - global *Same as corresponding10037 - wavefront - local load atomic acquire,10038 - generic except must generate10039 all instructions even10040 for OpenCL.*10041 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkm/vmcnt(0)10042 - generic10043 - Use lgkmcnt(0) if not10044 TgSplit execution mode10045 and vmcnt(0) if TgSplit10046 execution mode.10047 - s_waitcnt lgkmcnt(0) must10048 happen after10049 preceding10050 local/generic load10051 atomic/store10052 atomic/atomicrmw10053 with memory10054 ordering of seq_cst10055 and with equal or10056 wider sync scope.10057 (Note that seq_cst10058 fences have their10059 own s_waitcnt10060 lgkmcnt(0) and so do10061 not need to be10062 considered.)10063 - s_waitcnt vmcnt(0)10064 must happen after10065 preceding10066 global/generic load10067 atomic/store10068 atomic/atomicrmw10069 with memory10070 ordering of seq_cst10071 and with equal or10072 wider sync scope.10073 (Note that seq_cst10074 fences have their10075 own s_waitcnt10076 vmcnt(0) and so do10077 not need to be10078 considered.)10079 - Ensures any10080 preceding10081 sequential10082 consistent global/local10083 memory instructions10084 have completed10085 before executing10086 this sequentially10087 consistent10088 instruction. This10089 prevents reordering10090 a seq_cst store10091 followed by a10092 seq_cst load. (Note10093 that seq_cst is10094 stronger than10095 acquire/release as10096 the reordering of10097 load acquire10098 followed by a store10099 release is10100 prevented by the10101 s_waitcnt of10102 the release, but10103 there is nothing10104 preventing a store10105 release followed by10106 load acquire from10107 completing out of10108 order. The s_waitcnt10109 could be placed after10110 seq_store or before10111 the seq_load. We10112 choose the load to10113 make the s_waitcnt be10114 as late as possible10115 so that the store10116 may have already10117 completed.)10118 10119 2. *Following10120 instructions same as10121 corresponding load10122 atomic acquire,10123 except must generate10124 all instructions even10125 for OpenCL.*10126 load atomic seq_cst - workgroup - local *If TgSplit execution mode,10127 local address space cannot10128 be used.*10129 10130 *Same as corresponding10131 load atomic acquire,10132 except must generate10133 all instructions even10134 for OpenCL.*10135 10136 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) &10137 - system - generic vmcnt(0)10138 10139 - If TgSplit execution mode,10140 omit lgkmcnt(0).10141 - Could be split into10142 separate s_waitcnt10143 vmcnt(0)10144 and s_waitcnt10145 lgkmcnt(0) to allow10146 them to be10147 independently moved10148 according to the10149 following rules.10150 - s_waitcnt lgkmcnt(0)10151 must happen after10152 preceding10153 global/generic load10154 atomic/store10155 atomic/atomicrmw10156 with memory10157 ordering of seq_cst10158 and with equal or10159 wider sync scope.10160 (Note that seq_cst10161 fences have their10162 own s_waitcnt10163 lgkmcnt(0) and so do10164 not need to be10165 considered.)10166 - s_waitcnt vmcnt(0)10167 must happen after10168 preceding10169 global/generic load10170 atomic/store10171 atomic/atomicrmw10172 with memory10173 ordering of seq_cst10174 and with equal or10175 wider sync scope.10176 (Note that seq_cst10177 fences have their10178 own s_waitcnt10179 vmcnt(0) and so do10180 not need to be10181 considered.)10182 - Ensures any10183 preceding10184 sequential10185 consistent global10186 memory instructions10187 have completed10188 before executing10189 this sequentially10190 consistent10191 instruction. This10192 prevents reordering10193 a seq_cst store10194 followed by a10195 seq_cst load. (Note10196 that seq_cst is10197 stronger than10198 acquire/release as10199 the reordering of10200 load acquire10201 followed by a store10202 release is10203 prevented by the10204 s_waitcnt of10205 the release, but10206 there is nothing10207 preventing a store10208 release followed by10209 load acquire from10210 completing out of10211 order. The s_waitcnt10212 could be placed after10213 seq_store or before10214 the seq_load. We10215 choose the load to10216 make the s_waitcnt be10217 as late as possible10218 so that the store10219 may have already10220 completed.)10221 10222 2. *Following10223 instructions same as10224 corresponding load10225 atomic acquire,10226 except must generate10227 all instructions even10228 for OpenCL.*10229 store atomic seq_cst - singlethread - global *Same as corresponding10230 - wavefront - local store atomic release,10231 - workgroup - generic except must generate10232 - agent all instructions even10233 - system for OpenCL.*10234 atomicrmw seq_cst - singlethread - global *Same as corresponding10235 - wavefront - local atomicrmw acq_rel,10236 - workgroup - generic except must generate10237 - agent all instructions even10238 - system for OpenCL.*10239 fence seq_cst - singlethread *none* *Same as corresponding10240 - wavefront fence acq_rel,10241 - workgroup except must generate10242 - agent all instructions even10243 - system for OpenCL.*10244 ============ ============ ============== ========== ================================10245 10246.. _amdgpu-amdhsa-memory-model-gfx942:10247 10248Memory Model GFX94210249+++++++++++++++++++10250 10251For GFX942:10252 10253* Each agent has multiple shader arrays (SA).10254* Each SA has multiple compute units (CU).10255* Each CU has multiple SIMDs that execute wavefronts.10256* The wavefronts for a single work-group are executed in the same CU but may be10257 executed by different SIMDs. The exception is when in tgsplit execution mode10258 when the wavefronts may be executed by different SIMDs in different CUs.10259* Each CU has a single LDS memory shared by the wavefronts of the work-groups10260 executing on it. The exception is when in tgsplit execution mode when no LDS10261 is allocated as wavefronts of the same work-group can be in different CUs.10262* All LDS operations of a CU are performed as wavefront wide operations in a10263 global order and involve no caching. Completion is reported to a wavefront in10264 execution order.10265* The LDS memory has multiple request queues shared by the SIMDs of a10266 CU. Therefore, the LDS operations performed by different wavefronts of a10267 work-group can be reordered relative to each other, which can result in10268 reordering the visibility of vector memory operations with respect to LDS10269 operations of other wavefronts in the same work-group. A ``s_waitcnt10270 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and10271 vector memory operations between wavefronts of a work-group, but not between10272 operations performed by the same wavefront.10273* The vector memory operations are performed as wavefront wide operations and10274 completion is reported to a wavefront in execution order. The exception is10275 that ``flat_load/store/atomic`` instructions can report out of vector memory10276 order if they access LDS memory, and out of LDS operation order if they access10277 global memory.10278* The vector memory operations access a single vector L1 cache shared by all10279 SIMDs a CU. Therefore:10280 10281 * No special action is required for coherence between the lanes of a single10282 wavefront.10283 10284 * No special action is required for coherence between wavefronts in the same10285 work-group since they execute on the same CU. The exception is when in10286 tgsplit execution mode as wavefronts of the same work-group can be in10287 different CUs and so a ``buffer_inv sc0`` is required which will invalidate10288 the L1 cache.10289 10290 * A ``buffer_inv sc0`` is required to invalidate the L1 cache for coherence10291 between wavefronts executing in different work-groups as they may be10292 executing on different CUs.10293 10294 * Atomic read-modify-write instructions implicitly bypass the L1 cache.10295 Therefore, they do not use the sc0 bit for coherence and instead use it to10296 indicate if the instruction returns the original value being updated. They10297 do use sc1 to indicate system or agent scope coherence.10298 10299* The scalar memory operations access a scalar L1 cache shared by all wavefronts10300 on a group of CUs. The scalar and vector L1 caches are not coherent. However,10301 scalar operations are used in a restricted way so do not impact the memory10302 model. See :ref:`amdgpu-amdhsa-memory-spaces`.10303* The vector and scalar memory operations use an L2 cache.10304 10305 * The gfx942 can be configured as a number of smaller agents with each having10306 a single L2 shared by all CUs on the same agent, or as fewer (possibly one)10307 larger agents with groups of CUs on each agent each sharing separate L210308 caches.10309 * The L2 cache has independent channels to service disjoint ranges of virtual10310 addresses.10311 * Each CU has a separate request queue per channel for its associated L2.10312 Therefore, the vector and scalar memory operations performed by wavefronts10313 executing with different L1 caches and the same L2 cache can be reordered10314 relative to each other.10315 * A ``s_waitcnt vmcnt(0)`` is required to ensure synchronization between10316 vector memory operations of different CUs. It ensures a previous vector10317 memory operation has completed before executing a subsequent vector memory10318 or LDS operation and so can be used to meet the requirements of acquire and10319 release.10320 * An L2 cache can be kept coherent with other L2 caches by using the MTYPE RW10321 (read-write) for memory local to the L2, and MTYPE NC (non-coherent) with10322 the PTE C-bit set for memory not local to the L2.10323 10324 * Any local memory cache lines will be automatically invalidated by writes10325 from CUs associated with other L2 caches, or writes from the CPU, due to10326 the cache probe caused by the PTE C-bit.10327 * XGMI accesses from the CPU to local memory may be cached on the CPU.10328 Subsequent access from the GPU will automatically invalidate or writeback10329 the CPU cache due to the L2 probe filter.10330 * To ensure coherence of local memory writes of CUs with different L1 caches10331 in the same agent a ``buffer_wbl2`` is required. It does nothing if the10332 agent is configured to have a single L2, or will writeback dirty L2 cache10333 lines if configured to have multiple L2 caches.10334 * To ensure coherence of local memory writes of CUs in different agents a10335 ``buffer_wbl2 sc1`` is required. It will writeback dirty L2 cache lines.10336 * To ensure coherence of local memory reads of CUs with different L1 caches10337 in the same agent a ``buffer_inv sc1`` is required. It does nothing if the10338 agent is configured to have a single L2, or will invalidate non-local L210339 cache lines if configured to have multiple L2 caches.10340 * To ensure coherence of local memory reads of CUs in different agents a10341 ``buffer_inv sc0 sc1`` is required. It will invalidate non-local L2 cache10342 lines if configured to have multiple L2 caches.10343 10344 * PCIe access from the GPU to the CPU can be kept coherent by using the MTYPE10345 UC (uncached) which bypasses the L2.10346 10347Scalar memory operations are only used to access memory that is proven to not10348change during the execution of the kernel dispatch. This includes constant10349address space and global address space for program scope ``const`` variables.10350Therefore, the kernel machine code does not have to maintain the scalar cache to10351ensure it is coherent with the vector caches. The scalar and vector caches are10352invalidated between kernel dispatches by CP since constant address space data10353may change between kernel dispatch executions. See10354:ref:`amdgpu-amdhsa-memory-spaces`.10355 10356The one exception is if scalar writes are used to spill SGPR registers. In this10357case the AMDGPU backend ensures the memory location used to spill is never10358accessed by vector memory operations at the same time. If scalar writes are used10359then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function10360return since the locations may be used for vector memory instructions by a10361future wavefront that uses the same scratch area, or a function call that10362creates a frame at the same address, respectively. There is no need for a10363``s_dcache_inv`` as all scalar writes are write-before-read in the same thread.10364 10365For kernarg backing memory:10366 10367* CP invalidates the L1 cache at the start of each kernel dispatch.10368* On dGPU over XGMI or PCIe the kernarg backing memory is allocated in host10369 memory accessed as MTYPE UC (uncached) to avoid needing to invalidate the L210370 cache. This also causes it to be treated as non-volatile and so is not10371 invalidated by ``*_vol``.10372* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and10373 so the L2 cache will be coherent with the CPU and other agents.10374 10375Scratch backing memory (which is used for the private address space) is accessed10376with MTYPE NC_NV (non-coherent non-volatile). Since the private address space is10377only accessed by a single thread, and is always write-before-read, there is10378never a need to invalidate these entries from the L1 cache. Hence all cache10379invalidates are done as ``*_vol`` to only invalidate the volatile cache lines.10380 10381The code sequences used to implement the memory model for GFX942 are defined in10382table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx942-table`.10383 10384 .. table:: AMDHSA Memory Model Code Sequences GFX94210385 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx942-table10386 10387 ============ ============ ============== ========== ================================10388 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code10389 Ordering Sync Scope Address GFX94210390 Space10391 ============ ============ ============== ========== ================================10392 **Non-Atomic**10393 ------------------------------------------------------------------------------------10394 load *none* *none* - global - !volatile & !nontemporal10395 - generic10396 - private 1. buffer/global/flat_load10397 - constant10398 - !volatile & nontemporal10399 10400 1. buffer/global/flat_load10401 nt=110402 10403 - volatile10404 10405 1. buffer/global/flat_load10406 sc0=1 sc1=110407 2. s_waitcnt vmcnt(0)10408 10409 - Must happen before10410 any following volatile10411 global/generic10412 load/store.10413 - Ensures that10414 volatile10415 operations to10416 different10417 addresses will not10418 be reordered by10419 hardware.10420 10421 load *none* *none* - local 1. ds_load10422 store *none* *none* - global - !volatile & !nontemporal10423 - generic10424 - private 1. GFX94210425 - constant buffer/global/flat_store10426 10427 - !volatile & nontemporal10428 10429 1. GFX94210430 buffer/global/flat_store10431 nt=110432 10433 - volatile10434 10435 1. buffer/global/flat_store10436 sc0=1 sc1=110437 2. s_waitcnt vmcnt(0)10438 10439 - Must happen before10440 any following volatile10441 global/generic10442 load/store.10443 - Ensures that10444 volatile10445 operations to10446 different10447 addresses will not10448 be reordered by10449 hardware.10450 10451 store *none* *none* - local 1. ds_store10452 **Unordered Atomic**10453 ------------------------------------------------------------------------------------10454 load atomic unordered *any* *any* *Same as non-atomic*.10455 store atomic unordered *any* *any* *Same as non-atomic*.10456 atomicrmw unordered *any* *any* *Same as monotonic atomic*.10457 **Monotonic Atomic**10458 ------------------------------------------------------------------------------------10459 load atomic monotonic - singlethread - global 1. buffer/global/flat_load10460 - wavefront - generic10461 load atomic monotonic - workgroup - global 1. buffer/global/flat_load10462 - generic sc0=110463 load atomic monotonic - singlethread - local *If TgSplit execution mode,10464 - wavefront local address space cannot10465 - workgroup be used.*10466 10467 1. ds_load10468 load atomic monotonic - agent - global 1. buffer/global/flat_load10469 - generic sc1=110470 load atomic monotonic - system - global 1. buffer/global/flat_load10471 - generic sc0=1 sc1=110472 store atomic monotonic - singlethread - global 1. buffer/global/flat_store10473 - wavefront - generic10474 store atomic monotonic - workgroup - global 1. buffer/global/flat_store10475 - generic sc0=110476 store atomic monotonic - agent - global 1. buffer/global/flat_store10477 - generic sc1=110478 store atomic monotonic - system - global 1. buffer/global/flat_store10479 - generic sc0=1 sc1=110480 store atomic monotonic - singlethread - local *If TgSplit execution mode,10481 - wavefront local address space cannot10482 - workgroup be used.*10483 10484 1. ds_store10485 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic10486 - wavefront - generic10487 - workgroup10488 - agent10489 atomicrmw monotonic - system - global 1. buffer/global/flat_atomic10490 - generic sc1=110491 atomicrmw monotonic - singlethread - local *If TgSplit execution mode,10492 - wavefront local address space cannot10493 - workgroup be used.*10494 10495 1. ds_atomic10496 **Acquire Atomic**10497 ------------------------------------------------------------------------------------10498 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load10499 - wavefront - local10500 - generic10501 load atomic acquire - workgroup - global 1. buffer/global_load sc0=110502 2. s_waitcnt vmcnt(0)10503 10504 - If not TgSplit execution10505 mode, omit.10506 - Must happen before the10507 following buffer_inv.10508 10509 3. buffer_inv sc0=110510 10511 - If not TgSplit execution10512 mode, omit.10513 - Must happen before10514 any following10515 global/generic10516 load/load10517 atomic/store/store10518 atomic/atomicrmw.10519 - Ensures that10520 following10521 loads will not see10522 stale data.10523 10524 load atomic acquire - workgroup - local *If TgSplit execution mode,10525 local address space cannot10526 be used.*10527 10528 1. ds_load10529 2. s_waitcnt lgkmcnt(0)10530 10531 - If OpenCL, omit.10532 - Must happen before10533 any following10534 global/generic10535 load/load10536 atomic/store/store10537 atomic/atomicrmw.10538 - Ensures any10539 following global10540 data read is no10541 older than the local load10542 atomic value being10543 acquired.10544 10545 load atomic acquire - workgroup - generic 1. flat_load sc0=110546 2. s_waitcnt lgkm/vmcnt(0)10547 10548 - Use lgkmcnt(0) if not10549 TgSplit execution mode10550 and vmcnt(0) if TgSplit10551 execution mode.10552 - If OpenCL, omit lgkmcnt(0).10553 - Must happen before10554 the following10555 buffer_inv and any10556 following global/generic10557 load/load10558 atomic/store/store10559 atomic/atomicrmw.10560 - Ensures any10561 following global10562 data read is no10563 older than a local load10564 atomic value being10565 acquired.10566 10567 3. buffer_inv sc0=110568 10569 - If not TgSplit execution10570 mode, omit.10571 - Ensures that10572 following10573 loads will not see10574 stale data.10575 10576 load atomic acquire - agent - global 1. buffer/global_load10577 sc1=110578 2. s_waitcnt vmcnt(0)10579 10580 - Must happen before10581 following10582 buffer_inv.10583 - Ensures the load10584 has completed10585 before invalidating10586 the cache.10587 10588 3. buffer_inv sc1=110589 10590 - Must happen before10591 any following10592 global/generic10593 load/load10594 atomic/atomicrmw.10595 - Ensures that10596 following10597 loads will not see10598 stale global data.10599 10600 load atomic acquire - system - global 1. buffer/global/flat_load10601 sc0=1 sc1=110602 2. s_waitcnt vmcnt(0)10603 10604 - Must happen before10605 following10606 buffer_inv.10607 - Ensures the load10608 has completed10609 before invalidating10610 the cache.10611 10612 3. buffer_inv sc0=1 sc1=110613 10614 - Must happen before10615 any following10616 global/generic10617 load/load10618 atomic/atomicrmw.10619 - Ensures that10620 following10621 loads will not see10622 stale MTYPE NC global data.10623 MTYPE RW and CC memory will10624 never be stale due to the10625 memory probes.10626 10627 load atomic acquire - agent - generic 1. flat_load sc1=110628 2. s_waitcnt vmcnt(0) &10629 lgkmcnt(0)10630 10631 - If TgSplit execution mode,10632 omit lgkmcnt(0).10633 - If OpenCL omit10634 lgkmcnt(0).10635 - Must happen before10636 following10637 buffer_inv.10638 - Ensures the flat_load10639 has completed10640 before invalidating10641 the cache.10642 10643 3. buffer_inv sc1=110644 10645 - Must happen before10646 any following10647 global/generic10648 load/load10649 atomic/atomicrmw.10650 - Ensures that10651 following loads10652 will not see stale10653 global data.10654 10655 load atomic acquire - system - generic 1. flat_load sc0=1 sc1=110656 2. s_waitcnt vmcnt(0) &10657 lgkmcnt(0)10658 10659 - If TgSplit execution mode,10660 omit lgkmcnt(0).10661 - If OpenCL omit10662 lgkmcnt(0).10663 - Must happen before10664 the following10665 buffer_inv.10666 - Ensures the flat_load10667 has completed10668 before invalidating10669 the caches.10670 10671 3. buffer_inv sc0=1 sc1=110672 10673 - Must happen before10674 any following10675 global/generic10676 load/load10677 atomic/atomicrmw.10678 - Ensures that10679 following10680 loads will not see10681 stale MTYPE NC global data.10682 MTYPE RW and CC memory will10683 never be stale due to the10684 memory probes.10685 10686 atomicrmw acquire - singlethread - global 1. buffer/global/flat_atomic10687 - wavefront - generic10688 atomicrmw acquire - singlethread - local *If TgSplit execution mode,10689 - wavefront local address space cannot10690 be used.*10691 10692 1. ds_atomic10693 atomicrmw acquire - workgroup - global 1. buffer/global_atomic10694 2. s_waitcnt vmcnt(0)10695 10696 - If not TgSplit execution10697 mode, omit.10698 - Must happen before the10699 following buffer_inv.10700 - Ensures the atomicrmw10701 has completed10702 before invalidating10703 the cache.10704 10705 3. buffer_inv sc0=110706 10707 - If not TgSplit execution10708 mode, omit.10709 - Must happen before10710 any following10711 global/generic10712 load/load10713 atomic/atomicrmw.10714 - Ensures that10715 following loads10716 will not see stale10717 global data.10718 10719 atomicrmw acquire - workgroup - local *If TgSplit execution mode,10720 local address space cannot10721 be used.*10722 10723 1. ds_atomic10724 2. s_waitcnt lgkmcnt(0)10725 10726 - If OpenCL, omit.10727 - Must happen before10728 any following10729 global/generic10730 load/load10731 atomic/store/store10732 atomic/atomicrmw.10733 - Ensures any10734 following global10735 data read is no10736 older than the local10737 atomicrmw value10738 being acquired.10739 10740 atomicrmw acquire - workgroup - generic 1. flat_atomic10741 2. s_waitcnt lgkm/vmcnt(0)10742 10743 - Use lgkmcnt(0) if not10744 TgSplit execution mode10745 and vmcnt(0) if TgSplit10746 execution mode.10747 - If OpenCL, omit lgkmcnt(0).10748 - Must happen before10749 the following10750 buffer_inv and10751 any following10752 global/generic10753 load/load10754 atomic/store/store10755 atomic/atomicrmw.10756 - Ensures any10757 following global10758 data read is no10759 older than a local10760 atomicrmw value10761 being acquired.10762 10763 3. buffer_inv sc0=110764 10765 - If not TgSplit execution10766 mode, omit.10767 - Ensures that10768 following10769 loads will not see10770 stale data.10771 10772 atomicrmw acquire - agent - global 1. buffer/global_atomic10773 2. s_waitcnt vmcnt(0)10774 10775 - Must happen before10776 following10777 buffer_inv.10778 - Ensures the10779 atomicrmw has10780 completed before10781 invalidating the10782 cache.10783 10784 3. buffer_inv sc1=110785 10786 - Must happen before10787 any following10788 global/generic10789 load/load10790 atomic/atomicrmw.10791 - Ensures that10792 following loads10793 will not see stale10794 global data.10795 10796 atomicrmw acquire - system - global 1. buffer/global_atomic10797 sc1=110798 2. s_waitcnt vmcnt(0)10799 10800 - Must happen before10801 following10802 buffer_inv.10803 - Ensures the10804 atomicrmw has10805 completed before10806 invalidating the10807 caches.10808 10809 3. buffer_inv sc0=1 sc1=110810 10811 - Must happen before10812 any following10813 global/generic10814 load/load10815 atomic/atomicrmw.10816 - Ensures that10817 following10818 loads will not see10819 stale MTYPE NC global data.10820 MTYPE RW and CC memory will10821 never be stale due to the10822 memory probes.10823 10824 atomicrmw acquire - agent - generic 1. flat_atomic10825 2. s_waitcnt vmcnt(0) &10826 lgkmcnt(0)10827 10828 - If TgSplit execution mode,10829 omit lgkmcnt(0).10830 - If OpenCL, omit10831 lgkmcnt(0).10832 - Must happen before10833 following10834 buffer_inv.10835 - Ensures the10836 atomicrmw has10837 completed before10838 invalidating the10839 cache.10840 10841 3. buffer_inv sc1=110842 10843 - Must happen before10844 any following10845 global/generic10846 load/load10847 atomic/atomicrmw.10848 - Ensures that10849 following loads10850 will not see stale10851 global data.10852 10853 atomicrmw acquire - system - generic 1. flat_atomic sc1=110854 2. s_waitcnt vmcnt(0) &10855 lgkmcnt(0)10856 10857 - If TgSplit execution mode,10858 omit lgkmcnt(0).10859 - If OpenCL, omit10860 lgkmcnt(0).10861 - Must happen before10862 following10863 buffer_inv.10864 - Ensures the10865 atomicrmw has10866 completed before10867 invalidating the10868 caches.10869 10870 3. buffer_inv sc0=1 sc1=110871 10872 - Must happen before10873 any following10874 global/generic10875 load/load10876 atomic/atomicrmw.10877 - Ensures that10878 following10879 loads will not see10880 stale MTYPE NC global data.10881 MTYPE RW and CC memory will10882 never be stale due to the10883 memory probes.10884 10885 fence acquire - singlethread *none* *none*10886 - wavefront10887 fence acquire - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0)10888 10889 - Use lgkmcnt(0) if not10890 TgSplit execution mode10891 and vmcnt(0) if TgSplit10892 execution mode.10893 - If OpenCL and10894 address space is10895 not generic, omit10896 lgkmcnt(0).10897 - If OpenCL and10898 address space is10899 local, omit10900 vmcnt(0).10901 - See :ref:`amdgpu-fence-as` for10902 more details on fencing specific10903 address spaces.10904 - s_waitcnt vmcnt(0)10905 must happen after10906 any preceding10907 global/generic load10908 atomic/10909 atomicrmw10910 with an equal or10911 wider sync scope10912 and memory ordering10913 stronger than10914 unordered (this is10915 termed the10916 fence-paired-atomic).10917 - s_waitcnt lgkmcnt(0)10918 must happen after10919 any preceding10920 local/generic load10921 atomic/atomicrmw10922 with an equal or10923 wider sync scope10924 and memory ordering10925 stronger than10926 unordered (this is10927 termed the10928 fence-paired-atomic).10929 - Must happen before10930 the following10931 buffer_inv and10932 any following10933 global/generic10934 load/load10935 atomic/store/store10936 atomic/atomicrmw.10937 - Ensures any10938 following global10939 data read is no10940 older than the10941 value read by the10942 fence-paired-atomic.10943 10944 3. buffer_inv sc0=110945 10946 - If not TgSplit execution10947 mode, omit.10948 - Ensures that10949 following10950 loads will not see10951 stale data.10952 10953 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) &10954 vmcnt(0)10955 10956 - If TgSplit execution mode,10957 omit lgkmcnt(0).10958 - If OpenCL and10959 address space is10960 not generic, omit10961 lgkmcnt(0).10962 - See :ref:`amdgpu-fence-as` for10963 more details on fencing specific10964 address spaces.10965 - Could be split into10966 separate s_waitcnt10967 vmcnt(0) and10968 s_waitcnt10969 lgkmcnt(0) to allow10970 them to be10971 independently moved10972 according to the10973 following rules.10974 - s_waitcnt vmcnt(0)10975 must happen after10976 any preceding10977 global/generic load10978 atomic/atomicrmw10979 with an equal or10980 wider sync scope10981 and memory ordering10982 stronger than10983 unordered (this is10984 termed the10985 fence-paired-atomic).10986 - s_waitcnt lgkmcnt(0)10987 must happen after10988 any preceding10989 local/generic load10990 atomic/atomicrmw10991 with an equal or10992 wider sync scope10993 and memory ordering10994 stronger than10995 unordered (this is10996 termed the10997 fence-paired-atomic).10998 - Must happen before10999 the following11000 buffer_inv.11001 - Ensures that the11002 fence-paired atomic11003 has completed11004 before invalidating11005 the11006 cache. Therefore11007 any following11008 locations read must11009 be no older than11010 the value read by11011 the11012 fence-paired-atomic.11013 11014 2. buffer_inv sc1=111015 11016 - Must happen before any11017 following global/generic11018 load/load11019 atomic/store/store11020 atomic/atomicrmw.11021 - Ensures that11022 following loads11023 will not see stale11024 global data.11025 11026 fence acquire - system *none* 1. s_waitcnt lgkmcnt(0) &11027 vmcnt(0)11028 11029 - If TgSplit execution mode,11030 omit lgkmcnt(0).11031 - If OpenCL and11032 address space is11033 not generic, omit11034 lgkmcnt(0).11035 - See :ref:`amdgpu-fence-as` for11036 more details on fencing specific11037 address spaces.11038 - Could be split into11039 separate s_waitcnt11040 vmcnt(0) and11041 s_waitcnt11042 lgkmcnt(0) to allow11043 them to be11044 independently moved11045 according to the11046 following rules.11047 - s_waitcnt vmcnt(0)11048 must happen after11049 any preceding11050 global/generic load11051 atomic/atomicrmw11052 with an equal or11053 wider sync scope11054 and memory ordering11055 stronger than11056 unordered (this is11057 termed the11058 fence-paired-atomic).11059 - s_waitcnt lgkmcnt(0)11060 must happen after11061 any preceding11062 local/generic load11063 atomic/atomicrmw11064 with an equal or11065 wider sync scope11066 and memory ordering11067 stronger than11068 unordered (this is11069 termed the11070 fence-paired-atomic).11071 - Must happen before11072 the following11073 buffer_inv.11074 - Ensures that the11075 fence-paired atomic11076 has completed11077 before invalidating11078 the11079 cache. Therefore11080 any following11081 locations read must11082 be no older than11083 the value read by11084 the11085 fence-paired-atomic.11086 11087 2. buffer_inv sc0=1 sc1=111088 11089 - Must happen before any11090 following global/generic11091 load/load11092 atomic/store/store11093 atomic/atomicrmw.11094 - Ensures that11095 following loads11096 will not see stale11097 global data.11098 11099 **Release Atomic**11100 ------------------------------------------------------------------------------------11101 store atomic release - singlethread - global 1. GFX94211102 - wavefront - generic buffer/global/flat_store11103 11104 store atomic release - singlethread - local *If TgSplit execution mode,11105 - wavefront local address space cannot11106 be used.*11107 11108 1. ds_store11109 store atomic release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0)11110 - generic11111 - Use lgkmcnt(0) if not11112 TgSplit execution mode11113 and vmcnt(0) if TgSplit11114 execution mode.11115 - If OpenCL, omit lgkmcnt(0).11116 - s_waitcnt vmcnt(0)11117 must happen after11118 any preceding11119 global/generic load/store/11120 load atomic/store atomic/11121 atomicrmw.11122 - s_waitcnt lgkmcnt(0)11123 must happen after11124 any preceding11125 local/generic11126 load/store/load11127 atomic/store11128 atomic/atomicrmw.11129 - Must happen before11130 the following11131 store.11132 - Ensures that all11133 memory operations11134 have11135 completed before11136 performing the11137 store that is being11138 released.11139 11140 2. GFX94211141 buffer/global/flat_store11142 sc0=111143 store atomic release - workgroup - local *If TgSplit execution mode,11144 local address space cannot11145 be used.*11146 11147 1. ds_store11148 store atomic release - agent - global 1. buffer_wbl2 sc1=111149 - generic11150 - Must happen before11151 following s_waitcnt.11152 - Performs L2 writeback to11153 ensure previous11154 global/generic11155 store/atomicrmw are11156 visible at agent scope.11157 11158 2. s_waitcnt lgkmcnt(0) &11159 vmcnt(0)11160 11161 - If TgSplit execution mode,11162 omit lgkmcnt(0).11163 - If OpenCL and11164 address space is11165 not generic, omit11166 lgkmcnt(0).11167 - Could be split into11168 separate s_waitcnt11169 vmcnt(0) and11170 s_waitcnt11171 lgkmcnt(0) to allow11172 them to be11173 independently moved11174 according to the11175 following rules.11176 - s_waitcnt vmcnt(0)11177 must happen after11178 any preceding11179 global/generic11180 load/store/load11181 atomic/store11182 atomic/atomicrmw.11183 - s_waitcnt lgkmcnt(0)11184 must happen after11185 any preceding11186 local/generic11187 load/store/load11188 atomic/store11189 atomic/atomicrmw.11190 - Must happen before11191 the following11192 store.11193 - Ensures that all11194 memory operations11195 to memory have11196 completed before11197 performing the11198 store that is being11199 released.11200 11201 3. GFX94211202 buffer/global/flat_store11203 sc1=111204 store atomic release - system - global 1. buffer_wbl2 sc0=1 sc1=111205 - generic11206 - Must happen before11207 following s_waitcnt.11208 - Performs L2 writeback to11209 ensure previous11210 global/generic11211 store/atomicrmw are11212 visible at system scope.11213 11214 2. s_waitcnt lgkmcnt(0) &11215 vmcnt(0)11216 11217 - If TgSplit execution mode,11218 omit lgkmcnt(0).11219 - If OpenCL and11220 address space is11221 not generic, omit11222 lgkmcnt(0).11223 - Could be split into11224 separate s_waitcnt11225 vmcnt(0) and11226 s_waitcnt11227 lgkmcnt(0) to allow11228 them to be11229 independently moved11230 according to the11231 following rules.11232 - s_waitcnt vmcnt(0)11233 must happen after any11234 preceding11235 global/generic11236 load/store/load11237 atomic/store11238 atomic/atomicrmw.11239 - s_waitcnt lgkmcnt(0)11240 must happen after any11241 preceding11242 local/generic11243 load/store/load11244 atomic/store11245 atomic/atomicrmw.11246 - Must happen before11247 the following11248 store.11249 - Ensures that all11250 memory operations11251 to memory and the L211252 writeback have11253 completed before11254 performing the11255 store that is being11256 released.11257 11258 3. buffer/global/flat_store11259 sc0=1 sc1=111260 atomicrmw release - singlethread - global 1. buffer/global/flat_atomic11261 - wavefront - generic11262 atomicrmw release - singlethread - local *If TgSplit execution mode,11263 - wavefront local address space cannot11264 be used.*11265 11266 1. ds_atomic11267 atomicrmw release - workgroup - global 1. s_waitcnt lgkm/vmcnt(0)11268 - generic11269 - Use lgkmcnt(0) if not11270 TgSplit execution mode11271 and vmcnt(0) if TgSplit11272 execution mode.11273 - If OpenCL, omit11274 lgkmcnt(0).11275 - s_waitcnt vmcnt(0)11276 must happen after11277 any preceding11278 global/generic load/store/11279 load atomic/store atomic/11280 atomicrmw.11281 - s_waitcnt lgkmcnt(0)11282 must happen after11283 any preceding11284 local/generic11285 load/store/load11286 atomic/store11287 atomic/atomicrmw.11288 - Must happen before11289 the following11290 atomicrmw.11291 - Ensures that all11292 memory operations11293 have11294 completed before11295 performing the11296 atomicrmw that is11297 being released.11298 11299 2. buffer/global/flat_atomic sc0=111300 atomicrmw release - workgroup - local *If TgSplit execution mode,11301 local address space cannot11302 be used.*11303 11304 1. ds_atomic11305 atomicrmw release - agent - global 1. buffer_wbl2 sc1=111306 - generic11307 - Must happen before11308 following s_waitcnt.11309 - Performs L2 writeback to11310 ensure previous11311 global/generic11312 store/atomicrmw are11313 visible at agent scope.11314 11315 2. s_waitcnt lgkmcnt(0) &11316 vmcnt(0)11317 11318 - If TgSplit execution mode,11319 omit lgkmcnt(0).11320 - If OpenCL, omit11321 lgkmcnt(0).11322 - Could be split into11323 separate s_waitcnt11324 vmcnt(0) and11325 s_waitcnt11326 lgkmcnt(0) to allow11327 them to be11328 independently moved11329 according to the11330 following rules.11331 - s_waitcnt vmcnt(0)11332 must happen after11333 any preceding11334 global/generic11335 load/store/load11336 atomic/store11337 atomic/atomicrmw.11338 - s_waitcnt lgkmcnt(0)11339 must happen after11340 any preceding11341 local/generic11342 load/store/load11343 atomic/store11344 atomic/atomicrmw.11345 - Must happen before11346 the following11347 atomicrmw.11348 - Ensures that all11349 memory operations11350 to global and local11351 have completed11352 before performing11353 the atomicrmw that11354 is being released.11355 11356 3. buffer/global/flat_atomic sc1=111357 atomicrmw release - system - global 1. buffer_wbl2 sc0=1 sc1=111358 - generic11359 - Must happen before11360 following s_waitcnt.11361 - Performs L2 writeback to11362 ensure previous11363 global/generic11364 store/atomicrmw are11365 visible at system scope.11366 11367 2. s_waitcnt lgkmcnt(0) &11368 vmcnt(0)11369 11370 - If TgSplit execution mode,11371 omit lgkmcnt(0).11372 - If OpenCL, omit11373 lgkmcnt(0).11374 - Could be split into11375 separate s_waitcnt11376 vmcnt(0) and11377 s_waitcnt11378 lgkmcnt(0) to allow11379 them to be11380 independently moved11381 according to the11382 following rules.11383 - s_waitcnt vmcnt(0)11384 must happen after11385 any preceding11386 global/generic11387 load/store/load11388 atomic/store11389 atomic/atomicrmw.11390 - s_waitcnt lgkmcnt(0)11391 must happen after11392 any preceding11393 local/generic11394 load/store/load11395 atomic/store11396 atomic/atomicrmw.11397 - Must happen before11398 the following11399 atomicrmw.11400 - Ensures that all11401 memory operations11402 to memory and the L211403 writeback have11404 completed before11405 performing the11406 store that is being11407 released.11408 11409 3. buffer/global/flat_atomic11410 sc0=1 sc1=111411 fence release - singlethread *none* *none*11412 - wavefront11413 fence release - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0)11414 11415 - Use lgkmcnt(0) if not11416 TgSplit execution mode11417 and vmcnt(0) if TgSplit11418 execution mode.11419 - If OpenCL and11420 address space is11421 not generic, omit11422 lgkmcnt(0).11423 - If OpenCL and11424 address space is11425 local, omit11426 vmcnt(0).11427 - See :ref:`amdgpu-fence-as` for11428 more details on fencing specific11429 address spaces.11430 - s_waitcnt vmcnt(0)11431 must happen after11432 any preceding11433 global/generic11434 load/store/11435 load atomic/store atomic/11436 atomicrmw.11437 - s_waitcnt lgkmcnt(0)11438 must happen after11439 any preceding11440 local/generic11441 load/load11442 atomic/store/store11443 atomic/atomicrmw.11444 - Must happen before11445 any following store11446 atomic/atomicrmw11447 with an equal or11448 wider sync scope11449 and memory ordering11450 stronger than11451 unordered (this is11452 termed the11453 fence-paired-atomic).11454 - Ensures that all11455 memory operations11456 have11457 completed before11458 performing the11459 following11460 fence-paired-atomic.11461 11462 fence release - agent *none* 1. buffer_wbl2 sc1=111463 11464 - If OpenCL and11465 address space is11466 local, omit.11467 - Must happen before11468 following s_waitcnt.11469 - Performs L2 writeback to11470 ensure previous11471 global/generic11472 store/atomicrmw are11473 visible at agent scope.11474 11475 2. s_waitcnt lgkmcnt(0) &11476 vmcnt(0)11477 11478 - If TgSplit execution mode,11479 omit lgkmcnt(0).11480 - If OpenCL and11481 address space is11482 not generic, omit11483 lgkmcnt(0).11484 - If OpenCL and11485 address space is11486 local, omit11487 vmcnt(0).11488 - See :ref:`amdgpu-fence-as` for11489 more details on fencing specific11490 address spaces.11491 - Could be split into11492 separate s_waitcnt11493 vmcnt(0) and11494 s_waitcnt11495 lgkmcnt(0) to allow11496 them to be11497 independently moved11498 according to the11499 following rules.11500 - s_waitcnt vmcnt(0)11501 must happen after11502 any preceding11503 global/generic11504 load/store/load11505 atomic/store11506 atomic/atomicrmw.11507 - s_waitcnt lgkmcnt(0)11508 must happen after11509 any preceding11510 local/generic11511 load/store/load11512 atomic/store11513 atomic/atomicrmw.11514 - Must happen before11515 any following store11516 atomic/atomicrmw11517 with an equal or11518 wider sync scope11519 and memory ordering11520 stronger than11521 unordered (this is11522 termed the11523 fence-paired-atomic).11524 - Ensures that all11525 memory operations11526 have11527 completed before11528 performing the11529 following11530 fence-paired-atomic.11531 11532 fence release - system *none* 1. buffer_wbl2 sc0=1 sc1=111533 11534 - Must happen before11535 following s_waitcnt.11536 - Performs L2 writeback to11537 ensure previous11538 global/generic11539 store/atomicrmw are11540 visible at system scope.11541 11542 2. s_waitcnt lgkmcnt(0) &11543 vmcnt(0)11544 11545 - If TgSplit execution mode,11546 omit lgkmcnt(0).11547 - If OpenCL and11548 address space is11549 not generic, omit11550 lgkmcnt(0).11551 - If OpenCL and11552 address space is11553 local, omit11554 vmcnt(0).11555 - See :ref:`amdgpu-fence-as` for11556 more details on fencing specific11557 address spaces.11558 - Could be split into11559 separate s_waitcnt11560 vmcnt(0) and11561 s_waitcnt11562 lgkmcnt(0) to allow11563 them to be11564 independently moved11565 according to the11566 following rules.11567 - s_waitcnt vmcnt(0)11568 must happen after11569 any preceding11570 global/generic11571 load/store/load11572 atomic/store11573 atomic/atomicrmw.11574 - s_waitcnt lgkmcnt(0)11575 must happen after11576 any preceding11577 local/generic11578 load/store/load11579 atomic/store11580 atomic/atomicrmw.11581 - Must happen before11582 any following store11583 atomic/atomicrmw11584 with an equal or11585 wider sync scope11586 and memory ordering11587 stronger than11588 unordered (this is11589 termed the11590 fence-paired-atomic).11591 - Ensures that all11592 memory operations11593 have11594 completed before11595 performing the11596 following11597 fence-paired-atomic.11598 11599 **Acquire-Release Atomic**11600 ------------------------------------------------------------------------------------11601 atomicrmw acq_rel - singlethread - global 1. buffer/global/flat_atomic11602 - wavefront - generic11603 atomicrmw acq_rel - singlethread - local *If TgSplit execution mode,11604 - wavefront local address space cannot11605 be used.*11606 11607 1. ds_atomic11608 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkm/vmcnt(0)11609 11610 - Use lgkmcnt(0) if not11611 TgSplit execution mode11612 and vmcnt(0) if TgSplit11613 execution mode.11614 - If OpenCL, omit11615 lgkmcnt(0).11616 - Must happen after11617 any preceding11618 local/generic11619 load/store/load11620 atomic/store11621 atomic/atomicrmw.11622 - s_waitcnt vmcnt(0)11623 must happen after11624 any preceding11625 global/generic load/store/11626 load atomic/store atomic/11627 atomicrmw.11628 - s_waitcnt lgkmcnt(0)11629 must happen after11630 any preceding11631 local/generic11632 load/store/load11633 atomic/store11634 atomic/atomicrmw.11635 - Must happen before11636 the following11637 atomicrmw.11638 - Ensures that all11639 memory operations11640 have11641 completed before11642 performing the11643 atomicrmw that is11644 being released.11645 11646 2. buffer/global_atomic11647 3. s_waitcnt vmcnt(0)11648 11649 - If not TgSplit execution11650 mode, omit.11651 - Must happen before11652 the following11653 buffer_inv.11654 - Ensures any11655 following global11656 data read is no11657 older than the11658 atomicrmw value11659 being acquired.11660 11661 4. buffer_inv sc0=111662 11663 - If not TgSplit execution11664 mode, omit.11665 - Ensures that11666 following11667 loads will not see11668 stale data.11669 11670 atomicrmw acq_rel - workgroup - local *If TgSplit execution mode,11671 local address space cannot11672 be used.*11673 11674 1. ds_atomic11675 2. s_waitcnt lgkmcnt(0)11676 11677 - If OpenCL, omit.11678 - Must happen before11679 any following11680 global/generic11681 load/load11682 atomic/store/store11683 atomic/atomicrmw.11684 - Ensures any11685 following global11686 data read is no11687 older than the local load11688 atomic value being11689 acquired.11690 11691 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkm/vmcnt(0)11692 11693 - Use lgkmcnt(0) if not11694 TgSplit execution mode11695 and vmcnt(0) if TgSplit11696 execution mode.11697 - If OpenCL, omit11698 lgkmcnt(0).11699 - s_waitcnt vmcnt(0)11700 must happen after11701 any preceding11702 global/generic load/store/11703 load atomic/store atomic/11704 atomicrmw.11705 - s_waitcnt lgkmcnt(0)11706 must happen after11707 any preceding11708 local/generic11709 load/store/load11710 atomic/store11711 atomic/atomicrmw.11712 - Must happen before11713 the following11714 atomicrmw.11715 - Ensures that all11716 memory operations11717 have11718 completed before11719 performing the11720 atomicrmw that is11721 being released.11722 11723 2. flat_atomic11724 3. s_waitcnt lgkmcnt(0) &11725 vmcnt(0)11726 11727 - If not TgSplit execution11728 mode, omit vmcnt(0).11729 - If OpenCL, omit11730 lgkmcnt(0).11731 - Must happen before11732 the following11733 buffer_inv and11734 any following11735 global/generic11736 load/load11737 atomic/store/store11738 atomic/atomicrmw.11739 - Ensures any11740 following global11741 data read is no11742 older than a local load11743 atomic value being11744 acquired.11745 11746 3. buffer_inv sc0=111747 11748 - If not TgSplit execution11749 mode, omit.11750 - Ensures that11751 following11752 loads will not see11753 stale data.11754 11755 atomicrmw acq_rel - agent - global 1. buffer_wbl2 sc1=111756 11757 - Must happen before11758 following s_waitcnt.11759 - Performs L2 writeback to11760 ensure previous11761 global/generic11762 store/atomicrmw are11763 visible at agent scope.11764 11765 2. s_waitcnt lgkmcnt(0) &11766 vmcnt(0)11767 11768 - If TgSplit execution mode,11769 omit lgkmcnt(0).11770 - If OpenCL, omit11771 lgkmcnt(0).11772 - Could be split into11773 separate s_waitcnt11774 vmcnt(0) and11775 s_waitcnt11776 lgkmcnt(0) to allow11777 them to be11778 independently moved11779 according to the11780 following rules.11781 - s_waitcnt vmcnt(0)11782 must happen after11783 any preceding11784 global/generic11785 load/store/load11786 atomic/store11787 atomic/atomicrmw.11788 - s_waitcnt lgkmcnt(0)11789 must happen after11790 any preceding11791 local/generic11792 load/store/load11793 atomic/store11794 atomic/atomicrmw.11795 - Must happen before11796 the following11797 atomicrmw.11798 - Ensures that all11799 memory operations11800 to global have11801 completed before11802 performing the11803 atomicrmw that is11804 being released.11805 11806 3. buffer/global_atomic11807 4. s_waitcnt vmcnt(0)11808 11809 - Must happen before11810 following11811 buffer_inv.11812 - Ensures the11813 atomicrmw has11814 completed before11815 invalidating the11816 cache.11817 11818 5. buffer_inv sc1=111819 11820 - Must happen before11821 any following11822 global/generic11823 load/load11824 atomic/atomicrmw.11825 - Ensures that11826 following loads11827 will not see stale11828 global data.11829 11830 atomicrmw acq_rel - system - global 1. buffer_wbl2 sc0=1 sc1=111831 11832 - Must happen before11833 following s_waitcnt.11834 - Performs L2 writeback to11835 ensure previous11836 global/generic11837 store/atomicrmw are11838 visible at system scope.11839 11840 2. s_waitcnt lgkmcnt(0) &11841 vmcnt(0)11842 11843 - If TgSplit execution mode,11844 omit lgkmcnt(0).11845 - If OpenCL, omit11846 lgkmcnt(0).11847 - Could be split into11848 separate s_waitcnt11849 vmcnt(0) and11850 s_waitcnt11851 lgkmcnt(0) to allow11852 them to be11853 independently moved11854 according to the11855 following rules.11856 - s_waitcnt vmcnt(0)11857 must happen after11858 any preceding11859 global/generic11860 load/store/load11861 atomic/store11862 atomic/atomicrmw.11863 - s_waitcnt lgkmcnt(0)11864 must happen after11865 any preceding11866 local/generic11867 load/store/load11868 atomic/store11869 atomic/atomicrmw.11870 - Must happen before11871 the following11872 atomicrmw.11873 - Ensures that all11874 memory operations11875 to global and L2 writeback11876 have completed before11877 performing the11878 atomicrmw that is11879 being released.11880 11881 3. buffer/global_atomic11882 sc1=111883 4. s_waitcnt vmcnt(0)11884 11885 - Must happen before11886 following11887 buffer_inv.11888 - Ensures the11889 atomicrmw has11890 completed before11891 invalidating the11892 caches.11893 11894 5. buffer_inv sc0=1 sc1=111895 11896 - Must happen before11897 any following11898 global/generic11899 load/load11900 atomic/atomicrmw.11901 - Ensures that11902 following loads11903 will not see stale11904 MTYPE NC global data.11905 MTYPE RW and CC memory will11906 never be stale due to the11907 memory probes.11908 11909 atomicrmw acq_rel - agent - generic 1. buffer_wbl2 sc1=111910 11911 - Must happen before11912 following s_waitcnt.11913 - Performs L2 writeback to11914 ensure previous11915 global/generic11916 store/atomicrmw are11917 visible at agent scope.11918 11919 2. s_waitcnt lgkmcnt(0) &11920 vmcnt(0)11921 11922 - If TgSplit execution mode,11923 omit lgkmcnt(0).11924 - If OpenCL, omit11925 lgkmcnt(0).11926 - Could be split into11927 separate s_waitcnt11928 vmcnt(0) and11929 s_waitcnt11930 lgkmcnt(0) to allow11931 them to be11932 independently moved11933 according to the11934 following rules.11935 - s_waitcnt vmcnt(0)11936 must happen after11937 any preceding11938 global/generic11939 load/store/load11940 atomic/store11941 atomic/atomicrmw.11942 - s_waitcnt lgkmcnt(0)11943 must happen after11944 any preceding11945 local/generic11946 load/store/load11947 atomic/store11948 atomic/atomicrmw.11949 - Must happen before11950 the following11951 atomicrmw.11952 - Ensures that all11953 memory operations11954 to global have11955 completed before11956 performing the11957 atomicrmw that is11958 being released.11959 11960 3. flat_atomic11961 4. s_waitcnt vmcnt(0) &11962 lgkmcnt(0)11963 11964 - If TgSplit execution mode,11965 omit lgkmcnt(0).11966 - If OpenCL, omit11967 lgkmcnt(0).11968 - Must happen before11969 following11970 buffer_inv.11971 - Ensures the11972 atomicrmw has11973 completed before11974 invalidating the11975 cache.11976 11977 5. buffer_inv sc1=111978 11979 - Must happen before11980 any following11981 global/generic11982 load/load11983 atomic/atomicrmw.11984 - Ensures that11985 following loads11986 will not see stale11987 global data.11988 11989 atomicrmw acq_rel - system - generic 1. buffer_wbl2 sc0=1 sc1=111990 11991 - Must happen before11992 following s_waitcnt.11993 - Performs L2 writeback to11994 ensure previous11995 global/generic11996 store/atomicrmw are11997 visible at system scope.11998 11999 2. s_waitcnt lgkmcnt(0) &12000 vmcnt(0)12001 12002 - If TgSplit execution mode,12003 omit lgkmcnt(0).12004 - If OpenCL, omit12005 lgkmcnt(0).12006 - Could be split into12007 separate s_waitcnt12008 vmcnt(0) and12009 s_waitcnt12010 lgkmcnt(0) to allow12011 them to be12012 independently moved12013 according to the12014 following rules.12015 - s_waitcnt vmcnt(0)12016 must happen after12017 any preceding12018 global/generic12019 load/store/load12020 atomic/store12021 atomic/atomicrmw.12022 - s_waitcnt lgkmcnt(0)12023 must happen after12024 any preceding12025 local/generic12026 load/store/load12027 atomic/store12028 atomic/atomicrmw.12029 - Must happen before12030 the following12031 atomicrmw.12032 - Ensures that all12033 memory operations12034 to global and L2 writeback12035 have completed before12036 performing the12037 atomicrmw that is12038 being released.12039 12040 3. flat_atomic sc1=112041 4. s_waitcnt vmcnt(0) &12042 lgkmcnt(0)12043 12044 - If TgSplit execution mode,12045 omit lgkmcnt(0).12046 - If OpenCL, omit12047 lgkmcnt(0).12048 - Must happen before12049 following12050 buffer_inv.12051 - Ensures the12052 atomicrmw has12053 completed before12054 invalidating the12055 caches.12056 12057 5. buffer_inv sc0=1 sc1=112058 12059 - Must happen before12060 any following12061 global/generic12062 load/load12063 atomic/atomicrmw.12064 - Ensures that12065 following loads12066 will not see stale12067 MTYPE NC global data.12068 MTYPE RW and CC memory will12069 never be stale due to the12070 memory probes.12071 12072 fence acq_rel - singlethread *none* *none*12073 - wavefront12074 fence acq_rel - workgroup *none* 1. s_waitcnt lgkm/vmcnt(0)12075 12076 - Use lgkmcnt(0) if not12077 TgSplit execution mode12078 and vmcnt(0) if TgSplit12079 execution mode.12080 - If OpenCL and12081 address space is12082 not generic, omit12083 lgkmcnt(0).12084 - If OpenCL and12085 address space is12086 local, omit12087 vmcnt(0).12088 - However,12089 since LLVM12090 currently has no12091 address space on12092 the fence need to12093 conservatively12094 always generate12095 (see comment for12096 previous fence).12097 - s_waitcnt vmcnt(0)12098 must happen after12099 any preceding12100 global/generic12101 load/store/12102 load atomic/store atomic/12103 atomicrmw.12104 - s_waitcnt lgkmcnt(0)12105 must happen after12106 any preceding12107 local/generic12108 load/load12109 atomic/store/store12110 atomic/atomicrmw.12111 - Must happen before12112 any following12113 global/generic12114 load/load12115 atomic/store/store12116 atomic/atomicrmw.12117 - Ensures that all12118 memory operations12119 have12120 completed before12121 performing any12122 following global12123 memory operations.12124 - Ensures that the12125 preceding12126 local/generic load12127 atomic/atomicrmw12128 with an equal or12129 wider sync scope12130 and memory ordering12131 stronger than12132 unordered (this is12133 termed the12134 acquire-fence-paired-atomic)12135 has completed12136 before following12137 global memory12138 operations. This12139 satisfies the12140 requirements of12141 acquire.12142 - Ensures that all12143 previous memory12144 operations have12145 completed before a12146 following12147 local/generic store12148 atomic/atomicrmw12149 with an equal or12150 wider sync scope12151 and memory ordering12152 stronger than12153 unordered (this is12154 termed the12155 release-fence-paired-atomic).12156 This satisfies the12157 requirements of12158 release.12159 - Must happen before12160 the following12161 buffer_inv.12162 - Ensures that the12163 acquire-fence-paired12164 atomic has completed12165 before invalidating12166 the12167 cache. Therefore12168 any following12169 locations read must12170 be no older than12171 the value read by12172 the12173 acquire-fence-paired-atomic.12174 12175 3. buffer_inv sc0=112176 12177 - If not TgSplit execution12178 mode, omit.12179 - Ensures that12180 following12181 loads will not see12182 stale data.12183 12184 fence acq_rel - agent *none* 1. buffer_wbl2 sc1=112185 12186 - If OpenCL and12187 address space is12188 local, omit.12189 - Must happen before12190 following s_waitcnt.12191 - Performs L2 writeback to12192 ensure previous12193 global/generic12194 store/atomicrmw are12195 visible at agent scope.12196 12197 2. s_waitcnt lgkmcnt(0) &12198 vmcnt(0)12199 12200 - If TgSplit execution mode,12201 omit lgkmcnt(0).12202 - If OpenCL and12203 address space is12204 not generic, omit12205 lgkmcnt(0).12206 - See :ref:`amdgpu-fence-as` for12207 more details on fencing specific12208 address spaces.12209 - Could be split into12210 separate s_waitcnt12211 vmcnt(0) and12212 s_waitcnt12213 lgkmcnt(0) to allow12214 them to be12215 independently moved12216 according to the12217 following rules.12218 - s_waitcnt vmcnt(0)12219 must happen after12220 any preceding12221 global/generic12222 load/store/load12223 atomic/store12224 atomic/atomicrmw.12225 - s_waitcnt lgkmcnt(0)12226 must happen after12227 any preceding12228 local/generic12229 load/store/load12230 atomic/store12231 atomic/atomicrmw.12232 - Must happen before12233 the following12234 buffer_inv.12235 - Ensures that the12236 preceding12237 global/local/generic12238 load12239 atomic/atomicrmw12240 with an equal or12241 wider sync scope12242 and memory ordering12243 stronger than12244 unordered (this is12245 termed the12246 acquire-fence-paired-atomic)12247 has completed12248 before invalidating12249 the cache. This12250 satisfies the12251 requirements of12252 acquire.12253 - Ensures that all12254 previous memory12255 operations have12256 completed before a12257 following12258 global/local/generic12259 store12260 atomic/atomicrmw12261 with an equal or12262 wider sync scope12263 and memory ordering12264 stronger than12265 unordered (this is12266 termed the12267 release-fence-paired-atomic).12268 This satisfies the12269 requirements of12270 release.12271 12272 3. buffer_inv sc1=112273 12274 - Must happen before12275 any following12276 global/generic12277 load/load12278 atomic/store/store12279 atomic/atomicrmw.12280 - Ensures that12281 following loads12282 will not see stale12283 global data. This12284 satisfies the12285 requirements of12286 acquire.12287 12288 fence acq_rel - system *none* 1. buffer_wbl2 sc0=1 sc1=112289 12290 - If OpenCL and12291 address space is12292 local, omit.12293 - Must happen before12294 following s_waitcnt.12295 - Performs L2 writeback to12296 ensure previous12297 global/generic12298 store/atomicrmw are12299 visible at system scope.12300 12301 1. s_waitcnt lgkmcnt(0) &12302 vmcnt(0)12303 12304 - If TgSplit execution mode,12305 omit lgkmcnt(0).12306 - If OpenCL and12307 address space is12308 not generic, omit12309 lgkmcnt(0).12310 - See :ref:`amdgpu-fence-as` for12311 more details on fencing specific12312 address spaces.12313 - Could be split into12314 separate s_waitcnt12315 vmcnt(0) and12316 s_waitcnt12317 lgkmcnt(0) to allow12318 them to be12319 independently moved12320 according to the12321 following rules.12322 - s_waitcnt vmcnt(0)12323 must happen after12324 any preceding12325 global/generic12326 load/store/load12327 atomic/store12328 atomic/atomicrmw.12329 - s_waitcnt lgkmcnt(0)12330 must happen after12331 any preceding12332 local/generic12333 load/store/load12334 atomic/store12335 atomic/atomicrmw.12336 - Must happen before12337 the following12338 buffer_inv.12339 - Ensures that the12340 preceding12341 global/local/generic12342 load12343 atomic/atomicrmw12344 with an equal or12345 wider sync scope12346 and memory ordering12347 stronger than12348 unordered (this is12349 termed the12350 acquire-fence-paired-atomic)12351 has completed12352 before invalidating12353 the cache. This12354 satisfies the12355 requirements of12356 acquire.12357 - Ensures that all12358 previous memory12359 operations have12360 completed before a12361 following12362 global/local/generic12363 store12364 atomic/atomicrmw12365 with an equal or12366 wider sync scope12367 and memory ordering12368 stronger than12369 unordered (this is12370 termed the12371 release-fence-paired-atomic).12372 This satisfies the12373 requirements of12374 release.12375 12376 2. buffer_inv sc0=1 sc1=112377 12378 - Must happen before12379 any following12380 global/generic12381 load/load12382 atomic/store/store12383 atomic/atomicrmw.12384 - Ensures that12385 following loads12386 will not see stale12387 MTYPE NC global data.12388 MTYPE RW and CC memory will12389 never be stale due to the12390 memory probes.12391 12392 **Sequential Consistent Atomic**12393 ------------------------------------------------------------------------------------12394 load atomic seq_cst - singlethread - global *Same as corresponding12395 - wavefront - local load atomic acquire,12396 - generic except must generate12397 all instructions even12398 for OpenCL.*12399 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkm/vmcnt(0)12400 - generic12401 - Use lgkmcnt(0) if not12402 TgSplit execution mode12403 and vmcnt(0) if TgSplit12404 execution mode.12405 - s_waitcnt lgkmcnt(0) must12406 happen after12407 preceding12408 local/generic load12409 atomic/store12410 atomic/atomicrmw12411 with memory12412 ordering of seq_cst12413 and with equal or12414 wider sync scope.12415 (Note that seq_cst12416 fences have their12417 own s_waitcnt12418 lgkmcnt(0) and so do12419 not need to be12420 considered.)12421 - s_waitcnt vmcnt(0)12422 must happen after12423 preceding12424 global/generic load12425 atomic/store12426 atomic/atomicrmw12427 with memory12428 ordering of seq_cst12429 and with equal or12430 wider sync scope.12431 (Note that seq_cst12432 fences have their12433 own s_waitcnt12434 vmcnt(0) and so do12435 not need to be12436 considered.)12437 - Ensures any12438 preceding12439 sequential12440 consistent global/local12441 memory instructions12442 have completed12443 before executing12444 this sequentially12445 consistent12446 instruction. This12447 prevents reordering12448 a seq_cst store12449 followed by a12450 seq_cst load. (Note12451 that seq_cst is12452 stronger than12453 acquire/release as12454 the reordering of12455 load acquire12456 followed by a store12457 release is12458 prevented by the12459 s_waitcnt of12460 the release, but12461 there is nothing12462 preventing a store12463 release followed by12464 load acquire from12465 completing out of12466 order. The s_waitcnt12467 could be placed after12468 seq_store or before12469 the seq_load. We12470 choose the load to12471 make the s_waitcnt be12472 as late as possible12473 so that the store12474 may have already12475 completed.)12476 12477 2. *Following12478 instructions same as12479 corresponding load12480 atomic acquire,12481 except must generate12482 all instructions even12483 for OpenCL.*12484 load atomic seq_cst - workgroup - local *If TgSplit execution mode,12485 local address space cannot12486 be used.*12487 12488 *Same as corresponding12489 load atomic acquire,12490 except must generate12491 all instructions even12492 for OpenCL.*12493 12494 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) &12495 - system - generic vmcnt(0)12496 12497 - If TgSplit execution mode,12498 omit lgkmcnt(0).12499 - Could be split into12500 separate s_waitcnt12501 vmcnt(0)12502 and s_waitcnt12503 lgkmcnt(0) to allow12504 them to be12505 independently moved12506 according to the12507 following rules.12508 - s_waitcnt lgkmcnt(0)12509 must happen after12510 preceding12511 global/generic load12512 atomic/store12513 atomic/atomicrmw12514 with memory12515 ordering of seq_cst12516 and with equal or12517 wider sync scope.12518 (Note that seq_cst12519 fences have their12520 own s_waitcnt12521 lgkmcnt(0) and so do12522 not need to be12523 considered.)12524 - s_waitcnt vmcnt(0)12525 must happen after12526 preceding12527 global/generic load12528 atomic/store12529 atomic/atomicrmw12530 with memory12531 ordering of seq_cst12532 and with equal or12533 wider sync scope.12534 (Note that seq_cst12535 fences have their12536 own s_waitcnt12537 vmcnt(0) and so do12538 not need to be12539 considered.)12540 - Ensures any12541 preceding12542 sequential12543 consistent global12544 memory instructions12545 have completed12546 before executing12547 this sequentially12548 consistent12549 instruction. This12550 prevents reordering12551 a seq_cst store12552 followed by a12553 seq_cst load. (Note12554 that seq_cst is12555 stronger than12556 acquire/release as12557 the reordering of12558 load acquire12559 followed by a store12560 release is12561 prevented by the12562 s_waitcnt of12563 the release, but12564 there is nothing12565 preventing a store12566 release followed by12567 load acquire from12568 completing out of12569 order. The s_waitcnt12570 could be placed after12571 seq_store or before12572 the seq_load. We12573 choose the load to12574 make the s_waitcnt be12575 as late as possible12576 so that the store12577 may have already12578 completed.)12579 12580 2. *Following12581 instructions same as12582 corresponding load12583 atomic acquire,12584 except must generate12585 all instructions even12586 for OpenCL.*12587 store atomic seq_cst - singlethread - global *Same as corresponding12588 - wavefront - local store atomic release,12589 - workgroup - generic except must generate12590 - agent all instructions even12591 - system for OpenCL.*12592 atomicrmw seq_cst - singlethread - global *Same as corresponding12593 - wavefront - local atomicrmw acq_rel,12594 - workgroup - generic except must generate12595 - agent all instructions even12596 - system for OpenCL.*12597 fence seq_cst - singlethread *none* *Same as corresponding12598 - wavefront fence acq_rel,12599 - workgroup except must generate12600 - agent all instructions even12601 - system for OpenCL.*12602 ============ ============ ============== ========== ================================12603 12604.. _amdgpu-amdhsa-memory-model-gfx10-gfx11:12605 12606Memory Model GFX10-GFX1112607++++++++++++++++++++++++12608 12609For GFX10-GFX11:12610 12611* Each agent has multiple shader arrays (SA).12612* Each SA has multiple work-group processors (WGP).12613* Each WGP has multiple compute units (CU).12614* Each CU has multiple SIMDs that execute wavefronts.12615* The wavefronts for a single work-group are executed in the same12616 WGP. In CU wavefront execution mode the wavefronts may be executed by12617 different SIMDs in the same CU. In WGP wavefront execution mode the12618 wavefronts may be executed by different SIMDs in different CUs in the same12619 WGP.12620* Each WGP has a single LDS memory shared by the wavefronts of the work-groups12621 executing on it.12622* All LDS operations of a WGP are performed as wavefront wide operations in a12623 global order and involve no caching. Completion is reported to a wavefront in12624 execution order.12625* The LDS memory has multiple request queues shared by the SIMDs of a12626 WGP. Therefore, the LDS operations performed by different wavefronts of a12627 work-group can be reordered relative to each other, which can result in12628 reordering the visibility of vector memory operations with respect to LDS12629 operations of other wavefronts in the same work-group. A ``s_waitcnt12630 lgkmcnt(0)`` is required to ensure synchronization between LDS operations and12631 vector memory operations between wavefronts of a work-group, but not between12632 operations performed by the same wavefront.12633* The vector memory operations are performed as wavefront wide operations.12634 Completion of load/store/sample operations are reported to a wavefront in12635 execution order of other load/store/sample operations performed by that12636 wavefront.12637* The vector memory operations access a vector L0 cache. There is a single L012638 cache per CU. Each SIMD of a CU accesses the same L0 cache. Therefore, no12639 special action is required for coherence between the lanes of a single12640 wavefront. However, a ``buffer_gl0_inv`` is required for coherence between12641 wavefronts executing in the same work-group as they may be executing on SIMDs12642 of different CUs that access different L0s. A ``buffer_gl0_inv`` is also12643 required for coherence between wavefronts executing in different work-groups12644 as they may be executing on different WGPs.12645* The scalar memory operations access a scalar L0 cache shared by all wavefronts12646 on a WGP. The scalar and vector L0 caches are not coherent. However, scalar12647 operations are used in a restricted way so do not impact the memory model. See12648 :ref:`amdgpu-amdhsa-memory-spaces`.12649* The vector and scalar memory L0 caches use an L1 cache shared by all WGPs on12650 the same SA. Therefore, no special action is required for coherence between12651 the wavefronts of a single work-group. However, a ``buffer_gl1_inv`` is12652 required for coherence between wavefronts executing in different work-groups12653 as they may be executing on different SAs that access different L1s.12654* The L1 caches have independent quadrants to service disjoint ranges of virtual12655 addresses.12656* Each L0 cache has a separate request queue per L1 quadrant. Therefore, the12657 vector and scalar memory operations performed by different wavefronts, whether12658 executing in the same or different work-groups (which may be executing on12659 different CUs accessing different L0s), can be reordered relative to each12660 other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is required to ensure12661 synchronization between vector memory operations of different wavefronts. It12662 ensures a previous vector memory operation has completed before executing a12663 subsequent vector memory or LDS operation and so can be used to meet the12664 requirements of acquire, release and sequential consistency.12665* The L1 caches use an L2 cache shared by all SAs on the same agent.12666* The L2 cache has independent channels to service disjoint ranges of virtual12667 addresses.12668* Each L1 quadrant of a single SA accesses a different L2 channel. Each L112669 quadrant has a separate request queue per L2 channel. Therefore, the vector12670 and scalar memory operations performed by wavefronts executing in different12671 work-groups (which may be executing on different SAs) of an agent can be12672 reordered relative to each other. A ``s_waitcnt vmcnt(0) & vscnt(0)`` is12673 required to ensure synchronization between vector memory operations of12674 different SAs. It ensures a previous vector memory operation has completed12675 before executing a subsequent vector memory and so can be used to meet the12676 requirements of acquire, release and sequential consistency.12677* The L2 cache can be kept coherent with other agents on some targets, or ranges12678 of virtual addresses can be set up to bypass it to ensure system coherence.12679* On GFX10.3 and GFX11 a memory attached last level (MALL) cache exists for GPU memory.12680 The MALL cache is fully coherent with GPU memory and has no impact on system12681 coherence. All agents (GPU and CPU) access GPU memory through the MALL cache.12682 12683Scalar memory operations are only used to access memory that is proven to not12684change during the execution of the kernel dispatch. This includes constant12685address space and global address space for program scope ``const`` variables.12686Therefore, the kernel machine code does not have to maintain the scalar cache to12687ensure it is coherent with the vector caches. The scalar and vector caches are12688invalidated between kernel dispatches by CP since constant address space data12689may change between kernel dispatch executions. See12690:ref:`amdgpu-amdhsa-memory-spaces`.12691 12692The one exception is if scalar writes are used to spill SGPR registers. In this12693case the AMDGPU backend ensures the memory location used to spill is never12694accessed by vector memory operations at the same time. If scalar writes are used12695then a ``s_dcache_wb`` is inserted before the ``s_endpgm`` and before a function12696return since the locations may be used for vector memory instructions by a12697future wavefront that uses the same scratch area, or a function call that12698creates a frame at the same address, respectively. There is no need for a12699``s_dcache_inv`` as all scalar writes are write-before-read in the same thread.12700 12701For kernarg backing memory:12702 12703* CP invalidates the L0 and L1 caches at the start of each kernel dispatch.12704* On dGPU the kernarg backing memory is accessed as MTYPE UC (uncached) to avoid12705 needing to invalidate the L2 cache.12706* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and12707 so the L2 cache will be coherent with the CPU and other agents.12708 12709Scratch backing memory (which is used for the private address space) is accessed12710with MTYPE NC (non-coherent). Since the private address space is only accessed12711by a single thread, and is always write-before-read, there is never a need to12712invalidate these entries from the L0 or L1 caches.12713 12714Wavefronts are executed in native mode with in-order reporting of loads and12715sample instructions. In this mode vmcnt reports completion of load, atomic with12716return and sample instructions in order, and the vscnt reports the completion of12717store and atomic without return in order. See ``MEM_ORDERED`` field in12718:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx12-table`.12719 12720Wavefronts can be executed in WGP or CU wavefront execution mode:12721 12722* In WGP wavefront execution mode the wavefronts of a work-group are executed12723 on the SIMDs of both CUs of the WGP. Therefore, explicit management of the per12724 CU L0 caches is required for work-group synchronization. Also accesses to L112725 at work-group scope need to be explicitly ordered as the accesses from12726 different CUs are not ordered.12727* In CU wavefront execution mode the wavefronts of a work-group are executed on12728 the SIMDs of a single CU of the WGP. Therefore, all global memory access by12729 the work-group access the same L0 which in turn ensures L1 accesses are12730 ordered and so do not require explicit management of the caches for12731 work-group synchronization.12732 12733See ``WGP_MODE`` field in12734:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx12-table` and12735:ref:`amdgpu-target-features`.12736 12737The code sequences used to implement the memory model for GFX10-GFX11 are defined in12738table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx10-gfx11-table`.12739 12740 .. table:: AMDHSA Memory Model Code Sequences GFX10-GFX1112741 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx10-gfx11-table12742 12743 ============ ============ ============== ========== ================================12744 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code12745 Ordering Sync Scope Address GFX10-GFX1112746 Space12747 ============ ============ ============== ========== ================================12748 **Non-Atomic**12749 ------------------------------------------------------------------------------------12750 load *none* *none* - global - !volatile & !nontemporal12751 - generic12752 - private 1. buffer/global/flat_load12753 - constant12754 - !volatile & nontemporal12755 12756 1. buffer/global/flat_load12757 slc=1 dlc=112758 12759 - If GFX10, omit dlc=1.12760 12761 - volatile12762 12763 1. buffer/global/flat_load12764 glc=1 dlc=112765 12766 2. s_waitcnt vmcnt(0)12767 12768 - Must happen before12769 any following volatile12770 global/generic12771 load/store.12772 - Ensures that12773 volatile12774 operations to12775 different12776 addresses will not12777 be reordered by12778 hardware.12779 12780 load *none* *none* - local 1. ds_load12781 store *none* *none* - global - !volatile & !nontemporal12782 - generic12783 - private 1. buffer/global/flat_store12784 - constant12785 - !volatile & nontemporal12786 12787 1. buffer/global/flat_store12788 glc=1 slc=1 dlc=112789 12790 - If GFX10, omit dlc=1.12791 12792 - volatile12793 12794 1. buffer/global/flat_store12795 dlc=112796 12797 - If GFX10, omit dlc=1.12798 12799 2. s_waitcnt vscnt(0)12800 12801 - Must happen before12802 any following volatile12803 global/generic12804 load/store.12805 - Ensures that12806 volatile12807 operations to12808 different12809 addresses will not12810 be reordered by12811 hardware.12812 12813 store *none* *none* - local 1. ds_store12814 **Unordered Atomic**12815 ------------------------------------------------------------------------------------12816 load atomic unordered *any* *any* *Same as non-atomic*.12817 store atomic unordered *any* *any* *Same as non-atomic*.12818 atomicrmw unordered *any* *any* *Same as monotonic atomic*.12819 **Monotonic Atomic**12820 ------------------------------------------------------------------------------------12821 load atomic monotonic - singlethread - global 1. buffer/global/flat_load12822 - wavefront - generic12823 load atomic monotonic - workgroup - global 1. buffer/global/flat_load12824 - generic glc=112825 12826 - If CU wavefront execution12827 mode, omit glc=1.12828 12829 load atomic monotonic - singlethread - local 1. ds_load12830 - wavefront12831 - workgroup12832 load atomic monotonic - agent - global 1. buffer/global/flat_load12833 - system - generic glc=1 dlc=112834 12835 - If GFX11, omit dlc=1.12836 12837 store atomic monotonic - singlethread - global 1. buffer/global/flat_store12838 - wavefront - generic12839 - workgroup12840 - agent12841 - system12842 store atomic monotonic - singlethread - local 1. ds_store12843 - wavefront12844 - workgroup12845 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic12846 - wavefront - generic12847 - workgroup12848 - agent12849 - system12850 atomicrmw monotonic - singlethread - local 1. ds_atomic12851 - wavefront12852 - workgroup12853 **Acquire Atomic**12854 ------------------------------------------------------------------------------------12855 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load12856 - wavefront - local12857 - generic12858 load atomic acquire - workgroup - global 1. buffer/global_load glc=112859 12860 - If CU wavefront execution12861 mode, omit glc=1.12862 12863 2. s_waitcnt vmcnt(0)12864 12865 - If CU wavefront execution12866 mode, omit.12867 - Must happen before12868 the following buffer_gl0_inv12869 and before any following12870 global/generic12871 load/load12872 atomic/store/store12873 atomic/atomicrmw.12874 12875 3. buffer_gl0_inv12876 12877 - If CU wavefront execution12878 mode, omit.12879 - Ensures that12880 following12881 loads will not see12882 stale data.12883 12884 load atomic acquire - workgroup - local 1. ds_load12885 2. s_waitcnt lgkmcnt(0)12886 12887 - If OpenCL, omit.12888 - Must happen before12889 the following buffer_gl0_inv12890 and before any following12891 global/generic load/load12892 atomic/store/store12893 atomic/atomicrmw.12894 - Ensures any12895 following global12896 data read is no12897 older than the local load12898 atomic value being12899 acquired.12900 12901 3. buffer_gl0_inv12902 12903 - If CU wavefront execution12904 mode, omit.12905 - If OpenCL, omit.12906 - Ensures that12907 following12908 loads will not see12909 stale data.12910 12911 load atomic acquire - workgroup - generic 1. flat_load glc=112912 12913 - If CU wavefront execution12914 mode, omit glc=1.12915 12916 2. s_waitcnt lgkmcnt(0) &12917 vmcnt(0)12918 12919 - If CU wavefront execution12920 mode, omit vmcnt(0).12921 - If OpenCL, omit12922 lgkmcnt(0).12923 - Must happen before12924 the following12925 buffer_gl0_inv and any12926 following global/generic12927 load/load12928 atomic/store/store12929 atomic/atomicrmw.12930 - Ensures any12931 following global12932 data read is no12933 older than a local load12934 atomic value being12935 acquired.12936 12937 3. buffer_gl0_inv12938 12939 - If CU wavefront execution12940 mode, omit.12941 - Ensures that12942 following12943 loads will not see12944 stale data.12945 12946 load atomic acquire - agent - global 1. buffer/global_load12947 - system glc=1 dlc=112948 12949 - If GFX11, omit dlc=1.12950 12951 2. s_waitcnt vmcnt(0)12952 12953 - Must happen before12954 following12955 buffer_gl*_inv.12956 - Ensures the load12957 has completed12958 before invalidating12959 the caches.12960 12961 3. buffer_gl1_inv;12962 buffer_gl0_inv12963 12964 - Must happen before12965 any following12966 global/generic12967 load/load12968 atomic/atomicrmw.12969 - Ensures that12970 following12971 loads will not see12972 stale global data.12973 12974 load atomic acquire - agent - generic 1. flat_load glc=1 dlc=112975 - system12976 - If GFX11, omit dlc=1.12977 12978 2. s_waitcnt vmcnt(0) &12979 lgkmcnt(0)12980 12981 - If OpenCL omit12982 lgkmcnt(0).12983 - Must happen before12984 following12985 buffer_gl*_invl.12986 - Ensures the flat_load12987 has completed12988 before invalidating12989 the caches.12990 12991 3. buffer_gl1_inv;12992 buffer_gl0_inv12993 12994 - Must happen before12995 any following12996 global/generic12997 load/load12998 atomic/atomicrmw.12999 - Ensures that13000 following loads13001 will not see stale13002 global data.13003 13004 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic13005 - wavefront - local13006 - generic13007 atomicrmw acquire - workgroup - global 1. buffer/global_atomic13008 2. s_waitcnt vm/vscnt(0)13009 13010 - If CU wavefront execution13011 mode, omit.13012 - Use vmcnt(0) if atomic with13013 return and vscnt(0) if13014 atomic with no-return.13015 - Must happen before13016 the following buffer_gl0_inv13017 and before any following13018 global/generic13019 load/load13020 atomic/store/store13021 atomic/atomicrmw.13022 13023 3. buffer_gl0_inv13024 13025 - If CU wavefront execution13026 mode, omit.13027 - Ensures that13028 following13029 loads will not see13030 stale data.13031 13032 atomicrmw acquire - workgroup - local 1. ds_atomic13033 2. s_waitcnt lgkmcnt(0)13034 13035 - If OpenCL, omit.13036 - Must happen before13037 the following13038 buffer_gl0_inv.13039 - Ensures any13040 following global13041 data read is no13042 older than the local13043 atomicrmw value13044 being acquired.13045 13046 3. buffer_gl0_inv13047 13048 - If OpenCL omit.13049 - Ensures that13050 following13051 loads will not see13052 stale data.13053 13054 atomicrmw acquire - workgroup - generic 1. flat_atomic13055 2. s_waitcnt lgkmcnt(0) &13056 vm/vscnt(0)13057 13058 - If CU wavefront execution13059 mode, omit vm/vscnt(0).13060 - If OpenCL, omit lgkmcnt(0).13061 - Use vmcnt(0) if atomic with13062 return and vscnt(0) if13063 atomic with no-return.13064 - Must happen before13065 the following13066 buffer_gl0_inv.13067 - Ensures any13068 following global13069 data read is no13070 older than a local13071 atomicrmw value13072 being acquired.13073 13074 3. buffer_gl0_inv13075 13076 - If CU wavefront execution13077 mode, omit.13078 - Ensures that13079 following13080 loads will not see13081 stale data.13082 13083 atomicrmw acquire - agent - global 1. buffer/global_atomic13084 - system 2. s_waitcnt vm/vscnt(0)13085 13086 - Use vmcnt(0) if atomic with13087 return and vscnt(0) if13088 atomic with no-return.13089 - Must happen before13090 following13091 buffer_gl*_inv.13092 - Ensures the13093 atomicrmw has13094 completed before13095 invalidating the13096 caches.13097 13098 3. buffer_gl1_inv;13099 buffer_gl0_inv13100 13101 - Must happen before13102 any following13103 global/generic13104 load/load13105 atomic/atomicrmw.13106 - Ensures that13107 following loads13108 will not see stale13109 global data.13110 13111 atomicrmw acquire - agent - generic 1. flat_atomic13112 - system 2. s_waitcnt vm/vscnt(0) &13113 lgkmcnt(0)13114 13115 - If OpenCL, omit13116 lgkmcnt(0).13117 - Use vmcnt(0) if atomic with13118 return and vscnt(0) if13119 atomic with no-return.13120 - Must happen before13121 following13122 buffer_gl*_inv.13123 - Ensures the13124 atomicrmw has13125 completed before13126 invalidating the13127 caches.13128 13129 3. buffer_gl1_inv;13130 buffer_gl0_inv13131 13132 - Must happen before13133 any following13134 global/generic13135 load/load13136 atomic/atomicrmw.13137 - Ensures that13138 following loads13139 will not see stale13140 global data.13141 13142 fence acquire - singlethread *none* *none*13143 - wavefront13144 fence acquire - workgroup *none* 1. s_waitcnt lgkmcnt(0) &13145 vmcnt(0) & vscnt(0)13146 13147 - If CU wavefront execution13148 mode, omit vmcnt(0) and13149 vscnt(0).13150 - If OpenCL and13151 address space is13152 not generic, omit13153 lgkmcnt(0).13154 - If OpenCL and13155 address space is13156 local, omit13157 vmcnt(0) and vscnt(0).13158 - See :ref:`amdgpu-fence-as` for13159 more details on fencing specific13160 address spaces.13161 - Could be split into13162 separate s_waitcnt13163 vmcnt(0), s_waitcnt13164 vscnt(0) and s_waitcnt13165 lgkmcnt(0) to allow13166 them to be13167 independently moved13168 according to the13169 following rules.13170 - s_waitcnt vmcnt(0)13171 must happen after13172 any preceding13173 global/generic load13174 atomic/13175 atomicrmw-with-return-value13176 with an equal or13177 wider sync scope13178 and memory ordering13179 stronger than13180 unordered (this is13181 termed the13182 fence-paired-atomic).13183 - s_waitcnt vscnt(0)13184 must happen after13185 any preceding13186 global/generic13187 atomicrmw-no-return-value13188 with an equal or13189 wider sync scope13190 and memory ordering13191 stronger than13192 unordered (this is13193 termed the13194 fence-paired-atomic).13195 - s_waitcnt lgkmcnt(0)13196 must happen after13197 any preceding13198 local/generic load13199 atomic/atomicrmw13200 with an equal or13201 wider sync scope13202 and memory ordering13203 stronger than13204 unordered (this is13205 termed the13206 fence-paired-atomic).13207 - Must happen before13208 the following13209 buffer_gl0_inv.13210 - Ensures that the13211 fence-paired atomic13212 has completed13213 before invalidating13214 the13215 cache. Therefore13216 any following13217 locations read must13218 be no older than13219 the value read by13220 the13221 fence-paired-atomic.13222 13223 3. buffer_gl0_inv13224 13225 - If CU wavefront execution13226 mode, omit.13227 - Ensures that13228 following13229 loads will not see13230 stale data.13231 13232 fence acquire - agent *none* 1. s_waitcnt lgkmcnt(0) &13233 - system vmcnt(0) & vscnt(0)13234 13235 - If OpenCL and13236 address space is13237 not generic, omit13238 lgkmcnt(0).13239 - If OpenCL and13240 address space is13241 local, omit13242 vmcnt(0) and vscnt(0).13243 - See :ref:`amdgpu-fence-as` for13244 more details on fencing specific13245 address spaces.13246 - Could be split into13247 separate s_waitcnt13248 vmcnt(0), s_waitcnt13249 vscnt(0) and s_waitcnt13250 lgkmcnt(0) to allow13251 them to be13252 independently moved13253 according to the13254 following rules.13255 - s_waitcnt vmcnt(0)13256 must happen after13257 any preceding13258 global/generic load13259 atomic/13260 atomicrmw-with-return-value13261 with an equal or13262 wider sync scope13263 and memory ordering13264 stronger than13265 unordered (this is13266 termed the13267 fence-paired-atomic).13268 - s_waitcnt vscnt(0)13269 must happen after13270 any preceding13271 global/generic13272 atomicrmw-no-return-value13273 with an equal or13274 wider sync scope13275 and memory ordering13276 stronger than13277 unordered (this is13278 termed the13279 fence-paired-atomic).13280 - s_waitcnt lgkmcnt(0)13281 must happen after13282 any preceding13283 local/generic load13284 atomic/atomicrmw13285 with an equal or13286 wider sync scope13287 and memory ordering13288 stronger than13289 unordered (this is13290 termed the13291 fence-paired-atomic).13292 - Must happen before13293 the following13294 buffer_gl*_inv.13295 - Ensures that the13296 fence-paired atomic13297 has completed13298 before invalidating13299 the13300 caches. Therefore13301 any following13302 locations read must13303 be no older than13304 the value read by13305 the13306 fence-paired-atomic.13307 13308 2. buffer_gl1_inv;13309 buffer_gl0_inv13310 13311 - Must happen before any13312 following global/generic13313 load/load13314 atomic/store/store13315 atomic/atomicrmw.13316 - Ensures that13317 following loads13318 will not see stale13319 global data.13320 13321 **Release Atomic**13322 ------------------------------------------------------------------------------------13323 store atomic release - singlethread - global 1. buffer/global/ds/flat_store13324 - wavefront - local13325 - generic13326 store atomic release - workgroup - global 1. s_waitcnt lgkmcnt(0) &13327 - generic vmcnt(0) & vscnt(0)13328 13329 - If OpenCL, omit13330 lgkmcnt(0).13331 - Could be split into13332 separate s_waitcnt13333 vmcnt(0), s_waitcnt13334 vscnt(0) and s_waitcnt13335 lgkmcnt(0) to allow13336 them to be13337 independently moved13338 according to the13339 following rules.13340 - s_waitcnt vmcnt(0)13341 must happen after13342 any preceding13343 global/generic load/load13344 atomic/13345 atomicrmw-with-return-value.13346 - s_waitcnt vscnt(0)13347 must happen after13348 any preceding13349 global/generic13350 store/store13351 atomic/13352 atomicrmw-no-return-value.13353 - s_waitcnt lgkmcnt(0)13354 must happen after13355 any preceding13356 local/generic13357 load/store/load13358 atomic/store13359 atomic/atomicrmw.13360 - Must happen before13361 the following13362 store.13363 - Ensures that all13364 memory operations13365 have13366 completed before13367 performing the13368 store that is being13369 released.13370 13371 2. buffer/global/flat_store13372 store atomic release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0)13373 13374 - If OpenCL, omit.13375 - Could be split into13376 separate s_waitcnt13377 vmcnt(0) and s_waitcnt13378 vscnt(0) to allow13379 them to be13380 independently moved13381 according to the13382 following rules.13383 - s_waitcnt vmcnt(0)13384 must happen after13385 any preceding13386 global/generic load/load13387 atomic/13388 atomicrmw-with-return-value.13389 - s_waitcnt vscnt(0)13390 must happen after13391 any preceding13392 global/generic13393 store/store atomic/13394 atomicrmw-no-return-value.13395 - Must happen before13396 the following13397 store.13398 - Ensures that all13399 global memory13400 operations have13401 completed before13402 performing the13403 store that is being13404 released.13405 13406 2. ds_store13407 store atomic release - agent - global 1. s_waitcnt lgkmcnt(0) &13408 - system - generic vmcnt(0) & vscnt(0)13409 13410 - If OpenCL and13411 address space is13412 not generic, omit13413 lgkmcnt(0).13414 - Could be split into13415 separate s_waitcnt13416 vmcnt(0), s_waitcnt vscnt(0)13417 and s_waitcnt13418 lgkmcnt(0) to allow13419 them to be13420 independently moved13421 according to the13422 following rules.13423 - s_waitcnt vmcnt(0)13424 must happen after13425 any preceding13426 global/generic13427 load/load13428 atomic/13429 atomicrmw-with-return-value.13430 - s_waitcnt vscnt(0)13431 must happen after13432 any preceding13433 global/generic13434 store/store atomic/13435 atomicrmw-no-return-value.13436 - s_waitcnt lgkmcnt(0)13437 must happen after13438 any preceding13439 local/generic13440 load/store/load13441 atomic/store13442 atomic/atomicrmw.13443 - Must happen before13444 the following13445 store.13446 - Ensures that all13447 memory operations13448 have13449 completed before13450 performing the13451 store that is being13452 released.13453 13454 2. buffer/global/flat_store13455 atomicrmw release - singlethread - global 1. buffer/global/ds/flat_atomic13456 - wavefront - local13457 - generic13458 atomicrmw release - workgroup - global 1. s_waitcnt lgkmcnt(0) &13459 - generic vmcnt(0) & vscnt(0)13460 13461 - If OpenCL, omit lgkmcnt(0).13462 - Could be split into13463 separate s_waitcnt13464 vmcnt(0), s_waitcnt13465 vscnt(0) and s_waitcnt13466 lgkmcnt(0) to allow13467 them to be13468 independently moved13469 according to the13470 following rules.13471 - s_waitcnt vmcnt(0)13472 must happen after13473 any preceding13474 global/generic load/load13475 atomic/13476 atomicrmw-with-return-value.13477 - s_waitcnt vscnt(0)13478 must happen after13479 any preceding13480 global/generic13481 store/store13482 atomic/13483 atomicrmw-no-return-value.13484 - s_waitcnt lgkmcnt(0)13485 must happen after13486 any preceding13487 local/generic13488 load/store/load13489 atomic/store13490 atomic/atomicrmw.13491 - Must happen before13492 the following13493 atomicrmw.13494 - Ensures that all13495 memory operations13496 have13497 completed before13498 performing the13499 atomicrmw that is13500 being released.13501 13502 2. buffer/global/flat_atomic13503 atomicrmw release - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0)13504 13505 - If OpenCL, omit.13506 - Could be split into13507 separate s_waitcnt13508 vmcnt(0) and s_waitcnt13509 vscnt(0) to allow13510 them to be13511 independently moved13512 according to the13513 following rules.13514 - s_waitcnt vmcnt(0)13515 must happen after13516 any preceding13517 global/generic load/load13518 atomic/13519 atomicrmw-with-return-value.13520 - s_waitcnt vscnt(0)13521 must happen after13522 any preceding13523 global/generic13524 store/store atomic/13525 atomicrmw-no-return-value.13526 - Must happen before13527 the following13528 store.13529 - Ensures that all13530 global memory13531 operations have13532 completed before13533 performing the13534 store that is being13535 released.13536 13537 2. ds_atomic13538 atomicrmw release - agent - global 1. s_waitcnt lgkmcnt(0) &13539 - system - generic vmcnt(0) & vscnt(0)13540 13541 - If OpenCL, omit13542 lgkmcnt(0).13543 - Could be split into13544 separate s_waitcnt13545 vmcnt(0), s_waitcnt13546 vscnt(0) and s_waitcnt13547 lgkmcnt(0) to allow13548 them to be13549 independently moved13550 according to the13551 following rules.13552 - s_waitcnt vmcnt(0)13553 must happen after13554 any preceding13555 global/generic13556 load/load atomic/13557 atomicrmw-with-return-value.13558 - s_waitcnt vscnt(0)13559 must happen after13560 any preceding13561 global/generic13562 store/store atomic/13563 atomicrmw-no-return-value.13564 - s_waitcnt lgkmcnt(0)13565 must happen after13566 any preceding13567 local/generic13568 load/store/load13569 atomic/store13570 atomic/atomicrmw.13571 - Must happen before13572 the following13573 atomicrmw.13574 - Ensures that all13575 memory operations13576 to global and local13577 have completed13578 before performing13579 the atomicrmw that13580 is being released.13581 13582 2. buffer/global/flat_atomic13583 fence release - singlethread *none* *none*13584 - wavefront13585 fence release - workgroup *none* 1. s_waitcnt lgkmcnt(0) &13586 vmcnt(0) & vscnt(0)13587 13588 - If OpenCL and13589 address space is13590 not generic, omit13591 lgkmcnt(0).13592 - If OpenCL and13593 address space is13594 local, omit13595 vmcnt(0) and vscnt(0).13596 - See :ref:`amdgpu-fence-as` for13597 more details on fencing specific13598 address spaces.13599 - Could be split into13600 separate s_waitcnt13601 vmcnt(0), s_waitcnt13602 vscnt(0) and s_waitcnt13603 lgkmcnt(0) to allow13604 them to be13605 independently moved13606 according to the13607 following rules.13608 - s_waitcnt vmcnt(0)13609 must happen after13610 any preceding13611 global/generic13612 load/load13613 atomic/13614 atomicrmw-with-return-value.13615 - s_waitcnt vscnt(0)13616 must happen after13617 any preceding13618 global/generic13619 store/store atomic/13620 atomicrmw-no-return-value.13621 - s_waitcnt lgkmcnt(0)13622 must happen after13623 any preceding13624 local/generic13625 load/store/load13626 atomic/store atomic/13627 atomicrmw.13628 - Must happen before13629 any following store13630 atomic/atomicrmw13631 with an equal or13632 wider sync scope13633 and memory ordering13634 stronger than13635 unordered (this is13636 termed the13637 fence-paired-atomic).13638 - Ensures that all13639 memory operations13640 have13641 completed before13642 performing the13643 following13644 fence-paired-atomic.13645 13646 fence release - agent *none* 1. s_waitcnt lgkmcnt(0) &13647 - system vmcnt(0) & vscnt(0)13648 13649 - If OpenCL and13650 address space is13651 not generic, omit13652 lgkmcnt(0).13653 - If OpenCL and13654 address space is13655 local, omit13656 vmcnt(0) and vscnt(0).13657 - See :ref:`amdgpu-fence-as` for13658 more details on fencing specific13659 address spaces.13660 - Could be split into13661 separate s_waitcnt13662 vmcnt(0), s_waitcnt13663 vscnt(0) and s_waitcnt13664 lgkmcnt(0) to allow13665 them to be13666 independently moved13667 according to the13668 following rules.13669 - s_waitcnt vmcnt(0)13670 must happen after13671 any preceding13672 global/generic13673 load/load atomic/13674 atomicrmw-with-return-value.13675 - s_waitcnt vscnt(0)13676 must happen after13677 any preceding13678 global/generic13679 store/store atomic/13680 atomicrmw-no-return-value.13681 - s_waitcnt lgkmcnt(0)13682 must happen after13683 any preceding13684 local/generic13685 load/store/load13686 atomic/store13687 atomic/atomicrmw.13688 - Must happen before13689 any following store13690 atomic/atomicrmw13691 with an equal or13692 wider sync scope13693 and memory ordering13694 stronger than13695 unordered (this is13696 termed the13697 fence-paired-atomic).13698 - Ensures that all13699 memory operations13700 have13701 completed before13702 performing the13703 following13704 fence-paired-atomic.13705 13706 **Acquire-Release Atomic**13707 ------------------------------------------------------------------------------------13708 atomicrmw acq_rel - singlethread - global 1. buffer/global/ds/flat_atomic13709 - wavefront - local13710 - generic13711 atomicrmw acq_rel - workgroup - global 1. s_waitcnt lgkmcnt(0) &13712 vmcnt(0) & vscnt(0)13713 13714 - If OpenCL, omit13715 lgkmcnt(0).13716 - Must happen after13717 any preceding13718 local/generic13719 load/store/load13720 atomic/store13721 atomic/atomicrmw.13722 - Could be split into13723 separate s_waitcnt13724 vmcnt(0), s_waitcnt13725 vscnt(0), and s_waitcnt13726 lgkmcnt(0) to allow13727 them to be13728 independently moved13729 according to the13730 following rules.13731 - s_waitcnt vmcnt(0)13732 must happen after13733 any preceding13734 global/generic load/load13735 atomic/13736 atomicrmw-with-return-value.13737 - s_waitcnt vscnt(0)13738 must happen after13739 any preceding13740 global/generic13741 store/store13742 atomic/13743 atomicrmw-no-return-value.13744 - s_waitcnt lgkmcnt(0)13745 must happen after13746 any preceding13747 local/generic13748 load/store/load13749 atomic/store13750 atomic/atomicrmw.13751 - Must happen before13752 the following13753 atomicrmw.13754 - Ensures that all13755 memory operations13756 have13757 completed before13758 performing the13759 atomicrmw that is13760 being released.13761 13762 2. buffer/global_atomic13763 3. s_waitcnt vm/vscnt(0)13764 13765 - Use vmcnt(0) if atomic with13766 return and vscnt(0) if13767 atomic with no-return.13768 - Must happen before13769 the following13770 buffer_gl0_inv.13771 - Ensures any13772 following global13773 data read is no13774 older than the13775 atomicrmw value13776 being acquired.13777 13778 4. buffer_gl0_inv13779 13780 - If CU wavefront execution13781 mode, omit.13782 - Ensures that13783 following13784 loads will not see13785 stale data.13786 13787 atomicrmw acq_rel - workgroup - local 1. s_waitcnt vmcnt(0) & vscnt(0)13788 13789 - If OpenCL, omit.13790 - Could be split into13791 separate s_waitcnt13792 vmcnt(0) and s_waitcnt13793 vscnt(0) to allow13794 them to be13795 independently moved13796 according to the13797 following rules.13798 - s_waitcnt vmcnt(0)13799 must happen after13800 any preceding13801 global/generic load/load13802 atomic/13803 atomicrmw-with-return-value.13804 - s_waitcnt vscnt(0)13805 must happen after13806 any preceding13807 global/generic13808 store/store atomic/13809 atomicrmw-no-return-value.13810 - Must happen before13811 the following13812 store.13813 - Ensures that all13814 global memory13815 operations have13816 completed before13817 performing the13818 store that is being13819 released.13820 13821 2. ds_atomic13822 3. s_waitcnt lgkmcnt(0)13823 13824 - If OpenCL, omit.13825 - Must happen before13826 the following13827 buffer_gl0_inv.13828 - Ensures any13829 following global13830 data read is no13831 older than the local load13832 atomic value being13833 acquired.13834 13835 4. buffer_gl0_inv13836 13837 - If CU wavefront execution13838 mode, omit.13839 - If OpenCL omit.13840 - Ensures that13841 following13842 loads will not see13843 stale data.13844 13845 atomicrmw acq_rel - workgroup - generic 1. s_waitcnt lgkmcnt(0) &13846 vmcnt(0) & vscnt(0)13847 13848 - If OpenCL, omit lgkmcnt(0).13849 - Could be split into13850 separate s_waitcnt13851 vmcnt(0), s_waitcnt13852 vscnt(0) and s_waitcnt13853 lgkmcnt(0) to allow13854 them to be13855 independently moved13856 according to the13857 following rules.13858 - s_waitcnt vmcnt(0)13859 must happen after13860 any preceding13861 global/generic load/load13862 atomic/13863 atomicrmw-with-return-value.13864 - s_waitcnt vscnt(0)13865 must happen after13866 any preceding13867 global/generic13868 store/store13869 atomic/13870 atomicrmw-no-return-value.13871 - s_waitcnt lgkmcnt(0)13872 must happen after13873 any preceding13874 local/generic13875 load/store/load13876 atomic/store13877 atomic/atomicrmw.13878 - Must happen before13879 the following13880 atomicrmw.13881 - Ensures that all13882 memory operations13883 have13884 completed before13885 performing the13886 atomicrmw that is13887 being released.13888 13889 2. flat_atomic13890 3. s_waitcnt lgkmcnt(0) &13891 vmcnt(0) & vscnt(0)13892 13893 - If atomic with return, omit13894 vscnt(0), if atomic with13895 no-return, omit vmcnt(0).13896 - If OpenCL, omit lgkmcnt(0).13897 - Must happen before13898 the following13899 buffer_gl0_inv.13900 - Ensures any13901 following global13902 data read is no13903 older than the load13904 atomic value being13905 acquired.13906 13907 3. buffer_gl0_inv13908 13909 - If CU wavefront execution13910 mode, omit.13911 - Ensures that13912 following13913 loads will not see13914 stale data.13915 13916 atomicrmw acq_rel - agent - global 1. s_waitcnt lgkmcnt(0) &13917 - system vmcnt(0) & vscnt(0)13918 13919 - If OpenCL, omit13920 lgkmcnt(0).13921 - Could be split into13922 separate s_waitcnt13923 vmcnt(0), s_waitcnt13924 vscnt(0) and s_waitcnt13925 lgkmcnt(0) to allow13926 them to be13927 independently moved13928 according to the13929 following rules.13930 - s_waitcnt vmcnt(0)13931 must happen after13932 any preceding13933 global/generic13934 load/load atomic/13935 atomicrmw-with-return-value.13936 - s_waitcnt vscnt(0)13937 must happen after13938 any preceding13939 global/generic13940 store/store atomic/13941 atomicrmw-no-return-value.13942 - s_waitcnt lgkmcnt(0)13943 must happen after13944 any preceding13945 local/generic13946 load/store/load13947 atomic/store13948 atomic/atomicrmw.13949 - Must happen before13950 the following13951 atomicrmw.13952 - Ensures that all13953 memory operations13954 to global have13955 completed before13956 performing the13957 atomicrmw that is13958 being released.13959 13960 2. buffer/global_atomic13961 3. s_waitcnt vm/vscnt(0)13962 13963 - Use vmcnt(0) if atomic with13964 return and vscnt(0) if13965 atomic with no-return.13966 - Must happen before13967 following13968 buffer_gl*_inv.13969 - Ensures the13970 atomicrmw has13971 completed before13972 invalidating the13973 caches.13974 13975 4. buffer_gl1_inv;13976 buffer_gl0_inv13977 13978 - Must happen before13979 any following13980 global/generic13981 load/load13982 atomic/atomicrmw.13983 - Ensures that13984 following loads13985 will not see stale13986 global data.13987 13988 atomicrmw acq_rel - agent - generic 1. s_waitcnt lgkmcnt(0) &13989 - system vmcnt(0) & vscnt(0)13990 13991 - If OpenCL, omit13992 lgkmcnt(0).13993 - Could be split into13994 separate s_waitcnt13995 vmcnt(0), s_waitcnt13996 vscnt(0), and s_waitcnt13997 lgkmcnt(0) to allow13998 them to be13999 independently moved14000 according to the14001 following rules.14002 - s_waitcnt vmcnt(0)14003 must happen after14004 any preceding14005 global/generic14006 load/load atomic14007 atomicrmw-with-return-value.14008 - s_waitcnt vscnt(0)14009 must happen after14010 any preceding14011 global/generic14012 store/store atomic/14013 atomicrmw-no-return-value.14014 - s_waitcnt lgkmcnt(0)14015 must happen after14016 any preceding14017 local/generic14018 load/store/load14019 atomic/store14020 atomic/atomicrmw.14021 - Must happen before14022 the following14023 atomicrmw.14024 - Ensures that all14025 memory operations14026 have14027 completed before14028 performing the14029 atomicrmw that is14030 being released.14031 14032 2. flat_atomic14033 3. s_waitcnt vm/vscnt(0) &14034 lgkmcnt(0)14035 14036 - If OpenCL, omit14037 lgkmcnt(0).14038 - Use vmcnt(0) if atomic with14039 return and vscnt(0) if14040 atomic with no-return.14041 - Must happen before14042 following14043 buffer_gl*_inv.14044 - Ensures the14045 atomicrmw has14046 completed before14047 invalidating the14048 caches.14049 14050 4. buffer_gl1_inv;14051 buffer_gl0_inv14052 14053 - Must happen before14054 any following14055 global/generic14056 load/load14057 atomic/atomicrmw.14058 - Ensures that14059 following loads14060 will not see stale14061 global data.14062 14063 fence acq_rel - singlethread *none* *none*14064 - wavefront14065 fence acq_rel - workgroup *none* 1. s_waitcnt lgkmcnt(0) &14066 vmcnt(0) & vscnt(0)14067 14068 - If OpenCL and14069 address space is14070 not generic, omit14071 lgkmcnt(0).14072 - If OpenCL and14073 address space is14074 local, omit14075 vmcnt(0) and vscnt(0).14076 - However,14077 since LLVM14078 currently has no14079 address space on14080 the fence need to14081 conservatively14082 always generate14083 (see comment for14084 previous fence).14085 - Could be split into14086 separate s_waitcnt14087 vmcnt(0), s_waitcnt14088 vscnt(0) and s_waitcnt14089 lgkmcnt(0) to allow14090 them to be14091 independently moved14092 according to the14093 following rules.14094 - s_waitcnt vmcnt(0)14095 must happen after14096 any preceding14097 global/generic14098 load/load14099 atomic/14100 atomicrmw-with-return-value.14101 - s_waitcnt vscnt(0)14102 must happen after14103 any preceding14104 global/generic14105 store/store atomic/14106 atomicrmw-no-return-value.14107 - s_waitcnt lgkmcnt(0)14108 must happen after14109 any preceding14110 local/generic14111 load/store/load14112 atomic/store atomic/14113 atomicrmw.14114 - Must happen before14115 any following14116 global/generic14117 load/load14118 atomic/store/store14119 atomic/atomicrmw.14120 - Ensures that all14121 memory operations14122 have14123 completed before14124 performing any14125 following global14126 memory operations.14127 - Ensures that the14128 preceding14129 local/generic load14130 atomic/atomicrmw14131 with an equal or14132 wider sync scope14133 and memory ordering14134 stronger than14135 unordered (this is14136 termed the14137 acquire-fence-paired-atomic)14138 has completed14139 before following14140 global memory14141 operations. This14142 satisfies the14143 requirements of14144 acquire.14145 - Ensures that all14146 previous memory14147 operations have14148 completed before a14149 following14150 local/generic store14151 atomic/atomicrmw14152 with an equal or14153 wider sync scope14154 and memory ordering14155 stronger than14156 unordered (this is14157 termed the14158 release-fence-paired-atomic).14159 This satisfies the14160 requirements of14161 release.14162 - Must happen before14163 the following14164 buffer_gl0_inv.14165 - Ensures that the14166 acquire-fence-paired14167 atomic has completed14168 before invalidating14169 the14170 cache. Therefore14171 any following14172 locations read must14173 be no older than14174 the value read by14175 the14176 acquire-fence-paired-atomic.14177 14178 3. buffer_gl0_inv14179 14180 - If CU wavefront execution14181 mode, omit.14182 - Ensures that14183 following14184 loads will not see14185 stale data.14186 14187 fence acq_rel - agent *none* 1. s_waitcnt lgkmcnt(0) &14188 - system vmcnt(0) & vscnt(0)14189 14190 - If OpenCL and14191 address space is14192 not generic, omit14193 lgkmcnt(0).14194 - If OpenCL and14195 address space is14196 local, omit14197 vmcnt(0) and vscnt(0).14198 - See :ref:`amdgpu-fence-as` for14199 more details on fencing specific14200 address spaces.14201 - Could be split into14202 separate s_waitcnt14203 vmcnt(0), s_waitcnt14204 vscnt(0) and s_waitcnt14205 lgkmcnt(0) to allow14206 them to be14207 independently moved14208 according to the14209 following rules.14210 - s_waitcnt vmcnt(0)14211 must happen after14212 any preceding14213 global/generic14214 load/load14215 atomic/14216 atomicrmw-with-return-value.14217 - s_waitcnt vscnt(0)14218 must happen after14219 any preceding14220 global/generic14221 store/store atomic/14222 atomicrmw-no-return-value.14223 - s_waitcnt lgkmcnt(0)14224 must happen after14225 any preceding14226 local/generic14227 load/store/load14228 atomic/store14229 atomic/atomicrmw.14230 - Must happen before14231 the following14232 buffer_gl*_inv.14233 - Ensures that the14234 preceding14235 global/local/generic14236 load14237 atomic/atomicrmw14238 with an equal or14239 wider sync scope14240 and memory ordering14241 stronger than14242 unordered (this is14243 termed the14244 acquire-fence-paired-atomic)14245 has completed14246 before invalidating14247 the caches. This14248 satisfies the14249 requirements of14250 acquire.14251 - Ensures that all14252 previous memory14253 operations have14254 completed before a14255 following14256 global/local/generic14257 store14258 atomic/atomicrmw14259 with an equal or14260 wider sync scope14261 and memory ordering14262 stronger than14263 unordered (this is14264 termed the14265 release-fence-paired-atomic).14266 This satisfies the14267 requirements of14268 release.14269 14270 2. buffer_gl1_inv;14271 buffer_gl0_inv14272 14273 - Must happen before14274 any following14275 global/generic14276 load/load14277 atomic/store/store14278 atomic/atomicrmw.14279 - Ensures that14280 following loads14281 will not see stale14282 global data. This14283 satisfies the14284 requirements of14285 acquire.14286 14287 **Sequential Consistent Atomic**14288 ------------------------------------------------------------------------------------14289 load atomic seq_cst - singlethread - global *Same as corresponding14290 - wavefront - local load atomic acquire,14291 - generic except must generate14292 all instructions even14293 for OpenCL.*14294 load atomic seq_cst - workgroup - global 1. s_waitcnt lgkmcnt(0) &14295 - generic vmcnt(0) & vscnt(0)14296 14297 - Could be split into14298 separate s_waitcnt14299 vmcnt(0), s_waitcnt14300 vscnt(0), and s_waitcnt14301 lgkmcnt(0) to allow14302 them to be14303 independently moved14304 according to the14305 following rules.14306 - s_waitcnt lgkmcnt(0) must14307 happen after14308 preceding14309 local/generic load14310 atomic/store14311 atomic/atomicrmw14312 with memory14313 ordering of seq_cst14314 and with equal or14315 wider sync scope.14316 (Note that seq_cst14317 fences have their14318 own s_waitcnt14319 lgkmcnt(0) and so do14320 not need to be14321 considered.)14322 - s_waitcnt vmcnt(0)14323 must happen after14324 preceding14325 global/generic load14326 atomic/14327 atomicrmw-with-return-value14328 with memory14329 ordering of seq_cst14330 and with equal or14331 wider sync scope.14332 (Note that seq_cst14333 fences have their14334 own s_waitcnt14335 vmcnt(0) and so do14336 not need to be14337 considered.)14338 - s_waitcnt vscnt(0)14339 Must happen after14340 preceding14341 global/generic store14342 atomic/14343 atomicrmw-no-return-value14344 with memory14345 ordering of seq_cst14346 and with equal or14347 wider sync scope.14348 (Note that seq_cst14349 fences have their14350 own s_waitcnt14351 vscnt(0) and so do14352 not need to be14353 considered.)14354 - Ensures any14355 preceding14356 sequential14357 consistent global/local14358 memory instructions14359 have completed14360 before executing14361 this sequentially14362 consistent14363 instruction. This14364 prevents reordering14365 a seq_cst store14366 followed by a14367 seq_cst load. (Note14368 that seq_cst is14369 stronger than14370 acquire/release as14371 the reordering of14372 load acquire14373 followed by a store14374 release is14375 prevented by the14376 s_waitcnt of14377 the release, but14378 there is nothing14379 preventing a store14380 release followed by14381 load acquire from14382 completing out of14383 order. The s_waitcnt14384 could be placed after14385 seq_store or before14386 the seq_load. We14387 choose the load to14388 make the s_waitcnt be14389 as late as possible14390 so that the store14391 may have already14392 completed.)14393 14394 2. *Following14395 instructions same as14396 corresponding load14397 atomic acquire,14398 except must generate14399 all instructions even14400 for OpenCL.*14401 load atomic seq_cst - workgroup - local14402 14403 1. s_waitcnt vmcnt(0) & vscnt(0)14404 14405 - Could be split into14406 separate s_waitcnt14407 vmcnt(0) and s_waitcnt14408 vscnt(0) to allow14409 them to be14410 independently moved14411 according to the14412 following rules.14413 - s_waitcnt vmcnt(0)14414 Must happen after14415 preceding14416 global/generic load14417 atomic/14418 atomicrmw-with-return-value14419 with memory14420 ordering of seq_cst14421 and with equal or14422 wider sync scope.14423 (Note that seq_cst14424 fences have their14425 own s_waitcnt14426 vmcnt(0) and so do14427 not need to be14428 considered.)14429 - s_waitcnt vscnt(0)14430 Must happen after14431 preceding14432 global/generic store14433 atomic/14434 atomicrmw-no-return-value14435 with memory14436 ordering of seq_cst14437 and with equal or14438 wider sync scope.14439 (Note that seq_cst14440 fences have their14441 own s_waitcnt14442 vscnt(0) and so do14443 not need to be14444 considered.)14445 - Ensures any14446 preceding14447 sequential14448 consistent global14449 memory instructions14450 have completed14451 before executing14452 this sequentially14453 consistent14454 instruction. This14455 prevents reordering14456 a seq_cst store14457 followed by a14458 seq_cst load. (Note14459 that seq_cst is14460 stronger than14461 acquire/release as14462 the reordering of14463 load acquire14464 followed by a store14465 release is14466 prevented by the14467 s_waitcnt of14468 the release, but14469 there is nothing14470 preventing a store14471 release followed by14472 load acquire from14473 completing out of14474 order. The s_waitcnt14475 could be placed after14476 seq_store or before14477 the seq_load. We14478 choose the load to14479 make the s_waitcnt be14480 as late as possible14481 so that the store14482 may have already14483 completed.)14484 14485 2. *Following14486 instructions same as14487 corresponding load14488 atomic acquire,14489 except must generate14490 all instructions even14491 for OpenCL.*14492 14493 load atomic seq_cst - agent - global 1. s_waitcnt lgkmcnt(0) &14494 - system - generic vmcnt(0) & vscnt(0)14495 14496 - Could be split into14497 separate s_waitcnt14498 vmcnt(0), s_waitcnt14499 vscnt(0) and s_waitcnt14500 lgkmcnt(0) to allow14501 them to be14502 independently moved14503 according to the14504 following rules.14505 - s_waitcnt lgkmcnt(0)14506 must happen after14507 preceding14508 local load14509 atomic/store14510 atomic/atomicrmw14511 with memory14512 ordering of seq_cst14513 and with equal or14514 wider sync scope.14515 (Note that seq_cst14516 fences have their14517 own s_waitcnt14518 lgkmcnt(0) and so do14519 not need to be14520 considered.)14521 - s_waitcnt vmcnt(0)14522 must happen after14523 preceding14524 global/generic load14525 atomic/14526 atomicrmw-with-return-value14527 with memory14528 ordering of seq_cst14529 and with equal or14530 wider sync scope.14531 (Note that seq_cst14532 fences have their14533 own s_waitcnt14534 vmcnt(0) and so do14535 not need to be14536 considered.)14537 - s_waitcnt vscnt(0)14538 Must happen after14539 preceding14540 global/generic store14541 atomic/14542 atomicrmw-no-return-value14543 with memory14544 ordering of seq_cst14545 and with equal or14546 wider sync scope.14547 (Note that seq_cst14548 fences have their14549 own s_waitcnt14550 vscnt(0) and so do14551 not need to be14552 considered.)14553 - Ensures any14554 preceding14555 sequential14556 consistent global14557 memory instructions14558 have completed14559 before executing14560 this sequentially14561 consistent14562 instruction. This14563 prevents reordering14564 a seq_cst store14565 followed by a14566 seq_cst load. (Note14567 that seq_cst is14568 stronger than14569 acquire/release as14570 the reordering of14571 load acquire14572 followed by a store14573 release is14574 prevented by the14575 s_waitcnt of14576 the release, but14577 there is nothing14578 preventing a store14579 release followed by14580 load acquire from14581 completing out of14582 order. The s_waitcnt14583 could be placed after14584 seq_store or before14585 the seq_load. We14586 choose the load to14587 make the s_waitcnt be14588 as late as possible14589 so that the store14590 may have already14591 completed.)14592 14593 2. *Following14594 instructions same as14595 corresponding load14596 atomic acquire,14597 except must generate14598 all instructions even14599 for OpenCL.*14600 store atomic seq_cst - singlethread - global *Same as corresponding14601 - wavefront - local store atomic release,14602 - workgroup - generic except must generate14603 - agent all instructions even14604 - system for OpenCL.*14605 atomicrmw seq_cst - singlethread - global *Same as corresponding14606 - wavefront - local atomicrmw acq_rel,14607 - workgroup - generic except must generate14608 - agent all instructions even14609 - system for OpenCL.*14610 fence seq_cst - singlethread *none* *Same as corresponding14611 - wavefront fence acq_rel,14612 - workgroup except must generate14613 - agent all instructions even14614 - system for OpenCL.*14615 ============ ============ ============== ========== ================================14616 14617 14618.. _amdgpu-amdhsa-memory-model-gfx12:14619 14620Memory Model GFX1214621++++++++++++++++++++++++14622 14623For GFX12:14624 14625* Each agent has multiple shader arrays (SA).14626* Each SA has multiple work-group processors (WGP).14627* Each WGP has multiple compute units (CU).14628* Each CU has multiple SIMDs that execute wavefronts.14629* The wavefronts for a single work-group are executed in the same14630 WGP.14631 14632 * In CU wavefront execution mode the wavefronts may be executed by different SIMDs14633 in the same CU.14634 * In WGP wavefront execution mode the wavefronts may be executed by different SIMDs14635 in different CUs in the same WGP.14636 14637* Each WGP has a single LDS memory shared by the wavefronts of the work-groups14638 executing on it.14639* All LDS operations of a WGP are performed as wavefront wide operations in a14640 global order and involve no caching. Completion is reported to a wavefront in14641 execution order.14642* The LDS memory has multiple request queues shared by the SIMDs of a14643 WGP. Therefore, the LDS operations performed by different wavefronts of a14644 work-group can be reordered relative to each other, which can result in14645 reordering the visibility of vector memory operations with respect to LDS14646 operations of other wavefronts in the same work-group. A ``s_wait_dscnt 0x0``14647 is required to ensure synchronization between LDS operations and14648 vector memory operations between wavefronts of a work-group, but not between14649 operations performed by the same wavefront.14650* The vector memory operations are performed as wavefront wide operations.14651 Vector memory operations are divided in different types. Completion of a14652 vector memory operation is reported to a wavefront in-order within a type,14653 but may be out of order between types. The types of vector memory operations14654 (and their associated ``s_wait`` instructions) are:14655 14656 * LDS: ``s_wait_dscnt``14657 * Load (global, scratch, flat, buffer and image): ``s_wait_loadcnt``14658 * Store (global, scratch, flat, buffer and image): ``s_wait_storecnt``14659 * Sample and Gather4: ``s_wait_samplecnt``14660 * BVH: ``s_wait_bvhcnt``14661 14662* Vector and scalar memory instructions contain a ``SCOPE`` field with values14663 corresponding to each cache level. The ``SCOPE`` determines whether a cache14664 can complete an operation locally or whether it needs to forward the operation14665 to the next cache level. The ``SCOPE`` values are:14666 14667 * ``SCOPE_CU``: Compute Unit (NOTE: not affected by CU/WGP mode)14668 * ``SCOPE_SE``: Shader Engine14669 * ``SCOPE_DEV``: Device/Agent14670 * ``SCOPE_SYS``: System14671 14672* When a memory operation with a given ``SCOPE`` reaches a cache with a smaller14673 ``SCOPE`` value, it is forwarded to the next level of cache.14674* When a memory operation with a given ``SCOPE`` reaches a cache with a ``SCOPE``14675 value greater than or equal to its own, the operation can proceed:14676 14677 * Reads can hit into the cache14678 * Writes can happen in this cache and the transaction is acknowledged14679 from this level of cache.14680 * RMW operations can be done locally.14681 14682* ``global_inv``, ``global_wb`` and ``global_wbinv`` instructions are used to14683 invalidate, write-back and write-back+invalidate caches. The affected14684 cache(s) are controlled by the ``SCOPE:`` of the instruction.14685* ``global_inv`` invalidates caches whose scope is strictly smaller than the14686 instruction's. The invalidation requests cannot be reordered with pending or14687 upcoming memory operations.14688* ``global_wb`` is a writeback operation that additionally ensures previous14689 memory operation done at a lower scope level have reached the ``SCOPE:``14690 of the ``global_wb``.14691 14692 * ``global_wb`` can be omitted for scopes other than ``SCOPE_SYS`` in14693 gfx120x.14694 14695* The vector memory operations access a vector L0 cache. There is a single L014696 cache per CU. Each SIMD of a CU accesses the same L0 cache. Therefore, no14697 special action is required for coherence between the lanes of a single14698 wavefront. To achieve coherence between wavefronts executing in the same14699 work-group:14700 14701 * In CU wavefront execution mode, no special action is required.14702 * In WGP wavefront execution mode, a ``global_inv scope:SCOPE_SE`` is required14703 as wavefronts may be executing on SIMDs of different CUs that access different L0s.14704 14705* The scalar memory operations access a scalar L0 cache shared by all wavefronts14706 on a WGP. The scalar and vector L0 caches are not coherent. However, scalar14707 operations are used in a restricted way so do not impact the memory model. See14708 :ref:`amdgpu-amdhsa-memory-spaces`.14709* The vector and scalar memory L0 caches use an L1 buffer shared by all WGPs on14710 the same SA. The L1 buffer acts as a bridge to L2 for clients within a SA.14711* The L1 buffers have independent quadrants to service disjoint ranges of virtual14712 addresses.14713* Each L0 cache has a separate request queue per L1 quadrant. Therefore, the14714 vector and scalar memory operations performed by different wavefronts, whether14715 executing in the same or different work-groups (which may be executing on14716 different CUs accessing different L0s), can be reordered relative to each14717 other. Some or all of the wait instructions below are required to ensure14718 synchronization between vector memory operations of different wavefronts. It14719 ensures a previous vector memory operation has completed before executing a14720 subsequent vector memory or LDS operation and so can be used to meet the14721 requirements of acquire, release and sequential consistency.14722 14723 * ``s_wait_loadcnt 0x0``14724 * ``s_wait_samplecnt 0x0``14725 * ``s_wait_bvhcnt 0x0``14726 * ``s_wait_storecnt 0x0``14727 14728* The L1 buffers use an L2 cache shared by all SAs on the same agent.14729* The L2 cache has independent channels to service disjoint ranges of virtual14730 addresses.14731* Each L1 quadrant of a single SA accesses a different L2 channel. Each L114732 quadrant has a separate request queue per L2 channel. Therefore, the vector14733 and scalar memory operations performed by wavefronts executing in different14734 work-groups (which may be executing on different SAs) of an agent can be14735 reordered relative to each other. Some or all of the wait instructions below are14736 required to ensure synchronization between vector memory operations of14737 different SAs. It ensures a previous vector memory operation has completed14738 before executing a subsequent vector memory and so can be used to meet the14739 requirements of acquire, release and sequential consistency.14740 14741 * ``s_wait_loadcnt 0x0``14742 * ``s_wait_samplecnt 0x0``14743 * ``s_wait_bvhcnt 0x0``14744 * ``s_wait_storecnt 0x0``14745 14746* The L2 cache can be kept coherent with other agents, or ranges14747 of virtual addresses can be set up to bypass it to ensure system coherence.14748* A memory attached last level (MALL) cache exists for GPU memory.14749 The MALL cache is fully coherent with GPU memory and has no impact on system14750 coherence. All agents (GPU and CPU) access GPU memory through the MALL cache.14751 14752Scalar memory operations are only used to access memory that is proven to not14753change during the execution of the kernel dispatch. This includes constant14754address space and global address space for program scope ``const`` variables.14755Therefore, the kernel machine code does not have to maintain the scalar cache to14756ensure it is coherent with the vector caches. The scalar and vector caches are14757invalidated between kernel dispatches by CP since constant address space data14758may change between kernel dispatch executions. See14759:ref:`amdgpu-amdhsa-memory-spaces`.14760 14761For kernarg backing memory:14762 14763* CP invalidates caches at the start of each kernel dispatch.14764* On dGPU the kernarg backing memory is accessed as MTYPE UC (uncached) to avoid14765 needing to invalidate the L2 cache.14766* On APU the kernarg backing memory is accessed as MTYPE CC (cache coherent) and14767 so the L2 cache will be coherent with the CPU and other agents.14768 14769Scratch backing memory (which is used for the private address space) is accessed14770with MTYPE NC (non-coherent). Since the private address space is only accessed14771by a single thread, and is always write-before-read, there is never a need to14772invalidate these entries from L0.14773 14774Wavefronts can be executed in WGP or CU wavefront execution mode:14775 14776* In WGP wavefront execution mode the wavefronts of a work-group are executed14777 on the SIMDs of both CUs of the WGP. Therefore, explicit management of the per14778 CU L0 caches is required for work-group synchronization. Also accesses to L114779 at work-group scope need to be explicitly ordered as the accesses from14780 different CUs are not ordered.14781* In CU wavefront execution mode the wavefronts of a work-group are executed on14782 the SIMDs of a single CU of the WGP. Therefore, all global memory access by14783 the work-group access the same L0 which in turn ensures L1 accesses are14784 ordered and so do not require explicit management of the caches for14785 work-group synchronization.14786 14787See ``WGP_MODE`` field in14788:ref:`amdgpu-amdhsa-compute_pgm_rsrc1-gfx6-gfx12-table` and14789:ref:`amdgpu-target-features`.14790 14791The code sequences used to implement the memory model for GFX12 are defined in14792table :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx12-table`.14793 14794The mapping of LLVM IR syncscope to GFX12 instruction ``scope`` operands is14795defined in :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx12-scopes-table`.14796 14797The table only applies if and only if it is directly referenced by an entry in14798:ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx12-table`, and it only applies to14799the instruction in the code sequence that references the table.14800 14801 .. table:: AMDHSA Memory Model Code Sequences GFX12 - Instruction Scopes14802 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx12-scopes-table14803 14804 =================== =================== ===================14805 LLVM syncscope CU wavefront WGP wavefront14806 execution execution14807 mode mode14808 =================== =================== ===================14809 *none* ``scope:SCOPE_SYS`` ``scope:SCOPE_SYS``14810 system ``scope:SCOPE_SYS`` ``scope:SCOPE_SYS``14811 agent ``scope:SCOPE_DEV`` ``scope:SCOPE_DEV``14812 workgroup *none* ``scope:SCOPE_SE``14813 wavefront *none* *none*14814 singlethread *none* *none*14815 one-as ``scope:SCOPE_SYS`` ``scope:SCOPE_SYS``14816 system-one-as ``scope:SCOPE_SYS`` ``scope:SCOPE_SYS``14817 agent-one-as ``scope:SCOPE_DEV`` ``scope:SCOPE_DEV``14818 workgroup-one-as *none* ``scope:SCOPE_SE``14819 wavefront-one-as *none* *none*14820 singlethread-one-as *none* *none*14821 =================== =================== ===================14822 14823 .. table:: AMDHSA Memory Model Code Sequences GFX1214824 :name: amdgpu-amdhsa-memory-model-code-sequences-gfx12-table14825 14826 ============ ============ ============== ========== ================================14827 LLVM Instr LLVM Memory LLVM Memory AMDGPU AMDGPU Machine Code14828 Ordering Sync Scope Address GFX1214829 Space14830 ============ ============ ============== ========== ================================14831 **Non-Atomic**14832 ------------------------------------------------------------------------------------14833 load *none* *none* - global - !volatile & !nontemporal14834 - generic14835 - private 1. buffer/global/flat_load14836 - constant14837 - !volatile & nontemporal14838 14839 1. buffer/global/flat_load14840 ``th:TH_LOAD_NT``14841 14842 - volatile14843 14844 1. buffer/global/flat_load14845 ``scope:SCOPE_SYS``14846 14847 2. ``s_wait_loadcnt 0x0``14848 14849 - Must happen before14850 any following volatile14851 global/generic14852 load/store.14853 - Ensures that14854 volatile14855 operations to14856 different14857 addresses will not14858 be reordered by14859 hardware.14860 14861 load *none* *none* - local 1. ds_load14862 store *none* *none* - global - !volatile & !nontemporal14863 - generic14864 - private 1. buffer/global/flat_store14865 - constant14866 - !volatile & nontemporal14867 14868 1. buffer/global/flat_store14869 ``th:TH_STORE_NT``14870 14871 - volatile14872 14873 1. buffer/global/flat_store14874 ``scope:SCOPE_SYS``14875 14876 2. ``s_wait_storecnt 0x0``14877 14878 - Must happen before14879 any following volatile14880 global/generic14881 load/store.14882 - Ensures that14883 volatile14884 operations to14885 different14886 addresses will not14887 be reordered by14888 hardware.14889 14890 store *none* *none* - local 1. ds_store14891 **Unordered Atomic**14892 ------------------------------------------------------------------------------------14893 load atomic unordered *any* *any* *Same as non-atomic*.14894 store atomic unordered *any* *any* *Same as non-atomic*.14895 atomicrmw unordered *any* *any* *Same as monotonic atomic*.14896 **Monotonic Atomic**14897 ------------------------------------------------------------------------------------14898 load atomic monotonic - singlethread - global 1. buffer/global/flat_load14899 - wavefront - generic14900 - workgroup - Apply :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx12-scopes-table`.14901 - agent14902 - system14903 load atomic monotonic - singlethread - local 1. ds_load14904 - wavefront14905 - workgroup14906 store atomic monotonic - singlethread - global 1. buffer/global/flat_store14907 - wavefront - generic14908 - workgroup - Apply :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx12-scopes-table`.14909 - agent14910 - system14911 store atomic monotonic - singlethread - local 1. ds_store14912 - wavefront14913 - workgroup14914 atomicrmw monotonic - singlethread - global 1. buffer/global/flat_atomic14915 - wavefront - generic14916 - workgroup - Apply :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx12-scopes-table`.14917 - agent14918 - system14919 atomicrmw monotonic - singlethread - local 1. ds_atomic14920 - wavefront14921 - workgroup14922 **Acquire Atomic**14923 ------------------------------------------------------------------------------------14924 load atomic acquire - singlethread - global 1. buffer/global/ds/flat_load14925 - wavefront - local14926 - generic14927 load atomic acquire - workgroup - global 1. buffer/global_load ``scope:SCOPE_SE``14928 14929 - Apply :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx12-scopes-table`.14930 14931 2. ``s_wait_loadcnt 0x0``14932 14933 - If CU wavefront execution14934 mode, omit.14935 - Must happen before14936 the following ``global_inv``14937 and before any following14938 global/generic14939 load/load14940 atomic/store/store14941 atomic/atomicrmw.14942 14943 3. ``global_inv scope:SCOPE_SE``14944 14945 - If CU wavefront execution14946 mode, omit.14947 - Ensures that14948 following14949 loads will not see14950 stale data.14951 14952 load atomic acquire - workgroup - local 1. ds_load14953 2. ``s_wait_dscnt 0x0``14954 14955 - If OpenCL, omit.14956 - Must happen before14957 the following ``global_inv``14958 and before any following14959 global/generic load/load14960 atomic/store/store14961 atomic/atomicrmw.14962 - Ensures any14963 following global14964 data read is no14965 older than the local load14966 atomic value being14967 acquired.14968 14969 3. ``global_inv scope:SCOPE_SE``14970 14971 - If OpenCL or CU wavefront14972 execution mode, omit.14973 - Ensures that14974 following14975 loads will not see14976 stale data.14977 14978 load atomic acquire - workgroup - generic 1. flat_load14979 14980 - Apply :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx12-scopes-table`.14981 14982 2. | ``s_wait_loadcnt 0x0``14983 | ``s_wait_dscnt 0x0``14984 | **CU wavefront execution mode:**14985 | ``s_wait_dscnt 0x0``14986 14987 - If OpenCL, omit ``s_wait_dscnt 0x0``14988 - Must happen before14989 the following14990 ``global_inv`` and any14991 following global/generic14992 load/load14993 atomic/store/store14994 atomic/atomicrmw.14995 - Ensures any14996 following global14997 data read is no14998 older than a local load14999 atomic value being15000 acquired.15001 15002 3. ``global_inv scope:SCOPE_SE``15003 15004 - If CU wavefront execution15005 mode, omit.15006 - Ensures that15007 following15008 loads will not see15009 stale data.15010 15011 load atomic acquire - agent - global 1. buffer/global_load15012 - system15013 - Apply :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx12-scopes-table`.15014 15015 2. ``s_wait_loadcnt 0x0``15016 15017 - Must happen before15018 following15019 ``global_inv``.15020 - Ensures the load15021 has completed15022 before invalidating15023 the caches.15024 15025 3. ``global_inv``15026 15027 - Apply :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx12-scopes-table`.15028 - Must happen before15029 any following15030 global/generic15031 load/load15032 atomic/atomicrmw.15033 - Ensures that15034 following15035 loads will not see15036 stale global data.15037 15038 load atomic acquire - agent - generic 1. flat_load15039 - system15040 - Apply :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx12-scopes-table`.15041 15042 2. | ``s_wait_loadcnt 0x0``15043 | ``s_wait_dscnt 0x0``15044 15045 - If OpenCL, omit ``s_wait_dscnt 0x0``15046 - Must happen before15047 following15048 ``global_inv``.15049 - Ensures the flat_load15050 has completed15051 before invalidating15052 the caches.15053 15054 3. ``global_inv``15055 15056 - Apply :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx12-scopes-table`.15057 - Must happen before15058 any following15059 global/generic15060 load/load15061 atomic/atomicrmw.15062 - Ensures that15063 following loads15064 will not see stale15065 global data.15066 15067 atomicrmw acquire - singlethread - global 1. buffer/global/ds/flat_atomic15068 - wavefront - local15069 - generic15070 atomicrmw acquire - workgroup - global 1. buffer/global_atomic15071 15072 - Apply :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx12-scopes-table`.15073 - If atomic with return,15074 use ``th:TH_ATOMIC_RETURN``15075 15076 2. | **Atomic with return:**15077 | ``s_wait_loadcnt 0x0``15078 | **Atomic without return:**15079 | ``s_wait_storecnt 0x0``15080 15081 - If CU wavefront execution15082 mode, omit.15083 - Must happen before15084 the following ``global_inv``15085 and before any following15086 global/generic15087 load/load15088 atomic/store/store15089 atomic/atomicrmw.15090 15091 3. ``global_inv scope:SCOPE_SE``15092 15093 - If CU wavefront execution15094 mode, omit.15095 - Ensures that15096 following15097 loads will not see15098 stale data.15099 15100 atomicrmw acquire - workgroup - local 1. ds_atomic15101 2. ``s_wait_dscnt 0x0``15102 15103 - If OpenCL, omit.15104 - Must happen before15105 the following15106 ``global_inv``.15107 - Ensures any15108 following global15109 data read is no15110 older than the local15111 atomicrmw value15112 being acquired.15113 15114 3. ``global_inv scope:SCOPE_SE``15115 15116 - If OpenCL omit.15117 - If CU wavefront execution15118 mode, omit.15119 - Ensures that15120 following15121 loads will not see15122 stale data.15123 15124 atomicrmw acquire - workgroup - generic 1. flat_atomic15125 15126 - Apply :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx12-scopes-table`.15127 - If atomic with return,15128 use ``th:TH_ATOMIC_RETURN``15129 15130 2. | **Atomic with return:**15131 | ``s_wait_loadcnt 0x0``15132 | ``s_wait_dscnt 0x0``15133 | **Atomic without return:**15134 | ``s_wait_storecnt 0x0``15135 | ``s_wait_dscnt 0x0``15136 15137 - If CU wavefront execution mode,15138 omit all for atomics without15139 return, and only emit15140 ``s_wait_dscnt 0x0`` for atomics15141 with return.15142 - If OpenCL, omit ``s_wait_dscnt 0x0``15143 - Must happen before15144 the following15145 ``global_inv``.15146 - Ensures any15147 following global15148 data read is no15149 older than a local15150 atomicrmw value15151 being acquired.15152 15153 3. ``global_inv scope:SCOPE_SE``15154 15155 - If CU wavefront execution15156 mode, omit.15157 - Ensures that15158 following15159 loads will not see15160 stale data.15161 15162 atomicrmw acquire - agent - global 1. buffer/global_atomic15163 - system15164 - Apply :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx12-scopes-table`.15165 - If atomic with return,15166 use ``th:TH_ATOMIC_RETURN``15167 15168 2. | **Atomic with return:**15169 | ``s_wait_loadcnt 0x0``15170 | **Atomic without return:**15171 | ``s_wait_storecnt 0x0``15172 15173 - Must happen before15174 following ``global_inv``.15175 - Ensures the15176 atomicrmw has15177 completed before15178 invalidating the15179 caches.15180 15181 3. ``global_inv``15182 15183 - Apply :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx12-scopes-table`.15184 - Must happen before15185 any following15186 global/generic15187 load/load15188 atomic/atomicrmw.15189 - Ensures that15190 following loads15191 will not see stale15192 global data.15193 15194 atomicrmw acquire - agent - generic 1. flat_atomic15195 - system15196 - Apply :ref:`amdgpu-amdhsa-memory-model-code-sequences-gfx12-scopes-table`.15197 - If atomic with return,15198 use ``th:TH_ATOMIC_RETURN``15199 15200 2. | **Atomic with return:**15201 | ``s_wait_loadcnt 0x0``15202 | ``s_wait_dscnt 0x0``15203 | **Atomic without return:**15204 | ``s_wait_storecnt 0x0``15205 | ``s_wait_dscnt 0x0``15206 15207 - If OpenCL, omit dscnt15208 - Must happen before15209 following15210 File truncated to the first 1 MiB.