408 lines · plain
1Support, Getting Involved, and FAQ2==================================3 4Please do not hesitate to reach out to us on the `Discourse forums (Runtimes - OpenMP) <https://discourse.llvm.org/c/runtimes/openmp/35>`_ or join5one of our :ref:`regular calls <calls>`. Some common questions are answered in6the :ref:`faq`.7 8.. _calls:9 10Calls11-----12 13OpenMP in LLVM Technical Call14^^^^^^^^^^^^^^^^^^^^^^^^^^^^^15 16- Development updates on OpenMP (and OpenACC) in the LLVM Project, including Clang, optimization, and runtime work.17- Join `OpenMP in LLVM Technical Call <https://bluejeans.com/544112769//webrtc>`__.18- Time: Weekly call on every Wednesday 7:00 AM Pacific time.19- Meeting minutes are `here <https://docs.google.com/document/d/1Tz8WFN13n7yJ-SCE0Qjqf9LmjGUw0dWO9Ts1ss4YOdg/edit>`__.20- Status tracking `page <https://openmp.llvm.org/docs>`__.21 22 23OpenMP in Flang Technical Call24^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^25- Development updates on OpenMP and OpenACC in the Flang Project.26- Join `OpenMP in Flang Technical Call <https://bit.ly/39eQW3o>`_27- Time: Weekly call on every Thursdays 8:00 AM Pacific time.28- Meeting minutes are `here <https://docs.google.com/document/d/1yA-MeJf6RYY-ZXpdol0t7YoDoqtwAyBhFLr5thu5pFI>`__.29- Status tracking `page <https://docs.google.com/spreadsheets/d/1FvHPuSkGbl4mQZRAwCIndvQx9dQboffiD-xD0oqxgU0/edit#gid=0>`__.30 31 32.. _faq:33 34FAQ35---36 37.. note::38 The FAQ is a work in progress and most of the expected content is not39 yet available. While you can expect changes, we always welcome feedback and40 additions. Please post on the `Discourse forums (Runtimes - OpenMP) <https://discourse.llvm.org/c/runtimes/openmp/35>`__.41 42 43Q: How to contribute a patch to the webpage or any other part?44^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^45 46All patches go through the regular `LLVM review process47<https://llvm.org/docs/Contributing.html#how-to-submit-a-patch>`_.48 49 50.. _build_offload_capable_compiler:51 52Q: How to build an OpenMP GPU offload capable compiler?53^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^54 55The easiest way to create an offload capable compiler is to use the provided 56CMake cache file. This will enable the projects and runtimes necessary for 57offloading as well as some extra options.58 59.. code-block:: sh60 61 $> cd llvm-project # The llvm-project checkout62 $> mkdir build63 $> cd build64 $> cmake ../llvm -G Ninja \65 -C ../offload/cmake/caches/Offload.cmake \ # The preset cache file66 -DCMAKE_BUILD_TYPE=<Debug|Release> \ # Select build type67 -DCMAKE_INSTALL_PREFIX=<PATH> \ # Where the libraries will live68 $> ninja install69 70To manually build an *effective* OpenMP offload capable compiler, only one extra CMake71option, ``LLVM_ENABLE_RUNTIMES="openmp;offload"``, is needed when building LLVM (Generic72information about building LLVM is available `here73<https://llvm.org/docs/GettingStarted.html>`__.). Make sure all backends that74are targeted by OpenMP are enabled. That can be done by adjusting the CMake 75option ``LLVM_TARGETS_TO_BUILD``. The corresponding targets for offloading to AMD 76and Nvidia GPUs are ``"AMDGPU"`` and ``"NVPTX"``, respectively. By default, 77Clang will be built with all backends enabled. When building with 78``LLVM_ENABLE_RUNTIMES="openmp"`` OpenMP should not be enabled in 79``LLVM_ENABLE_PROJECTS`` because it is enabled by default.80 81Support for the device library comes from a separate build of the OpenMP library82that targets the GPU architecture. Building it requires enabling the runtime83targets, or setting the target manually when doing a standalone build. This is84done with the ``LLVM_RUNTIME_TARGETS`` option and then enabling the OpenMP85runtime for the GPU target via ``RUNTIMES_<triple>_LLVM_ENABLE_RUNTIMES``.86It's possible to set different flags for each device library by using87``RUNTIMES_<triple>_CMAKE_CXX_FLAGS``. Refer to the cache file for the specific88invocation.89 90For Nvidia offload, please see :ref:`build_nvidia_offload_capable_compiler`.91For AMDGPU offload, please see :ref:`build_amdgpu_offload_capable_compiler`.92 93.. note::94 The compiler that generates the offload code should be the same (version) as95 the compiler that builds the OpenMP device runtimes. The OpenMP host runtime96 can be built by a different compiler.97 98.. _advanced_builds: https://llvm.org//docs/AdvancedBuilds.html99 100.. _build_nvidia_offload_capable_compiler:101 102Q: How to build an OpenMP Nvidia offload capable compiler?103^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^104The CUDA SDK is required on the machine that will build and execute the105offloading application. Normally this is only required at runtime by dynamically106opening the CUDA driver API. This can be disabled in the build by omitting107``cuda`` from the ``LIBOMPTARGET_DLOPEN_PLUGINS`` list which is present by108default. With this setting we will instead find the CUDA library at LLVM build109time and link against it directly.110 111.. _build_amdgpu_offload_capable_compiler:112 113Q: How to build an OpenMP AMDGPU offload capable compiler?114^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^115 116The OpenMP AMDGPU offloading support depends on the ROCm math libraries and the117HSA ROCr / ROCt runtimes. These are normally provided by a standard ROCm118installation, but can be built and used independently if desired. Building the119libraries does not depend on these libraries by default by dynamically loading120the HSA runtime at program execution. As in the CUDA case, this can be change by121omitting ``amdgpu`` from the ``LIBOMPTARGET_DLOPEN_PLUGINS`` list.122 123Q: What are the known limitations of OpenMP AMDGPU offload?124^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^125 126LD_LIBRARY_PATH or rpath/runpath are required to find libomp.so and127libomptarget.so correctly. The recommended way to configure this is with the128``-frtlib-add-rpath`` option. Alternatively, set the ``LD_LIBRARY_PATH``129environment variable to point to the installation. Normally, these libraries are130installed in the target specific runtime directory. For example, a typical131installation will have132``<install>/lib/x86_64-unknown-linux-gnu/llibomptarget.so``133 134Some versions of the driver for the radeon vii (gfx906) will error unless the135environment variable 'export HSA_IGNORE_SRAMECC_MISREPORT=1' is set.136 137Q: What are the LLVM components used in offloading and how are they found?138^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^139The libraries used by an executable compiled for target offloading are:140 141- ``libomp.so`` (or similar), the host openmp runtime142- ``libomptarget.so``, the target-agnostic target offloading openmp runtime143- ``libompdevice.a``, the device-side OpenMP runtime.144- dependencies of those plugins, e.g. cuda/rocr for nvptx/amdgpu145 146The compiled executable is dynamically linked against a host runtime, e.g.147``libomp.so``, and against the target offloading runtime, ``libomptarget.so``. These148are found like any other dynamic library, by setting rpath or runpath on the149executable, by setting ``LD_LIBRARY_PATH``, or by adding them to the system search.150 151``libomptarget.so`` is only supported to work with the associated ``clang`` 152compiler. On systems with globally installed ``libomptarget.so`` this can be 153problematic. For this reason it is recommended to use a `Clang configuration 154file <https://clang.llvm.org/docs/UsersManual.html#configuration-files>`__ to 155automatically configure the environment. For example, store the following file 156as ``openmp.cfg`` next to your ``clang`` executable.157 158.. code-block:: text159 160 # Library paths for OpenMP offloading.161 -L '<CFGDIR>/../lib'162 -Wl,-rpath='<CFGDIR>/../lib'163 164The plugins will try to find their dependencies in plugin-dependent fashion.165 166The cuda plugin is dynamically linked against libcuda if cmake found it at167compiler build time. Otherwise it will attempt to dlopen ``libcuda.so``. It does168not have rpath set.169 170The amdgpu plugin is linked against ROCr if cmake found it at compiler build171time. Otherwise it will attempt to dlopen ``libhsa-runtime64.so``. It has rpath172set to ``$ORIGIN``, so installing ``libhsa-runtime64.so`` in the same directory is a173way to locate it without environment variables.174 175In addition to those, there is a compiler runtime library called deviceRTL.176This is compiled from mostly common code into an architecture specific177bitcode library, e.g. ``libomptarget-nvptx-sm_70.bc``.178 179Clang and the deviceRTL need to match closely as the interface between them180changes frequently. Using both from the same monorepo checkout is strongly181recommended.182 183Unlike the host side which lets environment variables select components, the184deviceRTL that is located in the clang lib directory is preferred. Only if185it is absent, the ``LIBRARY_PATH`` environment variable is searched to find a186bitcode file with the right name. This can be overridden by passing a clang187flag, ``--libomptarget-nvptx-bc-path`` or ``--libomptarget-amdgcn-bc-path``. That188can specify a directory or an exact bitcode file to use.189 190 191Q: Does OpenMP offloading support work in pre-packaged LLVM releases?192^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^193For now, the answer is most likely *no*. Please see :ref:`build_offload_capable_compiler`.194 195Q: Does OpenMP offloading support work in packages distributed as part of my OS?196^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^197For now, the answer is most likely *no*. Please see :ref:`build_offload_capable_compiler`.198 199.. _math_and_complex_in_target_regions:200 201Q: Does Clang support `<math.h>` and `<complex.h>` operations in OpenMP target on GPUs?202^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^203 204Yes, LLVM/Clang allows math functions and complex arithmetic inside of OpenMP205target regions that are compiled for GPUs.206 207Clang provides a set of wrapper headers that are found first when `math.h` and208`complex.h`, for C, `cmath` and `complex`, for C++, or similar headers are209included by the application. These wrappers will eventually include the system210version of the corresponding header file after setting up a target device211specific environment. The fact that the system header is included is important212because they differ based on the architecture and operating system and may213contain preprocessor, variable, and function definitions that need to be214available in the target region regardless of the targeted device architecture.215However, various functions may require specialized device versions, e.g.,216`sin`, and others are only available on certain devices, e.g., `__umul64hi`. To217provide "native" support for math and complex on the respective architecture,218Clang will wrap the "native" math functions, e.g., as provided by the device219vendor, in an OpenMP begin/end declare variant. These functions will then be220picked up instead of the host versions while host only variables and function221definitions are still available. Complex arithmetic and functions are support222through a similar mechanism. It is worth noting that this support requires223`extensions to the OpenMP begin/end declare variant context selector224<https://clang.llvm.org/docs/AttributeReference.html#pragma-omp-declare-variant>`__225that are exposed through LLVM/Clang to the user as well.226 227Q: Can I use dynamically linked libraries with OpenMP offloading?228^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^229 230Dynamically linked libraries can be used if there is no device code shared231between the library and application. Anything declared on the device inside the232shared library will not be visible to the application when it's linked. This is233because device code only supports static linking.234 235Q: How to build an OpenMP offload capable compiler with an outdated host compiler?236^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^237 238Enabling the OpenMP runtime will perform a two-stage build for you.239If your host compiler is different from your system-wide compiler, you may need240to set ``CMAKE_{C,CXX}_FLAGS`` like241``--gcc-install-dir=/usr/lib/gcc/x86_64-linux-gnu/12`` so that clang will be242able to find the correct GCC toolchain in the second stage of the build.243 244For example, if your system-wide GCC installation is too old to build LLVM and245you would like to use a newer GCC, set ``--gcc-install-dir=``246to inform clang of the GCC installation you would like to use in the second stage.247 248 249Q: What does 'Stack size for entry function cannot be statically determined' mean?250^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^251 252This is a warning that the Nvidia tools will sometimes emit if the offloading253region is too complex. Normally, the CUDA tools attempt to statically determine254how much stack memory each thread. This way when the kernel is launched each255thread will have as much memory as it needs. If the control flow of the kernel256is too complex, containing recursive calls or nested parallelism, this analysis257can fail. If this warning is triggered it means that the kernel may run out of258stack memory during execution and crash. The environment variable259``LIBOMPTARGET_STACK_SIZE`` can be used to increase the stack size if this260occurs.261 262Q: Can OpenMP offloading compile for multiple architectures?263^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^264 265Since LLVM version 15.0, OpenMP offloading supports offloading to multiple266architectures at once. This allows for executables to be run on different267targets, such as offloading to AMD and NVIDIA GPUs simultaneously, as well as268multiple sub-architectures for the same target. Additionally, static libraries269will only extract archive members if an architecture is used, allowing users to270create generic libraries.271 272The architecture can either be specified manually using ``--offload-arch=``. If273``--offload-arch=`` is present and no ``-fopenmp-targets=`` flag is present then274the targets will be inferred from the architectures. Conversely, if275``--fopenmp-targets=`` is present with no ``--offload-arch`` then the target276architecture will be set to a default value, usually the architecture supported277by the system LLVM was built on by executing the ``offload-arch`` utility.278 279For example, an executable can be built that runs on AMDGPU and NVIDIA hardware280given that the necessary build tools are installed for both.281 282.. code-block:: shell283 284 clang example.c -fopenmp --offload-arch=gfx90a --offload-arch=sm_80285 286If just given the architectures we should be able to infer the triples,287otherwise we can specify them manually.288 289.. code-block:: shell290 291 clang example.c -fopenmp -fopenmp-targets=amdgcn-amd-amdhsa,nvptx64-nvidia-cuda \292 -Xopenmp-target=amdgcn-amd-amdhsa --offload-arch=gfx90a \293 -Xopenmp-target=nvptx64-nvidia-cuda --offload-arch=sm_80294 295When linking against a static library that contains device code for multiple296architectures, only the images used by the executable will be extracted.297 298.. code-block:: shell299 300 clang example.c -fopenmp --offload-arch=gfx90a,gfx90a,sm_70,sm_80 -c301 llvm-ar rcs libexample.a example.o302 clang app.c -fopenmp --offload-arch=gfx90a -o app303 304The supported device images can be viewed using the ``--offloading`` option with305``llvm-objdump``.306 307.. code-block:: shell308 309 clang example.c -fopenmp --offload-arch=gfx90a --offload-arch=sm_80 -o example310 llvm-objdump --offloading example311 312 a.out: file format elf64-x86-64313 314 OFFLOADING IMAGE [0]:315 kind elf316 arch gfx90a317 triple amdgcn-amd-amdhsa318 producer openmp319 320 OFFLOADING IMAGE [1]:321 kind elf322 arch sm_80323 triple nvptx64-nvidia-cuda324 producer openmp325 326Q: Can I link OpenMP offloading with CUDA or HIP?327^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^328 329OpenMP offloading files can currently be experimentally linked with CUDA and HIP330files. This will allow OpenMP to call a CUDA device function or vice-versa.331However, the global state will be distinct between the two images at runtime.332This means any global variables will potentially have different values when333queried from OpenMP or CUDA.334 335Linking CUDA and HIP currently requires enabling a different compilation mode336for CUDA / HIP with ``--offload-new-driver`` and to link using337``--offload-link``. Additionally, ``-fgpu-rdc`` must be used to create a338linkable device image.339 340.. code-block:: shell341 342 clang++ openmp.cpp -fopenmp --offload-arch=sm_80 -c343 clang++ cuda.cu --offload-new-driver --offload-arch=sm_80 -fgpu-rdc -c344 clang++ openmp.o cuda.o --offload-link -o app345 346Q: Are libomptarget and plugins backward compatible?347^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^348 349No. libomptarget and plugins are now built as LLVM libraries starting from LLVM35015. Because LLVM libraries are not backward compatible, libomptarget and plugins351are not as well. Given that fact, the interfaces between 1) the Clang compiler352and libomptarget, 2) the Clang compiler and device runtime library, and3533) libomptarget and plugins are not guaranteed to be compatible with an earlier354version. Users are responsible for ensuring compatibility when not using the355Clang compiler and runtime libraries from the same build. Nevertheless, in order356to better support third-party libraries and toolchains that depend on existing357libomptarget entry points, contributors are discouraged from making358modifications to them.359 360Q: Can I use libc functions on the GPU?361^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^362 363LLVM provides basic ``libc`` functionality through the LLVM C Library. For 364building instructions, refer to the associated `LLVM libc documentation 365<https://libc.llvm.org/gpu/using.html#building-the-gpu-library>`_. Once built, 366this provides a static library called ``libcgpu.a``. See the documentation for a 367list of `supported functions <https://libc.llvm.org/gpu/support.html>`_ as well. 368To utilize these functions, simply link this library as any other when building 369with OpenMP.370 371.. code-block:: shell372 373 clang++ openmp.cpp -fopenmp --offload-arch=gfx90a -Xoffload-linker -lc374 375For more information on how this is implemented in LLVM/OpenMP's offloading 376runtime, refer to the `runtime documentation <libomptarget_libc>`_.377 378Q: What command line options can I use for OpenMP?379^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^380We recommend taking a look at the OpenMP 381:doc:`command line argument reference <CommandLineArgumentReference>` page.382 383Q: Can I build the offloading runtimes without CUDA or HSA?384^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^385By default, the offloading runtime will load the associated vendor runtime 386during initialization rather than directly linking against them. This allows the 387program to be built and run on many machine. If you wish to directly link 388against these libraries, use the ``LIBOMPTARGET_DLOPEN_PLUGINS=""`` option to 389suppress it for each plugin. The default value is every plugin enabled with 390``LIBOMPTARGET_PLUGINS_TO_BUILD``.391 392Q: Why is my build taking a long time?393^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^394When installing OpenMP and other LLVM components, the build time on multicore 395systems can be significantly reduced with parallel build jobs. As suggested in 396*LLVM Techniques, Tips, and Best Practices*, one could consider using ``ninja`` as the397generator. This can be done with the CMake option ``cmake -G Ninja``. Afterward, 398use ``ninja install`` and specify the number of parallel jobs with ``-j``. The build399time can also be reduced by setting the build type to ``Release`` with the 400``CMAKE_BUILD_TYPE`` option. Recompilation can also be sped up by caching previous401compilations. Consider enabling ``Ccache`` with 402``CMAKE_CXX_COMPILER_LAUNCHER=ccache``.403 404Q: Did this FAQ not answer your question?405^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^406Feel free to post questions or browse old threads at 407`LLVM Discourse <https://discourse.llvm.org/c/runtimes/openmp/>`__.408