1296 lines · plain
1========================2Debugging C++ Coroutines3========================4 5.. contents::6 :local:7 8Introduction9============10 11Coroutines in C++ were introduced in C++20, and the user experience for12debugging them can still be challenging. This document guides you on how to most13efficiently debug coroutines and how to navigate existing shortcomings in14debuggers and compilers.15 16Coroutines are generally used either as generators or for asynchronous17programming. In this document, we will discuss both use cases. Even if you are18using coroutines for asynchronous programming, you should still read the19generators section, as it introduces foundational debugging techniques also20applicable to the debugging of asynchronous programs.21 22Both compilers (clang, gcc, ...) and debuggers (lldb, gdb, ...) are23still improving their support for coroutines. As such, we recommend using the24latest available version of your toolchain.25 26This document focuses on clang and lldb. The screenshots show27`lldb-dap <https://marketplace.visualstudio.com/items?itemName=llvm-vs-code-extensions.lldb-dap>`_28in combination with VS Code. The same techniques can also be used in other29IDEs.30 31Debugging clang-compiled binaries with gdb is possible, but requires more32scripting. This guide comes with a basic GDB script for coroutine debugging.33 34This guide will first showcase the more polished, bleeding-edge experience, but35will also show you how to debug coroutines with older toolchains. In general,36the older your toolchain, the deeper you will have to dive into the37implementation details of coroutines (such as their ABI). The further down you go in38this document, the more low-level, technical the content will become. If39you are on an up-to-date toolchain, you will hopefully be able to stop reading40earlier.41 42Debugging generators43====================44 45One of the two major use cases for coroutines in C++ is generators, i.e.,46functions which can produce values via ``co_yield``. Values are produced47lazily, on-demand. For this purpose, every time a new value is requested, the48coroutine gets resumed. As soon as it reaches a ``co_yield`` and thereby49returns the requested value, the coroutine is suspended again.50 51This logic is encapsulated in a ``generator`` type similar to this one:52 53.. code-block:: c++54 55 // generator.hpp56 #include <coroutine>57 58 // `generator` is a stripped down, minimal generator type.59 template<typename T>60 struct generator {61 struct promise_type {62 T current_value{};63 64 auto get_return_object() {65 return std::coroutine_handle<promise_type>::from_promise(*this);66 }67 auto initial_suspend() { return std::suspend_always(); }68 auto final_suspend() noexcept { return std::suspend_always(); }69 auto return_void() { return std::suspend_always(); }70 void unhandled_exception() { __builtin_unreachable(); }71 auto yield_value(T v) {72 current_value = v;73 return std::suspend_always();74 }75 };76 77 generator(std::coroutine_handle<promise_type> h) : hdl(h) { hdl.resume(); }78 ~generator() { hdl.destroy(); }79 80 generator<T>& operator++() { hdl.resume(); return *this; } // resume the coroutine81 T operator*() const { return hdl.promise().current_value; }82 83 private:84 std::coroutine_handle<promise_type> hdl;85 };86 87We can then use this ``generator`` class to print the Fibonacci sequence:88 89.. code-block:: c++90 91 #include "generator.hpp"92 #include <iostream>93 94 generator<int> fibonacci() {95 co_yield 0;96 int prev = 0;97 co_yield 1;98 int current = 1;99 while (true) {100 int next = current + prev;101 co_yield next;102 prev = current;103 current = next;104 }105 }106 107 template<typename T>108 void print10Elements(generator<T>& gen) {109 for (unsigned i = 0; i < 10; ++i) {110 std::cerr << *gen << "\n";111 ++gen;112 }113 }114 115 int main() {116 std::cerr << "Fibonacci sequence - here we go\n";117 generator<int> fib = fibonacci();118 for (unsigned i = 0; i < 5; ++i) {119 ++fib;120 }121 print10Elements(fib);122 }123 124To compile this code, use ``clang++ --std=c++23 generator-example.cpp -g``.125 126Breakpoints inside the generators127---------------------------------128 129We can set breakpoints inside coroutines just as we set them in regular130functions. For VS Code, that means clicking next the line number in the editor.131In the ``lldb`` CLI or in ``gdb``, you can use ``b`` to set a breakpoint.132 133Inspecting variables in a coroutine134-----------------------------------135 136If you hit a breakpoint inside the ``fibonacci`` function, you should be able137to inspect all local variables (``prev``, ``current``, ``next``) just like in138a regular function.139 140.. image:: ./coro-generator-variables.png141 142Note the two additional variables ``__promise`` and ``__coro_frame``. Those143show the internal state of the coroutine. They are not relevant for our144generator example but will be relevant for asynchronous programming described145in the next section.146 147Stepping out of a coroutine148---------------------------149 150When single-stepping, you will notice that the debugger will leave the151``fibonacci`` function as soon as you hit a ``co_yield`` statement. You might152find yourself inside some standard library code. After stepping out of the153library code, you will be back in the ``main`` function.154 155Stepping into a coroutine156-------------------------157 158If you stop at ``++fib`` and try to step into the generator, you will first159find yourself inside ``operator++``. Stepping into the ``handle.resume()`` will160not work by default.161 162This is because lldb does not step into functions from the standard library by163default. To make this work, you first need to run ``settings set164target.process.thread.step-avoid-regexp ""``. You can do so from the "Debug165Console" towards the bottom of the screen. With that setting change, you can166step through ``coroutine_handle::resume`` and into your generator.167 168You might find yourself at the top of the coroutine at first, instead of at169your previous suspension point. In that case, single-step and you will arrive170at the previously suspended ``co_yield`` statement.171 172 173Inspecting a suspended coroutine174--------------------------------175 176The ``print10Elements`` function receives an opaque ``generator`` type. Let's177assume we are suspended at the ``++gen;`` line and want to inspect the178generator and its internal state.179 180To do so, we can simply look into the ``gen.hdl`` variable. LLDB comes with a181pretty printer for ``std::coroutine_handle`` which will show us the internal182state of the coroutine. For GDB, the pretty printer is provided by a script,183see :ref:`gdb-script` for setup instructions.184 185.. image:: ./coro-generator-suspended.png186 187We can see two function pointers ``resume`` and ``destroy``. These pointers188point to the resume / destroy functions. By inspecting those function pointers,189we can see that our ``generator`` is actually backed by our ``fibonacci``190coroutine. When using VS Code + lldb-dap, you can Cmd+Click on the function191address (``0x555...`` in the screenshot) to jump directly to the function192definition backing your coroutine handle.193 194Next, we see the ``promise``. In our case, this reveals the current value of195our generator.196 197The ``coro_frame`` member represents the internal state of the coroutine. It198contains our internal coroutine state ``prev``, ``current``, ``next``.199Furthermore, it contains many internal, compiler-specific members, which are200named based on their type. These represent temporary values which the compiler201decided to spill across suspension points, but which were not declared in our202original source code and hence have no proper user-provided name.203 204Tracking the exact suspension point205-----------------------------------206 207Among the compiler-generated members, the ``__coro_index`` is particularly208important. This member identifies the suspension point at which the coroutine209is currently suspended. However, it is non-trivial to map this number back to210a source code location.211 212For GDB, the provided :ref:`gdb-script` already takes care of this and provides213the exact line number of the suspension point as part of the coroutine handle's214summary string. Unfortunately, LLDB's pretty-printer does not support this, yet.215Furthermore, those labels are only emitted starting with clang 21.0.216 217When debugging with LLDB or when using older clang versions, we will have to use218a different approach.219 220For simple cases, you might still be able to guess the suspension point correctly.221Alternatively, you might also want to modify your coroutine library to store222the line number of the current suspension point in the promise:223 224.. code-block:: c++225 226 // For all promise_types we need a new `_coro_return_address` variable:227 class promise_type {228 ...229 void* _coro_return_address = nullptr;230 };231 232 // For all the awaiter types we need:233 class awaiter {234 ...235 template <typename Promise>236 __attribute__((noinline)) auto await_suspend(std::coroutine_handle<Promise> handle) {237 ...238 handle.promise()._coro_return_address = __builtin_return_address(0);239 }240 };241 242This stores the return address of ``await_suspend`` within the promise.243Thereby, we can read it back from the promise of a suspended coroutine and map244it to an exact source code location. For a complete example, see the ``task``245type used below for asynchronous programming.246 247Alternatively, we can modify the C++ code to store the line number in the248promise type. We can use ``std::source_location`` to get the line number of249the await and store it inside the ``promise_type``. In the debugger, we can250then read the line number from the promise of the suspended coroutine.251 252.. code-block:: c++253 254 // For all the awaiter types we need:255 class awaiter {256 ...257 template <typename Promise>258 void await_suspend(std::coroutine_handle<Promise> handle,259 std::source_location sl = std::source_location::current()) {260 ...261 handle.promise().line_number = sl.line();262 }263 };264 265The downside of both approaches is that they come at the price of additional266runtime cost. In particular, the second approach increases binary size, since it267requires additional ``std::source_location`` objects, and those source268locations are not stripped by split-dwarf. Whether the first approach is worth269the additional runtime cost is a trade-off you need to make yourself.270 271Async stack traces272==================273 274Besides generators, the second common use case for coroutines in C++ is275asynchronous programming, usually involving libraries such as stdexec, folly,276cppcoro, boost::asio, or similar libraries. Some of those libraries already277provide custom debugging support, so in addition to this guide, you might want278to check out their documentation.279 280When using coroutines for asynchronous programming, your library usually281provides you with some ``task`` type. This type usually looks similar to this:282 283.. code-block:: c++284 285 // async-task-library.hpp286 #include <coroutine>287 #include <utility>288 289 struct task {290 struct promise_type {291 task get_return_object() { return std::coroutine_handle<promise_type>::from_promise(*this); }292 auto initial_suspend() { return std::suspend_always{}; }293 294 void unhandled_exception() noexcept {}295 296 auto final_suspend() noexcept {297 struct FinalSuspend {298 std::coroutine_handle<> continuation;299 auto await_ready() noexcept { return false; }300 auto await_suspend(std::coroutine_handle<> handle) noexcept {301 return continuation;302 }303 void await_resume() noexcept {}304 };305 return FinalSuspend{continuation};306 }307 308 void return_value(int res) { result = res; }309 310 std::coroutine_handle<> continuation = std::noop_coroutine();311 int result = 0;312 #ifndef NDEBUG313 void* _coro_suspension_point_addr = nullptr;314 #endif315 };316 317 task(std::coroutine_handle<promise_type> handle) : handle(handle) {}318 ~task() {319 if (handle)320 handle.destroy();321 }322 323 struct Awaiter {324 std::coroutine_handle<promise_type> handle;325 auto await_ready() { return false; }326 327 template <typename P>328 #ifndef NDEBUG329 __attribute__((noinline))330 #endif331 auto await_suspend(std::coroutine_handle<P> continuation) {332 handle.promise().continuation = continuation;333 #ifndef NDEBUG334 continuation.promise()._coro_suspension_point_addr = __builtin_return_address(0);335 #endif336 return handle;337 }338 int await_resume() {339 return handle.promise().result;340 }341 };342 343 auto operator co_await() {344 return Awaiter{handle};345 }346 347 int syncStart() {348 handle.resume();349 return handle.promise().result;350 }351 352 private:353 std::coroutine_handle<promise_type> handle;354 };355 356Note how the ``task::promise_type`` has a member variable357``std::coroutine_handle<> continuation``. This is the handle of the coroutine358that will be resumed when the current coroutine is finished executing (see359``final_suspend``). In a sense, this is the "return address" of the coroutine.360It is set inside ``operator co_await`` when another coroutine calls our361generator and awaits for the next value to be produced.362 363The result value is returned via the ``int result`` member. It is written in364``return_value`` and read by ``Awaiter::await_resume``. Usually, the result365type of a task is a template argument. For simplicity's sake, we hard-coded the366``int`` type in this example.367 368Stack traces of in-flight coroutines369------------------------------------370 371Let's assume you have the following program and set a breakpoint inside the372``write_output`` function. There are multiple call paths through which this373function could have been reached. How can we find out said call path?374 375.. code-block:: c++376 377 #include <iostream>378 #include <string_view>379 #include "async-task-library.hpp"380 381 static task write_output(std::string_view contents) {382 std::cout << contents << "\n";383 co_return contents.size();384 }385 386 static task greet() {387 int bytes_written = 0;388 bytes_written += co_await write_output("Hello");389 bytes_written += co_await write_output("World");390 co_return bytes_written;391 }392 393 int main() {394 int bytes_written = greet().syncStart();395 std::cout << "Bytes written: " << bytes_written << "\n";396 return 0;397 }398 399To do so, let's break inside ``write_output``. We can understand our call-stack400by looking into the special ``__promise`` variable. This artificial variable is401generated by the compiler and points to the ``promise_type`` instance402corresponding to the currently in-flight coroutine. In this case, the403``__promise`` variable contains the ``continuation`` which points to our404caller. That caller again contains a ``promise`` with a ``continuation`` which405points to our caller's caller.406 407.. image:: ./coro-async-task-continuations.png408 409We can figure out the involved coroutine functions and their current suspension410points as discussed above in the "Inspecting a suspended coroutine" section.411 412When using LLDB's CLI, the command ``p --ptr-depth 4 __promise`` might also be413useful to automatically dereference all the pointers up to the given depth.414 415To get a flat representation of that call stack, we can use a debugger script,416such as the one shown in the :ref:`lldb-script` section. With that417script, we can run ``coro bt`` to get the following stack trace:418 419.. code-block::420 421 (lldb) coro bt422 frame #0: write_output(std::basic_string_view<char, std::char_traits<char>>) at /home/avogelsgesang/Documents/corotest/async-task-example.cpp:6:16423 [async] frame #1: greet() at /home/avogelsgesang/Documents/corotest/async-task-example.cpp:12:20424 [async] frame #2: std::__n4861::coroutine_handle<std::__n4861::noop_coroutine_promise>::__frame::__dummy_resume_destroy() at /usr/include/c++/14/coroutine:298, suspension point unknown425 frame #3: std::__n4861::coroutine_handle<task::promise_type>::resume() const at /usr/include/c++/14/coroutine:242:29426 frame #4: task::syncStart() at /home/avogelsgesang/Documents/corotest/async-task-library.hpp:78:14427 frame #5: main at /home/avogelsgesang/Documents/corotest/async-task-example.cpp:18:11428 frame #6: __libc_start_call_main at sysdeps/nptl/libc_start_call_main.h:58:16429 frame #7: __libc_start_main_impl at csu/libc-start.c:360:3430 frame #8: _start at :4294967295431 432Note how the frames #1 and #2 are async frames.433 434The ``coro bt`` command already includes logic to identify the exact suspension435point of each frame based on the ``_coro_suspension_point_addr`` stored inside436the promise.437 438Stack traces of suspended coroutines439------------------------------------440 441Usually, while a coroutine is waiting for, e.g., an in-flight network request,442the suspended ``coroutine_handle`` is stored within the work queues inside the443IO scheduler. As soon as we get hold of the coroutine handle, we can backtrace444it by using ``coro bt <coro_handle>`` where ``<coro_handle>`` is an expression445evaluating to the coroutine handle of the suspended coroutine.446 447Keeping track of all existing coroutines448----------------------------------------449 450Usually, we should be able to get hold of all currently suspended coroutines by451inspecting the worker queues of the IO scheduler. In cases where this is not452possible, we can use the following approach to keep track of all currently453suspended coroutines.454 455One such solution is to store the list of in-flight coroutines in a collection:456 457.. code-block:: c++458 459 inline std::unordered_set<std::coroutine_handle<void>> inflight_coroutines;460 inline std::mutex inflight_coroutines_mutex;461 462 class promise_type {463 public:464 promise_type() {465 std::unique_lock<std::mutex> lock(inflight_coroutines_mutex);466 inflight_coroutines.insert(std::coroutine_handle<promise_type>::from_promise(*this));467 }468 ~promise_type() {469 std::unique_lock<std::mutex> lock(inflight_coroutines_mutex);470 inflight_coroutines.erase(std::coroutine_handle<promise_type>::from_promise(*this));471 }472 };473 474With this in place, it is possible to inspect ``inflight_coroutines`` from the475debugger and rely on LLDB's ``std::coroutine_handle`` pretty-printer to476inspect the coroutines.477 478This technique will track *all* coroutines, also the ones which are currently479awaiting another coroutine, though. To identify just the "roots" of our480in-flight coroutines, we can use the ``coro in-flight inflight_coroutines``481command provided by the :ref:`lldb-script`.482 483Please note that the above is expensive from a runtime performance perspective,484and requires locking to prevent data races. As such, it is not recommended to485use this approach in production code.486 487Known issues & workarounds for older LLDB versions488==================================================489 490LLDB before 21.0 did not yet show the ``__coro_frame`` inside491``coroutine_handle``. To inspect the coroutine frame, you had to use the492approach described in the :ref:`devirtualization` section.493 494LLDB before 18.0 hid the ``__promise`` and ``__coro_frame``495variables by default. The variables are still present, but they need to be496explicitly added to the "watch" pane in VS Code or requested via497``print __promise`` and ``print __coro_frame`` from the debugger console.498 499LLDB before 16.0 did not yet provide a pretty-printer for500``std::coroutine_handle``. To inspect the coroutine handle, you had to manually501use the approach described in the :ref:`devirtualization`502section.503 504Toolchain Implementation Details505================================506 507This section covers the ABI as well as additional compiler-specific behavior.508The ABI is followed by all compilers, on all major systems, including Windows,509Linux, and macOS. Different compilers emit different debug information, though.510 511Ramp, resume and destroy functions512----------------------------------513 514Every coroutine is split into three parts:515 516* The ramp function allocates the coroutine frame and initializes it, usually517 copying over all variables into the coroutine frame518* The resume function continues the coroutine from its previous suspension point519* The destroy function destroys and deallocates the coroutine frame520* The cleanup function destroys the coroutine frame but does not deallocate it.521 It is used when the coroutine's allocation was elided thanks to522 `Heap Allocation Elision (HALO) <https://www.open-std.org/JTC1/SC22/WG21/docs/papers/2018/p0981r0.html>`_523 524The ramp function is called by the coroutine's caller, and available under the525original function name used in the C++ source code. The resume function is526called via ``std::coroutine_handle::resume``. The destroy function is called527via ``std::coroutine_handle::destroy``.528 529Information between the three functions is passed via the coroutine frame, a530compiler-synthesized struct that contains all necessary internal state. The531resume function knows where to resume execution by reading the suspension point532index from the coroutine frame. Similarly, the destroy function relies on the533suspension point index to know which variables are currently in scope and need534to be destructed.535 536Usually, the destroy function calls all destructors and deallocates the537coroutine frame. When a coroutine frame was elided thanks to HALO, only the538destructors need to be called, but the coroutine frame must not be deallocated.539In those cases, the cleanup function is used instead of the destroy function.540 541For coroutines allocated with ``[[clang::coro_await_elidable]]``, clang also542generates a ``.noalloc`` variant of the ramp function, which does not allocate543the coroutine frame by itself, but instead expects the caller to allocate the544coroutine frame and pass it to the ramp function.545 546When trying to intercept all creations of new coroutines in the debugger, you547hence might have to set breakpoints in the ramp function and its ``.noalloc``548variant.549 550Artificial ``__promise`` and ``__coro_frame`` variables551-------------------------------------------------------552 553Inside all coroutine functions, clang / LLVM synthesize a ``__promise`` and554``__coro_frame`` variable. These variables are used to store the coroutine's555state. When inside the coroutine function, those can be used to directly556inspect the promise and the coroutine frame of the own function.557 558The ABI of a coroutine559----------------------560 561A ``std::coroutine_handle`` essentially only holds a pointer to a coroutine562frame. It resembles the following struct:563 564.. code-block:: c++565 566 template<typename promise_type>567 struct coroutine_handle {568 void* __coroutine_frame = nullptr;569 };570 571The structure of coroutine frames is defined as572 573.. code-block:: c++574 575 struct my_coroutine_frame {576 void (*__resume)(coroutine_frame*); // function pointer to the `resume` function577 void (*__destroy)(coroutine_frame*); // function pointer to the `destroy` function578 promise_type promise; // the corresponding `promise_type`579 ... // Internal coroutine state580 }581 582For each coroutine, the compiler synthesizes a different coroutine type,583storing all necessary internal state. The actual coroutine type is type-erased584behind the ``std::coroutine_handle``.585 586However, all coroutine frames always contain the ``resume`` and ``destroy``587functions as their first two members. As such, we can read the function588pointers from the coroutine frame and then obtain the function's name from its589address.590 591The promise is guaranteed to be at a 16-byte offset from the coroutine frame.592If we have a coroutine handle at address 0x416eb0, we can hence reinterpret-cast593the promise as follows:594 595.. code-block:: text596 597 print (task::promise_type)*(0x416eb0+16)598 599Implementation in clang / LLVM600------------------------------601 602The C++ Coroutines feature in the Clang compiler is implemented in two parts of603the compiler. Semantic analysis is performed in Clang, and coroutine604construction and optimization take place in the LLVM middle-end.605 606For each coroutine function, the frontend generates a single corresponding607LLVM-IR function. This function uses special ``llvm.coro.suspend`` intrinsics608to mark the suspension points of the coroutine. The middle end first optimizes609this function and applies, e.g., constant propagation across the whole,610non-split coroutine.611 612CoroSplit then splits the function into ramp, resume and destroy functions.613This pass also moves stack-local variables which are alive across suspension614points into the coroutine frame. Most of the heavy lifting to preserve debugging615information is done in this pass. This pass needs to rewrite all variable616locations to point into the coroutine frame.617 618Afterwards, a couple of additional optimizations are applied before code619gets emitted, but none of them are really interesting regarding debugging620information.621 622For more details on the IR representation of coroutines and the relevant623optimization passes, see `Coroutines in LLVM <https://llvm.org/docs/Coroutines.html>`_.624 625Emitting debug information inside ``CoroSplit`` forces us to generate626insufficient debugging information. Usually, the compiler generates debug627information in the frontend, as debug information is highly language specific.628However, this is not possible for coroutine frames because the frames are629constructed in the LLVM middle-end.630 631To mitigate this problem, the LLVM middle end attempts to generate some debug632information, which is unfortunately incomplete, since much of the633language-specific information is missing in the middle end.634 635.. _devirtualization:636 637Devirtualization of coroutine handles638-------------------------------------639 640Figuring out the promise type and the coroutine frame type of a coroutine641handle requires inspecting the ``resume`` and ``destroy`` function pointers.642There are two possible approaches to do so:643 6441. clang always names the type by appending ``.coro_frame_ty`` to the645 linkage name of the ramp function.6462. Both clang and GCC add the function-local ``__promise`` and647 ``__coro_frame`` variables to the resume and destroy functions.648 We can lookup their types and thereby get the types of promise649 and coroutine frame.650 651In general, the second approach is preferred, as it is more portable.652 653To do so, we look up the types in the destroy function and not the resume function654because the resume function pointer will be set to a ``nullptr`` as soon as a655coroutine reaches its final suspension point. If we used the resume function,656devirtualization would hence fail for all coroutines that have reached their final657suspension point.658 659LLDB comes with devirtualization support out of the box, as part of the660pretty-printer for ``std::coroutine_handle``. For GDB, a similar pretty-printer661is provided by the :ref:`gdb-script`.662 663Interpreting the coroutine frame in optimized builds664----------------------------------------------------665 666The ``__coro_frame`` variable usually refers to the coroutine frame of an667*in-flight* coroutine. This means the coroutine is currently executing.668However, the compiler only guarantees the coroutine frame to be in a consistent669state while the coroutine is suspended. As such, the variables inside the670``__coro_frame`` variable might be outdated, particularly when optimizations671are enabled.672 673Furthermore, when optimizations are enabled, the compiler will layout the674coroutine frame more aggressively. Unused values are optimized out, and the675state will usually contain only the minimal information required to reconstruct676the coroutine's state.677 678clang / LLVM usually use variables like ``__int_32_0`` to represent this679optimized storage. Those values usually do not directly correspond to variables680in the source code.681 682When compiling the program683 684.. code-block:: c++685 686 static task coro_task(int v) {687 int a = v;688 co_await some_other_task();689 a++; // __int_32_0 is 43 here690 std::cout << a << "\n";691 a++; // __int_32_0 is still 43 here692 std::cout << a << "\n";693 a++; // __int_32_0 is still 43 here!694 std::cout << a << "\n";695 co_await some_other_task();696 a++; // __int_32_0 is still 43 here!!697 std::cout << a << "\n";698 a++; // Why is __int_32_0 still 43 here?699 std::cout << a << "\n";700 }701 702clang creates a single entry ``__int_32_0`` in the coroutine state.703 704Intuitively, one might assume that ``__int_32_0`` represents the value of the705local variable ``a``. However, inspecting ``__int_32_0`` in the debugger while706single-stepping will reveal that the value of ``__int_32_0`` stays constant,707despite ``a`` being frequently incremented.708 709While this might be surprising, this is a result of the optimizer recognizing710that it can eliminate most of the load/store operations.711The above code is optimized to the equivalent of:712 713.. code-block:: c++714 715 static task coro_task(int v) {716 store v into __int_32_0 in the frame717 co_await await_counter{};718 a = load __int_32_0719 std::cout << a+1 << "\n";720 std::cout << a+2 << "\n";721 std::cout << a+3 << "\n";722 co_await await_counter{};723 a = load __int_32_0724 std::cout << a+4 << "\n";725 std::cout << a+5 << "\n";726 }727 728It should now be obvious why the value of ``__int_32_0`` remains unchanged729throughout the function. It is important to recognize that ``__int_32_0`` does730not directly correspond to ``a``, but is instead a variable generated to assist731the compiler in code generation. The variables in an optimized coroutine frame732should not be thought of as directly representing the variables in the C++733source.734 735 736Mapping suspension point indices to source code locations737---------------------------------------------------------738 739To aid in mapping a ``__coro_index`` back to a source code location, clang 21.0740and newer emit special, compiler-generated labels for the suspension points.741 742In gdb, we can use the ``info line`` command to get the source code location of743the suspension point.744 745::746 747 (gdb) info line -function coro_task -label __coro_resume_2748 Line 45 of "llvm-example.cpp" starts at address 0x1b1b <_ZL9coro_taski.resume+555> and ends at 0x1b46 <_ZL9coro_taski.resume+598>.749 Line 45 of "llvm-example.cpp" starts at address 0x201b <_ZL9coro_taski.destroy+555> and ends at 0x2046 <_ZL9coro_taski.destroy+598>.750 Line 45 of "llvm-example.cpp" starts at address 0x253b <_ZL9coro_taski.cleanup+555> and ends at 0x2566 <_ZL9coro_taski.cleanup+598>.751 752LLDB does not support looking up labels, yet. For this reason, LLDB's pretty-printer753does not show the exact line number of the suspension point.754 755 756Resources757=========758 759.. _lldb-script:760 761LLDB Debugger Script762--------------------763 764The following script provides the ``coro bt`` and ``coro in-flight`` commands765discussed above. It can be loaded into LLDB using ``command script import766lldb_coro_debugging.py``. To load this by default, add this command to your767``~/.lldbinit`` file.768 769Note that this script requires LLDB 21.0 or newer.770 771.. code-block:: python772 773 # lldb_coro_debugging.py774 import lldb775 from lldb.plugins.parsed_cmd import ParsedCommand776 777 def _get_first_var_path(v, paths):778 """779 Tries multiple variable paths via `GetValueForExpressionPath`780 and returns the first one that succeeds, or None if none succeed.781 """782 for path in paths:783 var = v.GetValueForExpressionPath(path)784 if var.error.Success():785 return var786 return None787 788 789 def _print_async_bt(coro_hdl, result, *, curr_idx, start, limit, continuation_paths, prefix=""):790 """791 Prints a backtrace for an async coroutine stack starting from `coro_hdl`,792 using the given `continuation_paths` to get the next coroutine from the promise.793 """794 target = coro_hdl.GetTarget()795 while curr_idx < limit and coro_hdl is not None and coro_hdl.error.Success():796 # Print the stack frame, if in range797 if curr_idx >= start:798 # Figure out the function name799 destroy_func_var = coro_hdl.GetValueForExpressionPath(".destroy")800 destroy_addr = target.ResolveLoadAddress(destroy_func_var.GetValueAsAddress())801 func_name = destroy_addr.function.name802 # Figure out the line entry to show803 suspension_addr_var = coro_hdl.GetValueForExpressionPath(".promise._coro_suspension_point_addr")804 if suspension_addr_var.error.Success():805 line_entry = target.ResolveLoadAddress(suspension_addr_var.GetValueAsAddress()).line_entry806 print(f"{prefix} frame #{curr_idx}: {func_name} at {line_entry}", file=result)807 else:808 # We don't know the exact line, print the suspension point ID, so we at least show809 # the id of the current suspension point810 suspension_point_var = coro_hdl.GetValueForExpressionPath(".coro_frame.__coro_index")811 if suspension_point_var.error.Success():812 suspension_point = suspension_point_var.GetValueAsUnsigned()813 else:814 suspension_point = "unknown"815 line_entry = destroy_addr.line_entry816 print(f"{prefix} frame #{curr_idx}: {func_name} at {line_entry}, suspension point {suspension_point}", file=result)817 # Move to the next stack frame818 curr_idx += 1819 promise_var = coro_hdl.GetChildMemberWithName("promise")820 coro_hdl = _get_first_var_path(promise_var, continuation_paths)821 return curr_idx822 823 def _print_combined_bt(frame, result, *, unfiltered, curr_idx, start, limit, continuation_paths):824 """825 Prints a backtrace starting from `frame`, interleaving async coroutine frames826 with regular frames.827 """828 while curr_idx < limit and frame.IsValid():829 if curr_idx >= start and (unfiltered or not frame.IsHidden()):830 print(f"frame #{curr_idx}: {frame.name} at {frame.line_entry}", file=result)831 curr_idx += 1832 coro_var = _get_first_var_path(frame.GetValueForVariablePath("__promise"), continuation_paths)833 if coro_var:834 curr_idx = _print_async_bt(coro_var, result,835 curr_idx=curr_idx, start=start, limit=limit,836 continuation_paths=continuation_paths, prefix="[async]")837 frame = frame.parent838 839 840 class CoroBacktraceCommand(ParsedCommand):841 def get_short_help(self):842 return "Create a backtrace for C++-20 coroutines"843 844 def get_flags(self):845 return lldb.eCommandRequiresFrame | lldb.eCommandProcessMustBePaused846 847 def setup_command_definition(self):848 ov_parser = self.get_parser()849 ov_parser.add_option(850 "e",851 "continuation-expr",852 help = (853 "Semi-colon-separated list of expressions evaluated against the promise object"854 "to get the next coroutine (e.g. `.continuation;.coro_parent`)"855 ),856 value_type = lldb.eArgTypeNone,857 dest = "continuation_expr_arg",858 default = ".continuation",859 )860 ov_parser.add_option(861 "c",862 "count",863 help = "How many frames to display (0 for all)",864 value_type = lldb.eArgTypeCount,865 dest = "count_arg",866 default = 20,867 )868 ov_parser.add_option(869 "s",870 "start",871 help = "Frame in which to start the backtrace",872 value_type = lldb.eArgTypeIndex,873 dest = "frame_index_arg",874 default = 0,875 )876 ov_parser.add_option(877 "u",878 "unfiltered",879 help = "Do not filter out frames according to installed frame recognizers",880 value_type = lldb.eArgTypeBoolean,881 dest = "unfiltered_arg",882 default = False,883 )884 ov_parser.add_argument_set([885 ov_parser.make_argument_element(886 lldb.eArgTypeExpression,887 repeat="optional"888 )889 ])890 891 def __call__(self, debugger, args_array, exe_ctx, result):892 ov_parser = self.get_parser()893 continuation_paths = ov_parser.continuation_expr_arg.split(";")894 count = ov_parser.count_arg895 if count == 0:896 count = 99999897 frame_index = ov_parser.frame_index_arg898 unfiltered = ov_parser.unfiltered_arg899 900 frame = exe_ctx.GetFrame()901 if not frame.IsValid():902 result.SetError("invalid frame")903 return904 905 if len(args_array) > 1:906 result.SetError("At most one expression expected")907 return908 elif len(args_array) == 1:909 expr = args_array.GetItemAtIndex(0).GetStringValue(9999)910 coro_hdl = frame.EvaluateExpression(expr)911 if not coro_hdl.error.Success():912 result.AppendMessage(913 f'error: expression failed {expr} => {coro_hdl.error}'914 )915 result.SetError(f"Expression `{expr}` failed to evaluate")916 return917 _print_async_bt(coro_hdl, result,918 curr_idx = 0, start = frame_index, limit = frame_index + count,919 continuation_paths = continuation_paths)920 else:921 _print_combined_bt(frame, result, unfiltered=unfiltered,922 curr_idx = 0, start = frame_index, limit = frame_index + count,923 continuation_paths = continuation_paths)924 925 926 class CoroInflightCommand(ParsedCommand):927 def get_short_help(self):928 return "Identify all in-flight coroutines"929 930 def get_flags(self):931 return lldb.eCommandRequiresTarget | lldb.eCommandProcessMustBePaused932 933 def setup_command_definition(self):934 ov_parser = self.get_parser()935 ov_parser.add_option(936 "e",937 "continuation-expr",938 help = (939 "Semi-colon-separated list of expressions evaluated against the promise object"940 "to get the next coroutine (e.g. `.continuation;.coro_parent`)"941 ),942 value_type = lldb.eArgTypeNone,943 dest = "continuation_expr_arg",944 default = ".continuation",945 )946 ov_parser.add_option(947 "c",948 "count",949 help = "How many frames to display (0 for all)",950 value_type = lldb.eArgTypeCount,951 dest = "count_arg",952 default = 5,953 )954 ov_parser.add_argument_set([955 ov_parser.make_argument_element(956 lldb.eArgTypeExpression,957 repeat="plus"958 )959 ])960 961 def __call__(self, debugger, args_array, exe_ctx, result):962 ov_parser = self.get_parser()963 continuation_paths = ov_parser.continuation_expr_arg.split(";")964 count = ov_parser.count_arg965 966 # Collect all coroutine_handles from the provided containers967 all_coros = []968 for entry in args_array:969 expr = entry.GetStringValue(9999)970 if exe_ctx.frame.IsValid():971 coro_container = exe_ctx.frame.EvaluateExpression(expr)972 else:973 coro_container = exe_ctx.target.EvaluateExpression(expr)974 if not coro_container.error.Success():975 result.AppendMessage(976 f'error: expression failed {expr} => {coro_container.error}'977 )978 result.SetError(f"Expression `{expr}` failed to evaluate")979 return980 for entry in coro_container.children:981 if "coroutine_handle" not in entry.GetType().name:982 result.SetError(f"Found entry of type {entry.GetType().name} in {expr},"983 " expected a coroutine handle")984 return985 all_coros.append(entry)986 987 # Remove all coroutines that are currently waiting for other coroutines to finish988 coro_roots = {c.GetChildMemberWithName("coro_frame").GetValueAsAddress(): c for c in all_coros}989 for coro_hdl in all_coros:990 parent_coro = _get_first_var_path(coro_hdl.GetChildMemberWithName("promise"), continuation_paths)991 parent_addr = parent_coro.GetChildMemberWithName("coro_frame").GetValueAsAddress()992 if parent_addr in coro_roots:993 del coro_roots[parent_addr]994 995 # Print all remaining coroutines996 for addr, root_hdl in coro_roots.items():997 print(f"coroutine root 0x{addr:x}", file=result)998 _print_async_bt(root_hdl, result,999 curr_idx=0, start=0, limit=count,1000 continuation_paths=continuation_paths, prefix=" ")1001 1002 1003 def __lldb_init_module(debugger, internal_dict):1004 debugger.HandleCommand("command container add -h 'Debugging utilities for C++20 coroutines' coro")1005 debugger.HandleCommand(f"command script add -o -p -c {__name__}.CoroBacktraceCommand coro bt")1006 debugger.HandleCommand(f"command script add -o -p -c {__name__}.CoroInflightCommand coro in-flight")1007 print("Coro debugging utilities installed. Use `help coro` to see available commands.")1008 1009 if __name__ == '__main__':1010 print("This script should be loaded from LLDB using `command script import <filename>`")1011 1012.. _gdb-script:1013 1014GDB Debugger Script1015-------------------1016 1017The following script provides:1018 1019* a pretty-printer for coroutine handles1020* a frame filter to add coroutine frames to the built-in ``bt`` command1021* the ``get_coro_frame`` and ``get_coro_promise`` functions to be used in1022 expressions, e.g. ``p get_coro_promise(fib.coro_hdl)->current_state``1023 1024It can be loaded into GDB using ``source gdb_coro_debugging.py``.1025To load this by default, add this command to your ``~/.gdbinit`` file.1026 1027.. code-block:: python1028 1029 # gdb_coro_debugging.py1030 import gdb1031 from gdb.FrameDecorator import FrameDecorator1032 1033 import typing1034 import re1035 1036 def _load_pointer_at(addr: int):1037 return gdb.Value(addr).reinterpret_cast(gdb.lookup_type('void').pointer().pointer()).dereference()1038 1039 """1040 Devirtualized coroutine frame.1041 1042 Devirtualizes the promise and frame pointer types by inspecting1043 the destroy function.1044 1045 Implements `to_string` and `children` to be used by `gdb.printing.PrettyPrinter`.1046 Base class for `CoroutineHandlePrinter`.1047 """1048 class DevirtualizedCoroFrame:1049 def __init__(self, frame_ptr_raw: int, val: gdb.Value | None = None):1050 self.val = val1051 self.frame_ptr_raw = frame_ptr_raw1052 1053 # Get the resume and destroy pointers.1054 if frame_ptr_raw == 0:1055 self.resume_ptr = None1056 self.destroy_ptr = None1057 self.promise_ptr = None1058 self.frame_ptr = gdb.Value(frame_ptr_raw).reinterpret_cast(gdb.lookup_type("void").pointer())1059 return1060 1061 # Get the resume and destroy pointers.1062 self.resume_ptr = _load_pointer_at(frame_ptr_raw)1063 self.destroy_ptr = _load_pointer_at(frame_ptr_raw + 8)1064 1065 # Devirtualize the promise and frame pointer types.1066 frame_type = gdb.lookup_type("void")1067 promise_type = gdb.lookup_type("void")1068 self.destroy_func = gdb.block_for_pc(int(self.destroy_ptr))1069 if self.destroy_func is not None:1070 frame_var = gdb.lookup_symbol("__coro_frame", self.destroy_func, gdb.SYMBOL_VAR_DOMAIN)[0]1071 if frame_var is not None:1072 frame_type = frame_var.type1073 promise_var = gdb.lookup_symbol("__promise", self.destroy_func, gdb.SYMBOL_VAR_DOMAIN)[0]1074 if promise_var is not None:1075 promise_type = promise_var.type.strip_typedefs()1076 1077 # If the type has a template argument, prefer it over the devirtualized type.1078 if self.val is not None:1079 promise_type_template_arg = self.val.type.template_argument(0)1080 if promise_type_template_arg is not None and promise_type_template_arg.code != gdb.TYPE_CODE_VOID:1081 promise_type = promise_type_template_arg1082 1083 self.promise_ptr = gdb.Value(frame_ptr_raw + 16).reinterpret_cast(promise_type.pointer())1084 self.frame_ptr = gdb.Value(frame_ptr_raw).reinterpret_cast(frame_type.pointer())1085 1086 # Try to get the suspension point index and look up the exact line entry.1087 self.suspension_point_index = int(self.frame_ptr.dereference()["__coro_index"]) if frame_type.code == gdb.TYPE_CODE_STRUCT else None1088 self.resume_func = gdb.block_for_pc(int(self.resume_ptr))1089 self.resume_label = None1090 if self.resume_func is not None and self.suspension_point_index is not None:1091 label_name = f"__coro_resume_{self.suspension_point_index}"1092 self.resume_label = gdb.lookup_symbol(label_name, self.resume_func, gdb.SYMBOL_LABEL_DOMAIN)[0]1093 1094 def get_function_name(self):1095 if self.destroy_func is None:1096 return None1097 name = self.destroy_func.function.name1098 # Strip the "clone" suffix if it exists.1099 if "() [clone " in name:1100 name = name[:name.index("() [clone ")]1101 return name1102 1103 def to_string(self):1104 result = "coro(" + str(self.frame_ptr_raw) + ")"1105 if self.destroy_func is not None:1106 result += ": " + self.get_function_name()1107 if self.resume_label is not None:1108 result += ", line " + str(self.resume_label.line)1109 if self.suspension_point_index is not None:1110 result += ", suspension point " + str(self.suspension_point_index)1111 return result1112 1113 def children(self):1114 if self.resume_ptr is None:1115 return [1116 ("coro_frame", self.frame_ptr),1117 ]1118 else:1119 return [1120 ("resume", self.resume_ptr),1121 ("destroy", self.destroy_ptr),1122 ("promise", self.promise_ptr),1123 ("coro_frame", self.frame_ptr)1124 ]1125 1126 1127 # Works for both libc++ and libstdc++.1128 libcxx_corohdl_regex = re.compile('^std::__[A-Za-z0-9]+::coroutine_handle<.+>$|^std::coroutine_handle<.+>(( )?&)?$')1129 1130 def _extract_coro_frame_ptr_from_handle(val: gdb.Value):1131 if libcxx_corohdl_regex.match(val.type.strip_typedefs().name) is None:1132 raise ValueError("Expected a std::coroutine_handle, got %s" % val.type.strip_typedefs().name)1133 1134 # We expect the coroutine handle to have a single field, which is the frame pointer.1135 # This heuristic works for both libc++ and libstdc++.1136 fields = val.type.fields()1137 if len(fields) != 1:1138 raise ValueError("Expected 1 field, got %d" % len(fields))1139 return int(val[fields[0]])1140 1141 1142 """1143 Pretty printer for `std::coroutine_handle<T>`1144 1145 Works for both libc++ and libstdc++.1146 1147 It prints the coroutine handle as a struct with the following fields:1148 - resume: the resume function pointer1149 - destroy: the destroy function pointer1150 - promise: the promise pointer1151 - coro_frame: the coroutine frame pointer1152 1153 Most of the functionality is implemented in `DevirtualizedCoroFrame`.1154 """1155 class CoroutineHandlePrinter(DevirtualizedCoroFrame):1156 def __init__(self, val : gdb.Value):1157 frame_ptr_raw = _extract_coro_frame_ptr_from_handle(val)1158 super(CoroutineHandlePrinter, self).__init__(frame_ptr_raw, val)1159 1160 1161 def build_pretty_printer():1162 pp = gdb.printing.RegexpCollectionPrettyPrinter("coroutine")1163 pp.add_printer('std::coroutine_handle', libcxx_corohdl_regex, CoroutineHandlePrinter)1164 return pp1165 1166 gdb.printing.register_pretty_printer(1167 gdb.current_objfile(),1168 build_pretty_printer())1169 1170 1171 """1172 Get the coroutine frame pointer from a coroutine handle.1173 1174 Usage:1175 ```1176 p *get_coro_frame(coroutine_hdl)1177 ```1178 """1179 class GetCoroFrame(gdb.Function):1180 def __init__(self):1181 super(GetCoroFrame, self).__init__("get_coro_frame")1182 1183 def invoke(self, coroutine_hdl_raw):1184 return CoroutineHandlePrinter(coroutine_hdl_raw).frame_ptr1185 1186 GetCoroFrame()1187 1188 1189 """1190 Get the coroutine frame pointer from a coroutine handle.1191 1192 Usage:1193 ```1194 p *get_coro_promise(coroutine_hdl)1195 ```1196 """1197 class GetCoroFrame(gdb.Function):1198 def __init__(self):1199 super(GetCoroFrame, self).__init__("get_coro_promise")1200 1201 def invoke(self, coroutine_hdl_raw):1202 return CoroutineHandlePrinter(coroutine_hdl_raw).promise_ptr1203 1204 GetCoroFrame()1205 1206 1207 """1208 Decorator for coroutine frames.1209 1210 Used by `CoroutineFrameFilter` to add the coroutine frames to the built-in `bt` command.1211 """1212 class CoroutineFrameDecorator(FrameDecorator):1213 def __init__(self, coro_frame: DevirtualizedCoroFrame, inferior_frame: gdb.Frame):1214 super(CoroutineFrameDecorator, self).__init__(inferior_frame)1215 self.coro_frame = coro_frame1216 1217 def function(self):1218 func_name = self.coro_frame.get_function_name()1219 if func_name is not None:1220 return "[async] " + func_name1221 return "[async] coroutine (coro_frame=" + str(self.coro_frame.frame_ptr_raw) + ")"1222 1223 def address(self):1224 return None1225 1226 def filename(self):1227 if self.coro_frame.destroy_func is not None:1228 return self.coro_frame.destroy_func.function.symtab.filename1229 return None1230 1231 def line(self):1232 if self.coro_frame.resume_label is not None:1233 return self.coro_frame.resume_label.line1234 return None1235 1236 def frame_args(self):1237 return []1238 1239 def frame_locals(self):1240 return []1241 1242 1243 def _get_continuation(promise: gdb.Value) -> DevirtualizedCoroFrame | None:1244 try:1245 # TODO: adjust this according for your coroutine framework1246 return DevirtualizedCoroFrame(_extract_coro_frame_ptr_from_handle(promise["continuation"]))1247 except Exception as e:1248 return None1249 1250 1251 def _create_coroutine_frames(coro_frame: DevirtualizedCoroFrame, inferior_frame: gdb.Frame):1252 while coro_frame is not None:1253 yield CoroutineFrameDecorator(coro_frame, inferior_frame)1254 coro_frame = _get_continuation(coro_frame.promise_ptr)1255 1256 1257 """1258 Frame filter to add coroutine frames to the built-in `bt` command.1259 """1260 class CppCoroutineFrameFilter():1261 def __init__(self):1262 self.name = "CppCoroutineFrameFilter"1263 self.priority = 501264 self.enabled = True1265 # Register this frame filter with the global frame_filters dictionary.1266 gdb.frame_filters[self.name] = self1267 1268 def filter(self, frame_iter: typing.Iterable[gdb.FrameDecorator]):1269 for frame in frame_iter:1270 yield frame1271 inferior_frame = frame.inferior_frame()1272 try:1273 promise_ptr = inferior_frame.read_var("__promise")1274 except Exception:1275 continue1276 parent_coro = _get_continuation(promise_ptr)1277 if parent_coro is not None:1278 yield from _create_coroutine_frames(parent_coro, inferior_frame)1279 1280 CppCoroutineFrameFilter()1281 1282Further Reading1283---------------1284 1285The authors of the Folly libraries wrote a blog post series on how they debug coroutines:1286 1287* `Async stack traces in folly: Introduction <https://developers.facebook.com/blog/post/2021/09/16/async-stack-traces-folly-Introduction/>`_1288* `Async stack traces in folly: Synchronous and asynchronous stack traces <https://developers.facebook.com/blog/post/2021/09/23/async-stack-traces-folly-synchronous-asynchronous-stack-traces/>`_1289* `Async stack traces in folly: Forming an async stack from individual frames <https://developers.facebook.com/blog/post/2021/09/30/async-stack-traces-folly-forming-async-stack-individual-frames/>`_1290* `Async Stack Traces for C++ Coroutines in Folly: Walking the async stack <https://developers.facebook.com/blog/post/2021/10/14/async-stack-traces-c-plus-plus-coroutines-folly-walking-async-stack/>`_1291* `Async stack traces in folly: Improving debugging in the developer lifecycle <https://developers.facebook.com/blog/post/2021/10/21/async-stack-traces-folly-improving-debugging-developer-lifecycle/>`_1292 1293Besides some topics also covered here (stack traces from the debugger), Folly's blog post series also covers1294additional topics, such as capturing async stack traces in performance profiles via eBPF filters1295and printing async stack traces on crashes.1296