2256 lines · plain
1=====================================2Coroutines in LLVM3=====================================4 5.. contents::6 :local:7 :depth: 38 9.. warning::10 Compatibility across LLVM releases is not guaranteed.11 12Introduction13============14 15.. _coroutine handle:16 17LLVM coroutines are functions that have one or more `suspend points`_.18When a suspend point is reached, the execution of a coroutine is suspended and19control is returned back to its caller. A suspended coroutine can be resumed20to continue execution from the last suspend point or it can be destroyed.21 22In the following example, we call function `f` (which may or may not be a23coroutine itself) that returns a handle to a suspended coroutine24(**coroutine handle**) that is used by `main` to resume the coroutine twice and25then destroy it:26 27.. code-block:: llvm28 29 define i32 @main() {30 entry:31 %hdl = call ptr @f(i32 4)32 call void @llvm.coro.resume(ptr %hdl)33 call void @llvm.coro.resume(ptr %hdl)34 call void @llvm.coro.destroy(ptr %hdl)35 ret i32 036 }37 38.. _coroutine frame:39 40In addition to the function stack frame, which exists when a coroutine is41executing, there is an additional region of storage that contains objects that42keep the coroutine state when a coroutine is suspended. This region of storage43is called the **coroutine frame**. It is created when a coroutine is called44and destroyed when a coroutine either runs to completion or is destroyed45while suspended.46 47LLVM currently supports two styles of coroutine lowering. These styles48support substantially different sets of features, have substantially49different ABIs, and expect substantially different patterns of frontend50code generation. However, the styles also have a great deal in common.51 52In all cases, an LLVM coroutine is initially represented as an ordinary LLVM53function that has calls to `coroutine intrinsics`_ defining the structure of54the coroutine. The coroutine function is then, in the most general case,55rewritten by the coroutine lowering passes to become the "ramp function",56the initial entrypoint of the coroutine, which executes until a suspend point57is first reached. The remainder of the original coroutine function is split58out into some number of "resume functions". Any state which must persist59across suspensions is stored in the coroutine frame. The resume functions60must somehow be able to handle either a "normal" resumption, which continues61the normal execution of the coroutine, or an "abnormal" resumption, which62must unwind the coroutine without attempting to suspend it.63 64Switched-Resume Lowering65------------------------66 67In LLVM's standard switched-resume lowering, signaled by the use of68`llvm.coro.id`, the coroutine frame is stored as part of a "coroutine69object" which represents a handle to a particular invocation of the70coroutine. All coroutine objects support a common ABI allowing certain71features to be used without knowing anything about the coroutine's72implementation:73 74- A coroutine object can be queried to see if it has reached completion75 with `llvm.coro.done`.76 77- A coroutine object can be resumed normally if it has not already reached78 completion with `llvm.coro.resume`.79 80- A coroutine object can be destroyed, invalidating the coroutine object,81 with `llvm.coro.destroy`. This must be done separately even if the82 coroutine has reached completion normally.83 84- "Promise" storage, which is known to have a certain size and alignment,85 can be projected out of the coroutine object with `llvm.coro.promise`.86 The coroutine implementation must have been compiled to define a promise87 of the same size and alignment.88 89In general, interacting with a coroutine object in any of these ways while90it is running has undefined behavior.91 92The coroutine function is split into three functions, representing three93different ways that control can enter the coroutine:94 951. the ramp function that is initially invoked, which takes arbitrary96 arguments and returns a pointer to the coroutine object;97 982. a coroutine resume function that is invoked when the coroutine is resumed,99 which takes a pointer to the coroutine object and returns `void`;100 1013. a coroutine destroy function that is invoked when the coroutine is102 destroyed, which takes a pointer to the coroutine object and returns103 `void`.104 105Because the resume and destroy functions are shared across all suspend106points, suspend points must store the index of the active suspend in107the coroutine object, and the resume/destroy functions must switch over108that index to get back to the correct point. Hence the name of this109lowering.110 111Pointers to the resume and destroy functions are stored in the coroutine112object at known offsets which are fixed for all coroutines. A completed113coroutine is represented with a null resume function.114 115There is a somewhat complex protocol of intrinsics for allocating and116deallocating the coroutine object. It is complex in order to allow the117allocation to be elided due to inlining. This protocol is discussed118in further detail below.119 120The frontend may generate code to call the coroutine function directly;121this will become a call to the ramp function and will return a pointer122to the coroutine object. The frontend should always resume or destroy123the coroutine using the corresponding intrinsics.124 125Returned-Continuation Lowering126------------------------------127 128In returned-continuation lowering, signaled by the use of129`llvm.coro.id.retcon` or `llvm.coro.id.retcon.once`, some aspects of130the ABI must be handled more explicitly by the frontend.131 132In this lowering, every suspend point takes a list of "yielded values"133which are returned back to the caller along with a function pointer,134called the continuation function. The coroutine is resumed by simply135calling this continuation function pointer. The original coroutine136is divided into the ramp function and then an arbitrary number of137these continuation functions, one for each suspend point.138 139LLVM actually supports two closely-related returned-continuation140lowerings:141 142- In normal returned-continuation lowering, the coroutine may suspend143 itself multiple times. This means that a continuation function144 itself returns another continuation pointer, as well as a list of145 yielded values.146 147 The coroutine indicates that it has run to completion by returning148 a null continuation pointer. Any yielded values will be `undef` and149 should be ignored.150 151- In yield-once returned-continuation lowering, the coroutine must152 suspend itself exactly once (or throw an exception). The ramp153 function returns a continuation function pointer and yielded154 values, the continuation function may optionally return ordinary155 results when the coroutine has run to completion.156 157The coroutine frame is maintained in a fixed-size buffer that is158passed to the `coro.id` intrinsic, which guarantees a certain size159and alignment statically. The same buffer must be passed to the160continuation function(s). The coroutine will allocate memory if the161buffer is insufficient, in which case it will need to store at162least that pointer in the buffer; therefore, the buffer must always163be at least pointer-sized. How the coroutine uses the buffer may164vary between suspend points.165 166In addition to the buffer pointer, continuation functions take an167argument indicating whether the coroutine is being resumed normally168(zero) or abnormally (non-zero).169 170LLVM is currently ineffective at statically eliminating allocations171after fully inlining returned-continuation coroutines into a caller.172This may be acceptable if LLVM's coroutine support is primarily being173used for low-level lowering and inlining is expected to be applied174earlier in the pipeline.175 176Async Lowering177--------------178 179In async-continuation lowering, signaled by the use of `llvm.coro.id.async`,180handling of control-flow must be handled explicitly by the frontend.181 182In this lowering, a coroutine is assumed to take the current `async context` as183one of its arguments (the argument position is determined by184`llvm.coro.id.async`). It is used to marshal arguments and return values of the185coroutine. Therefore, an async coroutine returns `void`.186 187.. code-block:: llvm188 189 define swiftcc void @async_coroutine(ptr %async.ctxt, ptr, ptr) {190 }191 192Values live across a suspend point need to be stored in the coroutine frame to193be available in the continuation function. This frame is stored as a tail to the194`async context`.195 196Every suspend point takes a `context projection function` argument which197describes how-to obtain the continuations `async context` and every suspend198point has an associated `resume function` denoted by the199`llvm.coro.async.resume` intrinsic. The coroutine is resumed by calling this200`resume function` passing the `async context` as the one of its arguments201argument. The `resume function` can restore its (the caller's) `async context`202by applying a `context projection function` that is provided by the frontend as203a parameter to the `llvm.coro.suspend.async` intrinsic.204 205.. code-block:: c206 207 // For example:208 struct async_context {209 struct async_context *caller_context;210 ...211 }212 213 char *context_projection_function(struct async_context *callee_ctxt) {214 return callee_ctxt->caller_context;215 }216 217.. code-block:: llvm218 219 %resume_func_ptr = call ptr @llvm.coro.async.resume()220 call {ptr, ptr, ptr} (ptr, ptr, ...) @llvm.coro.suspend.async(221 ptr %resume_func_ptr,222 ptr %context_projection_function223 224The frontend should provide an `async function pointer` struct associated with225each async coroutine by `llvm.coro.id.async`'s argument. The initial size and226alignment of the `async context` must be provided as arguments to the227`llvm.coro.id.async` intrinsic. Lowering will update the size entry with the228coroutine frame requirements. The frontend is responsible for allocating the229memory for the `async context` but can use the `async function pointer` struct230to obtain the required size.231 232.. code-block:: c233 234 struct async_function_pointer {235 uint32_t relative_function_pointer_to_async_impl;236 uint32_t context_size;237 }238 239Lowering will split an async coroutine into a ramp function and one resume240function per suspend point.241 242How control-flow is passed between caller, suspension point, and back to243resume function is left up to the frontend.244 245The suspend point takes a function and its arguments. The function is intended246to model the transfer to the callee function. It will be tail called by247lowering and therefore must have the same signature and calling convention as248the async coroutine.249 250.. code-block:: llvm251 252 call {ptr, ptr, ptr} (ptr, ptr, ...) @llvm.coro.suspend.async(253 ptr %resume_func_ptr,254 ptr %context_projection_function,255 ptr %suspend_function,256 ptr %arg1, ptr %arg2, i8 %arg3)257 258Coroutines by Example259=====================260 261The examples below are all of switched-resume coroutines.262 263Coroutine Representation264------------------------265 266Let's look at an example of an LLVM coroutine with the behavior sketched267by the following pseudo-code.268 269.. code-block:: c++270 271 void *f(int n) {272 for(;;) {273 print(n++);274 <suspend> // returns a coroutine handle on first suspend275 }276 }277 278This coroutine calls some function `print` with value `n` as an argument and279suspends execution. Every time this coroutine resumes, it calls `print` again with an argument one bigger than the last time. This coroutine never completes by itself and must be destroyed explicitly. If we use this coroutine with280a `main` shown in the previous section. It will call `print` with values 4, 5281and 6 after which the coroutine will be destroyed.282 283The LLVM IR for this coroutine looks like this:284 285.. code-block:: llvm286 287 define ptr @f(i32 %n) presplitcoroutine {288 entry:289 %id = call token @llvm.coro.id(i32 0, ptr null, ptr null, ptr null)290 %size = call i32 @llvm.coro.size.i32()291 %alloc = call ptr @malloc(i32 %size)292 %hdl = call noalias ptr @llvm.coro.begin(token %id, ptr %alloc)293 br label %loop294 loop:295 %n.val = phi i32 [ %n, %entry ], [ %inc, %loop ]296 %inc = add nsw i32 %n.val, 1297 call void @print(i32 %n.val)298 %0 = call i8 @llvm.coro.suspend(token none, i1 false)299 switch i8 %0, label %suspend [i8 0, label %loop300 i8 1, label %cleanup]301 cleanup:302 %mem = call ptr @llvm.coro.free(token %id, ptr %hdl)303 call void @free(ptr %mem)304 br label %suspend305 suspend:306 call void @llvm.coro.end(ptr %hdl, i1 false, token none)307 ret ptr %hdl308 }309 310The `entry` block establishes the coroutine frame. The `coro.size`_ intrinsic is311lowered to a constant representing the size required for the coroutine frame.312The `coro.begin`_ intrinsic initializes the coroutine frame and returns the313coroutine handle. The second parameter of `coro.begin` is given a block of memory314to be used if the coroutine frame needs to be allocated dynamically.315 316The `coro.id`_ intrinsic serves as coroutine identity useful in cases when the317`coro.begin`_ intrinsic gets duplicated by optimization passes such as318jump-threading.319 320The `cleanup` block destroys the coroutine frame. The `coro.free`_ intrinsic,321given the coroutine handle, returns a pointer of the memory block to be freed or322`null` if the coroutine frame was not allocated dynamically. The `cleanup`323block is entered when coroutine runs to completion by itself or destroyed via324a call to the `coro.destroy`_ intrinsic.325 326The `suspend` block contains code to be executed when coroutine runs to327completion or suspended. The `coro.end`_ intrinsic marks the point where328a coroutine needs to return control back to the caller if it is not an initial329invocation of the coroutine.330 331The `loop` blocks represents the body of the coroutine. The `coro.suspend`_332intrinsic in combination with the following switch indicates what happens to333control flow when a coroutine is suspended (default case), resumed (case 0) or334destroyed (case 1).335 336Coroutine Transformation337------------------------338 339One of the steps of coroutine lowering is building the coroutine frame. The340def-use chains are analyzed to determine which objects need to be kept alive across341suspend points. In the coroutine shown in the previous section, use of virtual register342`%inc` is separated from the definition by a suspend point, therefore, it343cannot reside on the stack frame since the latter goes away once the coroutine344is suspended and control is returned back to the caller. An i32 slot is345allocated in the coroutine frame and `%inc` is spilled and reloaded from that346slot as needed.347 348We also store addresses of the resume and destroy functions so that the349`coro.resume` and `coro.destroy` intrinsics can resume and destroy the coroutine350when its identity cannot be determined statically at compile time. For our351example, the coroutine frame will be:352 353.. code-block:: llvm354 355 %f.frame = type { ptr, ptr, i32 }356 357After resume and destroy parts are outlined, function `f` will contain only the358code responsible for creation and initialization of the coroutine frame and359execution of the coroutine until a suspend point is reached:360 361.. code-block:: llvm362 363 define ptr @f(i32 %n) {364 entry:365 %id = call token @llvm.coro.id(i32 0, ptr null, ptr null, ptr null)366 %alloc = call noalias ptr @malloc(i32 24)367 %frame = call noalias ptr @llvm.coro.begin(token %id, ptr %alloc)368 %1 = getelementptr %f.frame, ptr %frame, i32 0, i32 0369 store ptr @f.resume, ptr %1370 %2 = getelementptr %f.frame, ptr %frame, i32 0, i32 1371 store ptr @f.destroy, ptr %2372 373 %inc = add nsw i32 %n, 1374 %inc.spill.addr = getelementptr inbounds %f.Frame, ptr %FramePtr, i32 0, i32 2375 store i32 %inc, ptr %inc.spill.addr376 call void @print(i32 %n)377 378 ret ptr %frame379 }380 381Outlined resume part of the coroutine will reside in function `f.resume`:382 383.. code-block:: llvm384 385 define internal fastcc void @f.resume(ptr %frame.ptr.resume) {386 entry:387 %inc.spill.addr = getelementptr %f.frame, ptr %frame.ptr.resume, i64 0, i32 2388 %inc.spill = load i32, ptr %inc.spill.addr, align 4389 %inc = add i32 %inc.spill, 1390 store i32 %inc, ptr %inc.spill.addr, align 4391 tail call void @print(i32 %inc)392 ret void393 }394 395Whereas function `f.destroy` will contain the cleanup code for the coroutine:396 397.. code-block:: llvm398 399 define internal fastcc void @f.destroy(ptr %frame.ptr.destroy) {400 entry:401 tail call void @free(ptr %frame.ptr.destroy)402 ret void403 }404 405Avoiding Heap Allocations406-------------------------407 408A particular coroutine usage pattern, which is illustrated by the `main`409function in the overview section, where a coroutine is created, manipulated and410destroyed by the same calling function, is common for coroutines implementing411RAII idiom and is suitable for allocation elision optimization which avoid412dynamic allocation by storing the coroutine frame as a static `alloca` in its413caller.414 415In the entry block, we will call `coro.alloc`_ intrinsic that will return `true`416when dynamic allocation is required, and `false` if dynamic allocation is417elided.418 419.. code-block:: llvm420 421 entry:422 %id = call token @llvm.coro.id(i32 0, ptr null, ptr null, ptr null)423 %need.dyn.alloc = call i1 @llvm.coro.alloc(token %id)424 br i1 %need.dyn.alloc, label %dyn.alloc, label %coro.begin425 dyn.alloc:426 %size = call i32 @llvm.coro.size.i32()427 %alloc = call ptr @CustomAlloc(i32 %size)428 br label %coro.begin429 coro.begin:430 %phi = phi ptr [ null, %entry ], [ %alloc, %dyn.alloc ]431 %hdl = call noalias ptr @llvm.coro.begin(token %id, ptr %phi)432 433In the cleanup block, we will make freeing the coroutine frame conditional on434`coro.free`_ intrinsic. If allocation is elided, `coro.free`_ returns `null`435thus skipping the deallocation code:436 437.. code-block:: llvm438 439 cleanup:440 %mem = call ptr @llvm.coro.free(token %id, ptr %hdl)441 %need.dyn.free = icmp ne ptr %mem, null442 br i1 %need.dyn.free, label %dyn.free, label %if.end443 dyn.free:444 call void @CustomFree(ptr %mem)445 br label %if.end446 if.end:447 ...448 449With allocations and deallocations represented as described as above, after450coroutine heap allocation elision optimization, the resulting main will be:451 452.. code-block:: llvm453 454 define i32 @main() {455 entry:456 call void @print(i32 4)457 call void @print(i32 5)458 call void @print(i32 6)459 ret i32 0460 }461 462Multiple Suspend Points463-----------------------464 465Let's consider the coroutine that has more than one suspend point:466 467.. code-block:: c++468 469 void *f(int n) {470 for(;;) {471 print(n++);472 <suspend>473 print(-n);474 <suspend>475 }476 }477 478Matching LLVM code would look like (with the rest of the code remaining the same479as the code in the previous section):480 481.. code-block:: llvm482 483 loop:484 %n.addr = phi i32 [ %n, %entry ], [ %inc, %loop.resume ]485 call void @print(i32 %n.addr) #4486 %2 = call i8 @llvm.coro.suspend(token none, i1 false)487 switch i8 %2, label %suspend [i8 0, label %loop.resume488 i8 1, label %cleanup]489 loop.resume:490 %inc = add nsw i32 %n.addr, 1491 %sub = xor i32 %n.addr, -1492 call void @print(i32 %sub)493 %3 = call i8 @llvm.coro.suspend(token none, i1 false)494 switch i8 %3, label %suspend [i8 0, label %loop495 i8 1, label %cleanup]496 497In this case, the coroutine frame would include a suspend index that will498indicate at which suspend point the coroutine needs to resume.499 500.. code-block:: llvm501 502 %f.frame = type { ptr, ptr, i32, i32 }503 504The resume function will use an index to jump to an appropriate basic block and will look505as follows:506 507.. code-block:: llvm508 509 define internal fastcc void @f.Resume(ptr %FramePtr) {510 entry.Resume:511 %index.addr = getelementptr inbounds %f.Frame, ptr %FramePtr, i64 0, i32 2512 %index = load i8, ptr %index.addr, align 1513 %switch = icmp eq i8 %index, 0514 %n.addr = getelementptr inbounds %f.Frame, ptr %FramePtr, i64 0, i32 3515 %n = load i32, ptr %n.addr, align 4516 517 br i1 %switch, label %loop.resume, label %loop518 519 loop.resume:520 %sub = sub nsw i32 0, %n521 call void @print(i32 %sub)522 br label %suspend523 loop:524 %inc = add nsw i32 %n, 1525 store i32 %inc, ptr %n.addr, align 4526 tail call void @print(i32 %inc)527 br label %suspend528 529 suspend:530 %storemerge = phi i8 [ 0, %loop ], [ 1, %loop.resume ]531 store i8 %storemerge, ptr %index.addr, align 1532 ret void533 }534 535If different cleanup code needs to be executed for different suspend points,536a similar switch will be in the `f.destroy` function.537 538.. note ::539 540 Using suspend index in a coroutine state and having a switch in `f.resume` and541 `f.destroy` is one of the possible implementation strategies. We explored542 another option where a distinct `f.resume1`, `f.resume2`, etc. are created for543 every suspend point, and instead of storing an index, the resume and destroy544 function pointers are updated at every suspend. Early testing showed that the545 current approach is easier on the optimizer than the latter so it is a546 lowering strategy implemented at the moment.547 548Distinct Save and Suspend549-------------------------550 551In the previous example, setting a resume index (or some other state change that552needs to happen to prepare a coroutine for resumption) happens at the same time as553a suspension of a coroutine. However, in certain cases, it is necessary to control554when coroutine is prepared for resumption and when it is suspended.555 556In the following example, a coroutine represents some activity that is driven557by completions of asynchronous operations `async_op1` and `async_op2` which get558a coroutine handle as a parameter and resume the coroutine once async559operation is finished.560 561.. code-block:: text562 563 void g() {564 for (;;)565 if (cond()) {566 async_op1(<coroutine-handle>); // will resume once async_op1 completes567 <suspend>568 do_one();569 }570 else {571 async_op2(<coroutine-handle>); // will resume once async_op2 completes572 <suspend>573 do_two();574 }575 }576 }577 578In this case, coroutine should be ready for resumption prior to a call to579`async_op1` and `async_op2`. The `coro.save`_ intrinsic is used to indicate a580point when coroutine should be ready for resumption (namely, when a resume index581should be stored in the coroutine frame, so that it can be resumed at the582correct resume point):583 584.. code-block:: llvm585 586 if.true:587 %save1 = call token @llvm.coro.save(ptr %hdl)588 call void @async_op1(ptr %hdl)589 %suspend1 = call i1 @llvm.coro.suspend(token %save1, i1 false)590 switch i8 %suspend1, label %suspend [i8 0, label %resume1591 i8 1, label %cleanup]592 if.false:593 %save2 = call token @llvm.coro.save(ptr %hdl)594 call void @async_op2(ptr %hdl)595 %suspend2 = call i1 @llvm.coro.suspend(token %save2, i1 false)596 switch i8 %suspend2, label %suspend [i8 0, label %resume2597 i8 1, label %cleanup]598 599.. _coroutine promise:600 601Coroutine Promise602-----------------603 604A coroutine author or a frontend may designate a distinguished `alloca` that can605be used to communicate with the coroutine. This distinguished alloca is called606**coroutine promise** and is provided as the second parameter to the607`coro.id`_ intrinsic.608 609The following coroutine designates a 32-bit integer `promise` and uses it to610store the current value produced by a coroutine.611 612.. code-block:: llvm613 614 define ptr @f(i32 %n) {615 entry:616 %promise = alloca i32617 %id = call token @llvm.coro.id(i32 0, ptr %promise, ptr null, ptr null)618 %need.dyn.alloc = call i1 @llvm.coro.alloc(token %id)619 br i1 %need.dyn.alloc, label %dyn.alloc, label %coro.begin620 dyn.alloc:621 %size = call i32 @llvm.coro.size.i32()622 %alloc = call ptr @malloc(i32 %size)623 br label %coro.begin624 coro.begin:625 %phi = phi ptr [ null, %entry ], [ %alloc, %dyn.alloc ]626 %hdl = call noalias ptr @llvm.coro.begin(token %id, ptr %phi)627 br label %loop628 loop:629 %n.val = phi i32 [ %n, %coro.begin ], [ %inc, %loop ]630 %inc = add nsw i32 %n.val, 1631 store i32 %n.val, ptr %promise632 %0 = call i8 @llvm.coro.suspend(token none, i1 false)633 switch i8 %0, label %suspend [i8 0, label %loop634 i8 1, label %cleanup]635 cleanup:636 %mem = call ptr @llvm.coro.free(token %id, ptr %hdl)637 call void @free(ptr %mem)638 br label %suspend639 suspend:640 call void @llvm.coro.end(ptr %hdl, i1 false, token none)641 ret ptr %hdl642 }643 644A coroutine consumer can rely on the `coro.promise`_ intrinsic to access the645coroutine promise.646 647.. code-block:: llvm648 649 define i32 @main() {650 entry:651 %hdl = call ptr @f(i32 4)652 %promise.addr = call ptr @llvm.coro.promise(ptr %hdl, i32 4, i1 false)653 %val0 = load i32, ptr %promise.addr654 call void @print(i32 %val0)655 call void @llvm.coro.resume(ptr %hdl)656 %val1 = load i32, ptr %promise.addr657 call void @print(i32 %val1)658 call void @llvm.coro.resume(ptr %hdl)659 %val2 = load i32, ptr %promise.addr660 call void @print(i32 %val2)661 call void @llvm.coro.destroy(ptr %hdl)662 ret i32 0663 }664 665After example in this section is compiled, result of the compilation will be:666 667.. code-block:: llvm668 669 define i32 @main() {670 entry:671 tail call void @print(i32 4)672 tail call void @print(i32 5)673 tail call void @print(i32 6)674 ret i32 0675 }676 677.. _final:678.. _final suspend:679 680Final Suspend681-------------682 683A coroutine author or a frontend may designate a particular suspend to be final,684by setting the second argument of the `coro.suspend`_ intrinsic to `true`.685Such a suspend point has two properties:686 687* it is possible to check whether a suspended coroutine is at the final suspend688 point via `coro.done`_ intrinsic;689 690* a resumption of a coroutine stopped at the final suspend point leads to691 undefined behavior. The only possible action for a coroutine at a final692 suspend point is destroying it via `coro.destroy`_ intrinsic.693 694From the user perspective, the final suspend point represents an idea of a695coroutine reaching the end. From the compiler perspective, it is an optimization696opportunity for reducing number of resume points (and therefore switch cases) in697the resume function.698 699The following is an example of a function that keeps resuming the coroutine700until the final suspend point is reached after which point the coroutine is701destroyed:702 703.. code-block:: llvm704 705 define i32 @main() {706 entry:707 %hdl = call ptr @f(i32 4)708 br label %while709 while:710 call void @llvm.coro.resume(ptr %hdl)711 %done = call i1 @llvm.coro.done(ptr %hdl)712 br i1 %done, label %end, label %while713 end:714 call void @llvm.coro.destroy(ptr %hdl)715 ret i32 0716 }717 718Usually, final suspend point is a frontend injected suspend point that does not719correspond to any explicitly authored suspend point of the high level language.720For example, for a Python generator that has only one suspend point:721 722.. code-block:: python723 724 def coroutine(n):725 for i in range(n):726 yield i727 728Python frontend would inject two more suspend points, so that the actual code729looks like this:730 731.. code-block:: c732 733 void* coroutine(int n) {734 int current_value;735 <designate current_value to be coroutine promise>736 <SUSPEND> // injected suspend point, so that the coroutine starts suspended737 for (int i = 0; i < n; ++i) {738 current_value = i; <SUSPEND>; // corresponds to "yield i"739 }740 <SUSPEND final=true> // injected final suspend point741 }742 743and Python iterator `__next__` would look like:744 745.. code-block:: c++746 747 int __next__(void* hdl) {748 coro.resume(hdl);749 if (coro.done(hdl)) throw StopIteration();750 return *(int*)coro.promise(hdl, 4, false);751 }752 753Custom ABIs and Plugin Libraries754--------------------------------755 756Plugin libraries can extend coroutine lowering enabling a wide variety of users757to utilize the coroutine transformation passes. An existing coroutine lowering758is extended by:759 760#. defining custom ABIs that inherit from the existing ABIs,761#. give a list of generators for the custom ABIs when constructing the `CoroSplit`_ pass, and762#. use `coro.begin.custom.abi`_ in place of `coro.begin`_ that has an additional parameter for the index of the generator/ABI to be used for the coroutine.763 764A custom ABI overriding the SwitchABI's materialization looks like:765 766.. code-block:: c++767 768 class CustomSwitchABI : public coro::SwitchABI {769 public:770 CustomSwitchABI(Function &F, coro::Shape &S)771 : coro::SwitchABI(F, S, ExtraMaterializable) {}772 };773 774Giving a list of custom ABI generators while constructing the `CoroSplit`775pass looks like:776 777.. code-block:: c++778 779 CoroSplitPass::BaseABITy GenCustomABI = [](Function &F, coro::Shape &S) {780 return std::make_unique<CustomSwitchABI>(F, S);781 };782 783 CGSCCPassManager CGPM;784 CGPM.addPass(CoroSplitPass({GenCustomABI}));785 786The LLVM IR for a coroutine using a Coroutine with a custom ABI looks like:787 788.. code-block:: llvm789 790 define ptr @f(i32 %n) presplitcoroutine_custom_abi {791 entry:792 %id = call token @llvm.coro.id(i32 0, ptr null, ptr null, ptr null)793 %size = call i32 @llvm.coro.size.i32()794 %alloc = call ptr @malloc(i32 %size)795 %hdl = call noalias ptr @llvm.coro.begin.custom.abi(token %id, ptr %alloc, i32 0)796 br label %loop797 loop:798 %n.val = phi i32 [ %n, %entry ], [ %inc, %loop ]799 %inc = add nsw i32 %n.val, 1800 call void @print(i32 %n.val)801 %0 = call i8 @llvm.coro.suspend(token none, i1 false)802 switch i8 %0, label %suspend [i8 0, label %loop803 i8 1, label %cleanup]804 cleanup:805 %mem = call ptr @llvm.coro.free(token %id, ptr %hdl)806 call void @free(ptr %mem)807 br label %suspend808 suspend:809 call void @llvm.coro.end(ptr %hdl, i1 false, token none)810 ret ptr %hdl811 }812 813Parameter Attributes814====================815Some parameter attributes, used to communicate additional information about the result or parameters of a function, require special handling.816 817ByVal818-----819A ByVal parameter on an argument indicates that the pointee should be treated as being passed by value to the function.820Prior to the coroutine transforms loads and stores to/from the pointer are generated where the value is needed.821Consequently, a ByVal argument is treated much like an alloca.822Space is allocated for it on the coroutine frame and the uses of the argument pointer are replaced with a pointer to the coroutine frame.823 824Swift Error825-----------826Clang supports the swiftcall calling convention in many common targets, and a user could call a function that takes a swifterror argument from a C++ coroutine.827The swifterror parameter attribute exists to model and optimize Swift error handling.828A swifterror alloca or parameter can only be loaded, stored, or passed as a swifterror call argument, and a swifterror call argument can only be a direct reference to a swifterror alloca or parameter. 829These rules, not coincidentally, mean that you can always perfectly model the data flow in the alloca, and LLVM CodeGen actually has to do that in order to emit code.830 831For coroutine lowering the default treatment of allocas breaks those rules — splitting will try to replace the alloca with an entry in the coro frame, which can lead to trying to pass that as a swifterror argument.832To pass a swifterror argument in a split function, we need to still have the alloca around, but we also potentially need the coro frame slot, since useful data can (in theory) be stored in the swifterror alloca slot across suspensions in the presplit coroutine.833When split a coroutine it is consequently necessary to keep both the frame slot as well as the alloca itself and then keep them in sync.834 835Intrinsics836==========837 838Coroutine Manipulation Intrinsics839---------------------------------840 841Intrinsics described in this section are used to manipulate an existing842coroutine. They can be used in any function which happen to have a pointer843to a `coroutine frame`_ or a pointer to a `coroutine promise`_.844 845.. _coro.destroy:846 847'llvm.coro.destroy' Intrinsic848^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^849 850Syntax:851"""""""852 853::854 855 declare void @llvm.coro.destroy(ptr <handle>)856 857Overview:858"""""""""859 860The '``llvm.coro.destroy``' intrinsic destroys a suspended861switched-resume coroutine.862 863Arguments:864""""""""""865 866The argument is a coroutine handle to a suspended coroutine.867 868Semantics:869""""""""""870 871When possible, the `coro.destroy` intrinsic is replaced with a direct call to872the coroutine destroy function. Otherwise it is replaced with an indirect call873based on the function pointer for the destroy function stored in the coroutine874frame. Destroying a coroutine that is not suspended leads to undefined behavior.875 876.. _coro.resume:877 878'llvm.coro.resume' Intrinsic879^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^880 881::882 883 declare void @llvm.coro.resume(ptr <handle>)884 885Overview:886"""""""""887 888The '``llvm.coro.resume``' intrinsic resumes a suspended switched-resume coroutine.889 890Arguments:891""""""""""892 893The argument is a handle to a suspended coroutine.894 895Semantics:896""""""""""897 898When possible, the `coro.resume` intrinsic is replaced with a direct call to the899coroutine resume function. Otherwise it is replaced with an indirect call based900on the function pointer for the resume function stored in the coroutine frame.901Resuming a coroutine that is not suspended leads to undefined behavior.902 903.. _coro.done:904 905'llvm.coro.done' Intrinsic906^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^907 908::909 910 declare i1 @llvm.coro.done(ptr <handle>)911 912Overview:913"""""""""914 915The '``llvm.coro.done``' intrinsic checks whether a suspended916switched-resume coroutine is at the final suspend point or not.917 918Arguments:919""""""""""920 921The argument is a handle to a suspended coroutine.922 923Semantics:924""""""""""925 926Using this intrinsic on a coroutine that does not have a `final suspend`_ point927or on a coroutine that is not suspended leads to undefined behavior.928 929.. _coro.promise:930 931'llvm.coro.promise' Intrinsic932^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^933 934::935 936 declare ptr @llvm.coro.promise(ptr <ptr>, i32 <alignment>, i1 <from>)937 938Overview:939"""""""""940 941The '``llvm.coro.promise``' intrinsic obtains a pointer to a942`coroutine promise`_ given a switched-resume coroutine handle and vice versa.943 944Arguments:945""""""""""946 947The first argument is a handle to a coroutine if `from` is false. Otherwise,948it is a pointer to a coroutine promise.949 950The second argument is an alignment requirements of the promise.951If a frontend designated `%promise = alloca i32` as a promise, the alignment952argument to `coro.promise` should be the alignment of `i32` on the target953platform. If a frontend designated `%promise = alloca i32, align 16` as a954promise, the alignment argument should be 16.955This argument only accepts constants.956 957The third argument is a boolean indicating a direction of the transformation.958If `from` is true, the intrinsic returns a coroutine handle given a pointer959to a promise. If `from` is false, the intrinsics return a pointer to a promise960from a coroutine handle. This argument only accepts constants.961 962Semantics:963""""""""""964 965Using this intrinsic on a coroutine that does not have a coroutine promise966leads to undefined behavior. It is possible to read and modify coroutine967promise of the coroutine which is currently executing. The coroutine author and968a coroutine user are responsible for ensuring no data races.969 970Example:971""""""""972 973.. code-block:: llvm974 975 define ptr @f(i32 %n) {976 entry:977 %promise = alloca i32978 ; the second argument to coro.id points to the coroutine promise.979 %id = call token @llvm.coro.id(i32 0, ptr %promise, ptr null, ptr null)980 ...981 %hdl = call noalias ptr @llvm.coro.begin(token %id, ptr %alloc)982 ...983 store i32 42, ptr %promise ; store something into the promise984 ...985 ret ptr %hdl986 }987 988 define i32 @main() {989 entry:990 %hdl = call ptr @f(i32 4) ; starts the coroutine and returns its handle991 %promise.addr = call ptr @llvm.coro.promise(ptr %hdl, i32 4, i1 false)992 %val = load i32, ptr %promise.addr ; load a value from the promise993 call void @print(i32 %val)994 call void @llvm.coro.destroy(ptr %hdl)995 ret i32 0996 }997 998.. _coroutine intrinsics:999 1000Coroutine Structure Intrinsics1001------------------------------1002Intrinsics described in this section are used within a coroutine to describe1003the coroutine structure. They should not be used outside of a coroutine.1004 1005.. _coro.size:1006 1007'llvm.coro.size' Intrinsic1008^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^1009::1010 1011 declare i32 @llvm.coro.size.i32()1012 declare i64 @llvm.coro.size.i64()1013 1014Overview:1015"""""""""1016 1017The '``llvm.coro.size``' intrinsic returns the number of bytes1018required to store a `coroutine frame`_. This is only supported for1019switched-resume coroutines.1020 1021Arguments:1022""""""""""1023 1024None1025 1026Semantics:1027""""""""""1028 1029The `coro.size` intrinsic is lowered to a constant representing the size of1030the coroutine frame.1031 1032.. _coro.align:1033 1034'llvm.coro.align' Intrinsic1035^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^1036::1037 1038 declare i32 @llvm.coro.align.i32()1039 declare i64 @llvm.coro.align.i64()1040 1041Overview:1042"""""""""1043 1044The '``llvm.coro.align``' intrinsic returns the alignment of a `coroutine frame`_.1045This is only supported for switched-resume coroutines.1046 1047Arguments:1048""""""""""1049 1050None1051 1052Semantics:1053""""""""""1054 1055The `coro.align` intrinsic is lowered to a constant representing the alignment of1056the coroutine frame.1057 1058.. _coro.begin:1059 1060'llvm.coro.begin' Intrinsic1061^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^1062::1063 1064 declare ptr @llvm.coro.begin(token <id>, ptr <mem>)1065 1066Overview:1067"""""""""1068 1069The '``llvm.coro.begin``' intrinsic returns an address of the coroutine frame.1070 1071Arguments:1072""""""""""1073 1074The first argument is a token returned by a call to '``llvm.coro.id``'1075identifying the coroutine.1076 1077The second argument is a pointer to a block of memory where coroutine frame1078will be stored if it is allocated dynamically. This pointer is ignored1079for returned-continuation coroutines.1080 1081Semantics:1082""""""""""1083 1084Depending on the alignment requirements of the objects in the coroutine frame1085and/or on the codegen compactness reasons the pointer returned from `coro.begin`1086may be at offset to the `%mem` argument. (This could be beneficial if1087instructions that express relative access to data can be more compactly encoded1088with small positive and negative offsets).1089 1090A frontend should emit exactly one `coro.begin` intrinsic per coroutine.1091 1092.. _coro.begin.custom.abi:1093 1094'llvm.coro.begin.custom.abi' Intrinsic1095^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^1096::1097 1098 declare ptr @llvm.coro.begin.custom.abi(token <id>, ptr <mem>, i32)1099 1100Overview:1101"""""""""1102 1103The '``llvm.coro.begin.custom.abi``' intrinsic is used in place of the1104`coro.begin` intrinsic that has an additional parameter to specify the custom1105ABI for the coroutine. The return is identical to that of the `coro.begin`1106intrinsic.1107 1108Arguments:1109""""""""""1110 1111The first and second arguments are identical to those of the `coro.begin`1112intrinsic.1113 1114The third argument is an i32 index of the generator list given to the1115`CoroSplit` pass specifying the custom ABI generator for this coroutine.1116 1117Semantics:1118""""""""""1119 1120The semantics are identical to those of the `coro.begin` intrinsic.1121 1122.. _coro.free:1123 1124'llvm.coro.free' Intrinsic1125^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^1126::1127 1128 declare ptr @llvm.coro.free(token %id, ptr <frame>)1129 1130Overview:1131"""""""""1132 1133The '``llvm.coro.free``' intrinsic returns a pointer to a block of memory where1134coroutine frame is stored or `null` if this instance of a coroutine did not use1135dynamically allocated memory for its coroutine frame. This intrinsic is not1136supported for returned-continuation coroutines.1137 1138Arguments:1139""""""""""1140 1141The first argument is a token returned by a call to '``llvm.coro.id``'1142identifying the coroutine.1143 1144The second argument is a pointer to the coroutine frame. This should be the same1145pointer that was returned by prior `coro.begin` call.1146 1147Example (custom deallocation function):1148"""""""""""""""""""""""""""""""""""""""1149 1150.. code-block:: llvm1151 1152 cleanup:1153 %mem = call ptr @llvm.coro.free(token %id, ptr %frame)1154 %mem_not_null = icmp ne ptr %mem, null1155 br i1 %mem_not_null, label %if.then, label %if.end1156 if.then:1157 call void @CustomFree(ptr %mem)1158 br label %if.end1159 if.end:1160 ret void1161 1162Example (standard deallocation functions):1163""""""""""""""""""""""""""""""""""""""""""1164 1165.. code-block:: llvm1166 1167 cleanup:1168 %mem = call ptr @llvm.coro.free(token %id, ptr %frame)1169 call void @free(ptr %mem)1170 ret void1171 1172.. _coro.alloc:1173 1174'llvm.coro.alloc' Intrinsic1175^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^1176::1177 1178 declare i1 @llvm.coro.alloc(token <id>)1179 1180Overview:1181"""""""""1182 1183The '``llvm.coro.alloc``' intrinsic returns `true` if dynamic allocation is1184required to obtain memory for the coroutine frame and `false` otherwise.1185This is not supported for returned-continuation coroutines.1186 1187Arguments:1188""""""""""1189 1190The first argument is a token returned by a call to '``llvm.coro.id``'1191identifying the coroutine.1192 1193Semantics:1194""""""""""1195 1196A frontend should emit at most one `coro.alloc` intrinsic per coroutine.1197The intrinsic is used to suppress dynamic allocation of the coroutine frame1198when possible.1199 1200Example:1201""""""""1202 1203.. code-block:: llvm1204 1205 entry:1206 %id = call token @llvm.coro.id(i32 0, ptr null, ptr null, ptr null)1207 %dyn.alloc.required = call i1 @llvm.coro.alloc(token %id)1208 br i1 %dyn.alloc.required, label %coro.alloc, label %coro.begin1209 1210 coro.alloc:1211 %frame.size = call i32 @llvm.coro.size()1212 %alloc = call ptr @MyAlloc(i32 %frame.size)1213 br label %coro.begin1214 1215 coro.begin:1216 %phi = phi ptr [ null, %entry ], [ %alloc, %coro.alloc ]1217 %frame = call ptr @llvm.coro.begin(token %id, ptr %phi)1218 1219.. _coro.noop:1220 1221'llvm.coro.noop' Intrinsic1222^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^1223::1224 1225 declare ptr @llvm.coro.noop()1226 1227Overview:1228"""""""""1229 1230The '``llvm.coro.noop``' intrinsic returns an address of the coroutine frame of1231a coroutine that does nothing when resumed or destroyed.1232 1233Arguments:1234""""""""""1235 1236None1237 1238Semantics:1239""""""""""1240 1241This intrinsic is lowered to refer to a private constant coroutine frame. The1242resume and destroy handlers for this frame are empty functions that do nothing.1243Note that in different translation units llvm.coro.noop may return different pointers.1244 1245.. _coro.frame:1246 1247'llvm.coro.frame' Intrinsic1248^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^1249::1250 1251 declare ptr @llvm.coro.frame()1252 1253Overview:1254"""""""""1255 1256The '``llvm.coro.frame``' intrinsic returns an address of the coroutine frame of1257the enclosing coroutine.1258 1259Arguments:1260""""""""""1261 1262None1263 1264Semantics:1265""""""""""1266 1267This intrinsic is lowered to refer to the `coro.begin`_ instruction. This is1268a frontend convenience intrinsic that makes it easier to refer to the1269coroutine frame.1270 1271.. _coro.id:1272 1273'llvm.coro.id' Intrinsic1274^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^1275::1276 1277 declare token @llvm.coro.id(i32 <align>, ptr <promise>, ptr <coroaddr>,1278 ptr <fnaddrs>)1279 1280Overview:1281"""""""""1282 1283The '``llvm.coro.id``' intrinsic returns a token identifying a1284switched-resume coroutine.1285 1286Arguments:1287""""""""""1288 1289The first argument provides information on the alignment of the memory returned1290by the allocation function and given to `coro.begin` by the first argument. If1291this argument is 0, the memory is assumed to be aligned to 2 * sizeof(ptr).1292This argument only accepts constants.1293 1294The second argument, if not `null`, designates a particular alloca instruction1295to be a `coroutine promise`_.1296 1297The third argument is `null` coming out of the frontend. The CoroEarly pass sets1298this argument to point to the function this coro.id belongs to.1299 1300The fourth argument is `null` before coroutine is split, and later is replaced1301to point to a private global constant array containing function pointers to1302outlined resume and destroy parts of the coroutine.1303 1304 1305Semantics:1306""""""""""1307 1308The purpose of this intrinsic is to tie together `coro.id`, `coro.alloc` and1309`coro.begin` belonging to the same coroutine to prevent optimization passes from1310duplicating any of these instructions unless entire body of the coroutine is1311duplicated.1312 1313A frontend should emit exactly one `coro.id` intrinsic per coroutine.1314 1315A frontend should emit function attribute `presplitcoroutine` for the coroutine.1316 1317.. _coro.id.async:1318 1319'llvm.coro.id.async' Intrinsic1320^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^1321::1322 1323 declare token @llvm.coro.id.async(i32 <context size>, i32 <align>,1324 ptr <context arg>,1325 ptr <async function pointer>)1326 1327Overview:1328"""""""""1329 1330The '``llvm.coro.id.async``' intrinsic returns a token identifying an async coroutine.1331 1332Arguments:1333""""""""""1334 1335The first argument provides the initial size of the `async context` as required1336from the frontend. Lowering will add to this size the size required by the frame1337storage and store that value to the `async function pointer`.1338 1339The second argument, is the alignment guarantee of the memory of the1340`async context`. The frontend guarantees that the memory will be aligned by this1341value.1342 1343The third argument is the `async context` argument in the current coroutine.1344 1345The fourth argument is the address of the `async function pointer` struct.1346Lowering will update the context size requirement in this struct by adding the1347coroutine frame size requirement to the initial size requirement as specified by1348the first argument of this intrinsic.1349 1350 1351Semantics:1352""""""""""1353 1354A frontend should emit exactly one `coro.id.async` intrinsic per coroutine.1355 1356A frontend should emit function attribute `presplitcoroutine` for the coroutine.1357 1358.. _coro.id.retcon:1359 1360'llvm.coro.id.retcon' Intrinsic1361^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^1362::1363 1364 declare token @llvm.coro.id.retcon(i32 <size>, i32 <align>, ptr <buffer>,1365 ptr <continuation prototype>,1366 ptr <alloc>, ptr <dealloc>)1367 1368Overview:1369"""""""""1370 1371The '``llvm.coro.id.retcon``' intrinsic returns a token identifying a1372multiple-suspend returned-continuation coroutine.1373 1374The 'result-type sequence' of the coroutine is defined as follows:1375 1376- if the return type of the coroutine function is ``void``, it is the1377 empty sequence;1378 1379- if the return type of the coroutine function is a ``struct``, it is the1380 element types of that ``struct`` in order;1381 1382- otherwise, it is just the return type of the coroutine function.1383 1384The first element of the result-type sequence must be a pointer type;1385continuation functions will be coerced to this type. The rest of1386the sequence are the 'yield types', and any suspends in the coroutine1387must take arguments of these types.1388 1389Arguments:1390""""""""""1391 1392The first and second arguments are the expected size and alignment of1393the buffer provided as the third argument. They must be constant.1394 1395The fourth argument must be a reference to a global function, called1396the 'continuation prototype function'. The type, calling convention,1397and attributes of any continuation functions will be taken from this1398declaration. The return type of the prototype function must match the1399return type of the current function. The first parameter type must be1400a pointer type. The second parameter type must be an integer type;1401it will be used only as a boolean flag.1402 1403The fifth argument must be a reference to a global function that will1404be used to allocate memory. It may not fail, either by returning null1405or throwing an exception. It must take an integer and return a pointer.1406 1407The sixth argument must be a reference to a global function that will1408be used to deallocate memory. It must take a pointer and return ``void``.1409 1410Semantics:1411""""""""""1412 1413A frontend should emit function attribute `presplitcoroutine` for the coroutine.1414 1415'llvm.coro.id.retcon.once' Intrinsic1416^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^1417::1418 1419 declare token @llvm.coro.id.retcon.once(i32 <size>, i32 <align>, ptr <buffer>,1420 ptr <prototype>,1421 ptr <alloc>, ptr <dealloc>)1422 1423Overview:1424"""""""""1425 1426The '``llvm.coro.id.retcon.once``' intrinsic returns a token identifying a1427unique-suspend returned-continuation coroutine.1428 1429Arguments:1430""""""""""1431 1432As for ``llvm.core.id.retcon``, except that the return type of the1433continuation prototype must represent the normal return type of the continuation1434(instead of matching the coroutine's return type).1435 1436Semantics:1437""""""""""1438 1439A frontend should emit function attribute `presplitcoroutine` for the coroutine.1440 1441.. _coro.end:1442 1443'llvm.coro.end' Intrinsic1444^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^1445::1446 1447 declare void @llvm.coro.end(ptr <handle>, i1 <unwind>, token <result.token>)1448 1449Overview:1450"""""""""1451 1452The '``llvm.coro.end``' marks the point where execution of the resume part of1453the coroutine should end and control should return to the caller.1454 1455 1456Arguments:1457""""""""""1458 1459The first argument should refer to the coroutine handle of the enclosing1460coroutine. A frontend is allowed to supply null as the first parameter, in this1461case `coro-early` pass will replace the null with an appropriate coroutine1462handle value.1463 1464The second argument should be `true` if this coro.end is in the block that is1465part of the unwind sequence leaving the coroutine body due to an exception and1466`false` otherwise.1467 1468Non-trivial (non-none) token argument can only be specified for unique-suspend1469returned-continuation coroutines where it must be a token value produced by1470'``llvm.coro.end.results``' intrinsic.1471 1472Only none token is allowed for coro.end calls in unwind sections1473 1474Semantics:1475""""""""""1476The purpose of this intrinsic is to allow frontends to mark the cleanup and1477other code that is only relevant during the initial invocation of the coroutine1478and should not be present in resume and destroy parts.1479 1480In returned-continuation lowering, ``llvm.coro.end`` fully destroys the1481coroutine frame. If the second argument is `false`, it also returns from1482the coroutine with a null continuation pointer, and the next instruction1483will be unreachable. If the second argument is `true`, it falls through1484so that the following logic can resume unwinding. In a yield-once1485coroutine, reaching a non-unwind ``llvm.coro.end`` without having first1486reached a ``llvm.coro.suspend.retcon`` has undefined behavior.1487 1488The remainder of this section describes the behavior under switched-resume1489lowering.1490 1491This intrinsic is lowered when a coroutine is split into1492the start, resume and destroy parts. In the start part, it is a no-op,1493in resume and destroy parts, it is replaced with `ret void` instruction and1494the rest of the block containing `coro.end` instruction is discarded.1495In landing pads it is replaced with an appropriate instruction to unwind to1496caller. The handling of coro.end differs depending on whether the target is1497using landingpad or WinEH exception model.1498 1499For landingpad based exception model, it is expected that frontend uses the1500`coro.end`_ intrinsic as follows:1501 1502.. code-block:: llvm1503 1504 ehcleanup:1505 call void @llvm.coro.end(ptr null, i1 true, token none)1506 %InRamp = call i1 @llvm.coro.is_in_ramp()1507 br i1 %InRamp, label %cleanup.cont, label %eh.resume1508 1509 cleanup.cont:1510 ; rest of the cleanup1511 1512 eh.resume:1513 %exn = load ptr, ptr %exn.slot, align 81514 %sel = load i32, ptr %ehselector.slot, align 41515 %lpad.val = insertvalue { ptr, i32 } undef, ptr %exn, 01516 %lpad.val29 = insertvalue { ptr, i32 } %lpad.val, i32 %sel, 11517 resume { ptr, i32 } %lpad.val291518 1519The `CoroSpit` pass replaces `coro.is_in_ramp` with ``True`` in the ramp functions,1520thus allowing to proceed to the rest of the cleanup code that is only needed during1521initial invocation of the coroutine. Otherwise, it is replaced with ``False``,1522thus leading to immediate unwind to the caller.1523 1524For Windows Exception handling model, a frontend should attach a funclet bundle1525referring to an enclosing cleanuppad as follows:1526 1527.. code-block:: llvm1528 1529 ehcleanup:1530 %tok = cleanuppad within none []1531 call void @llvm.coro.end(ptr null, i1 true, token none) [ "funclet"(token %tok) ]1532 cleanupret from %tok unwind label %RestOfTheCleanup1533 1534The `CoroSplit` pass, if the funclet bundle is present, will insert1535``cleanupret from %tok unwind to caller`` before1536the `coro.end`_ intrinsic and will remove the rest of the block.1537 1538In the unwind path (when the argument is `true`), `coro.end` will mark the coroutine1539as done, making it undefined behavior to resume the coroutine again and causing 1540`llvm.coro.done` to return `true`. This is not necessary in the normal path because1541the coroutine will already be marked as done by the final suspend.1542 1543The following table summarizes the handling of `coro.end`_ intrinsic.1544 1545+--------------------------+------------------------+---------------------------------+1546| | In Start Function | In Resume/Destroy Functions |1547+--------------------------+------------------------+---------------------------------+1548|unwind=false | nothing |``ret void`` |1549+------------+-------------+------------------------+---------------------------------+1550| | WinEH | mark coroutine as done || ``cleanupret unwind to caller``|1551| | | || mark coroutine done |1552|unwind=true +-------------+------------------------+---------------------------------+1553| | Landingpad | mark coroutine as done | mark coroutine done |1554+------------+-------------+------------------------+---------------------------------+1555 1556.. _coro.end.results:1557 1558'llvm.coro.end.results' Intrinsic1559^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^1560::1561 1562 declare token @llvm.coro.end.results(...)1563 1564Overview:1565"""""""""1566 1567The '``llvm.coro.end.results``' intrinsic captures values to be returned from1568unique-suspend returned-continuation coroutines.1569 1570Arguments:1571""""""""""1572 1573The number of arguments must match the return type of the continuation function:1574 1575- if the return type of the continuation function is ``void`` there must be no1576 arguments1577 1578- if the return type of the continuation function is a ``struct``, the arguments1579 will be of element types of that ``struct`` in order;1580 1581- otherwise, it is just the return value of the continuation function.1582 1583.. code-block:: llvm1584 1585 define {ptr, ptr} @g(ptr %buffer, ptr %ptr, i8 %val) presplitcoroutine {1586 entry:1587 %id = call token @llvm.coro.id.retcon.once(i32 8, i32 8, ptr %buffer,1588 ptr @prototype,1589 ptr @allocate, ptr @deallocate)1590 %hdl = call ptr @llvm.coro.begin(token %id, ptr null)1591 1592 ...1593 1594 cleanup:1595 %tok = call token (...) @llvm.coro.end.results(i8 %val)1596 call void @llvm.coro.end(ptr %hdl, i1 0, token %tok)1597 unreachable1598 1599 ...1600 1601 declare i8 @prototype(ptr, i1 zeroext)1602 1603 1604'llvm.coro.end.async' Intrinsic1605^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^1606::1607 1608 declare void @llvm.coro.end.async(ptr <handle>, i1 <unwind>, ...)1609 1610Overview:1611"""""""""1612 1613The '``llvm.coro.end.async``' marks the point where execution of the resume part1614of the coroutine should end and control should return to the caller. As part of1615its variable tail arguments this instruction allows to specify a function and1616the function's arguments that are to be tail called as the last action before1617returning.1618 1619 1620Arguments:1621""""""""""1622 1623The first argument should refer to the coroutine handle of the enclosing1624coroutine. A frontend is allowed to supply null as the first parameter, in this1625case `coro-early` pass will replace the null with an appropriate coroutine1626handle value.1627 1628The second argument should be `true` if this coro.end is in the block that is1629part of the unwind sequence leaving the coroutine body due to an exception and1630`false` otherwise.1631 1632The third argument, if present, should specify a function to be called.1633 1634If the third argument is present, the remaining arguments are the arguments to1635the function call.1636 1637.. code-block:: llvm1638 1639 call void (ptr, i1, ...) @llvm.coro.end.async(1640 ptr %hdl, i1 0,1641 ptr @must_tail_call_return,1642 ptr %ctxt, ptr %task, ptr %actor)1643 unreachable1644 1645.. _coro.suspend:1646.. _suspend points:1647 1648'llvm.coro.suspend' Intrinsic1649^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^1650::1651 1652 declare i8 @llvm.coro.suspend(token <save>, i1 <final>)1653 1654Overview:1655"""""""""1656 1657The '``llvm.coro.suspend``' marks the point where execution of a1658switched-resume coroutine is suspended and control is returned back1659to the caller. Conditional branches consuming the result of this1660intrinsic lead to basic blocks where coroutine should proceed when1661suspended (-1), resumed (0) or destroyed (1).1662 1663Arguments:1664""""""""""1665 1666The first argument refers to a token of `coro.save` intrinsic that marks the1667point when coroutine state is prepared for suspension. If `none` token is passed,1668the intrinsic behaves as if there were a `coro.save` immediately preceding1669the `coro.suspend` intrinsic.1670 1671The second argument indicates whether this suspension point is `final`_.1672The second argument only accepts constants. If more than one suspend point is1673designated as final, the resume and destroy branches should lead to the same1674basic blocks.1675 1676Example (normal suspend point):1677"""""""""""""""""""""""""""""""1678 1679.. code-block:: llvm1680 1681 %0 = call i8 @llvm.coro.suspend(token none, i1 false)1682 switch i8 %0, label %suspend [i8 0, label %resume1683 i8 1, label %cleanup]1684 1685Example (final suspend point):1686""""""""""""""""""""""""""""""1687 1688.. code-block:: llvm1689 1690 while.end:1691 %s.final = call i8 @llvm.coro.suspend(token none, i1 true)1692 switch i8 %s.final, label %suspend [i8 0, label %trap1693 i8 1, label %cleanup]1694 trap:1695 call void @llvm.trap()1696 unreachable1697 1698Semantics:1699""""""""""1700 1701If a coroutine that was suspended at the suspend point marked by this intrinsic1702is resumed via `coro.resume`_ the control will transfer to the basic block1703of the 0-case. If it is resumed via `coro.destroy`_, it will proceed to the1704basic block indicated by the 1-case. To suspend, coroutine proceeds to the1705default label.1706 1707If suspend intrinsic is marked as final, it can consider the `true` branch1708unreachable and can perform optimizations that can take advantage of that fact.1709 1710.. _coro.save:1711 1712'llvm.coro.save' Intrinsic1713^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^1714::1715 1716 declare token @llvm.coro.save(ptr <handle>)1717 1718Overview:1719"""""""""1720 1721The '``llvm.coro.save``' marks the point where a coroutine needs to update its1722state to prepare for resumption to be considered suspended (and thus eligible1723for resumption). It is illegal to merge two '``llvm.coro.save``' calls unless their1724'``llvm.coro.suspend``' users are also merged. So '``llvm.coro.save``' is currently1725tagged with the `no_merge` function attribute.1726 1727Arguments:1728""""""""""1729 1730The first argument points to a coroutine handle of the enclosing coroutine.1731 1732Semantics:1733""""""""""1734 1735Whatever coroutine state changes are required to enable resumption of1736the coroutine from the corresponding suspend point should be done at the point1737of `coro.save` intrinsic.1738 1739Example:1740""""""""1741 1742Separate save and suspend points are necessary when a coroutine is used to1743represent an asynchronous control flow driven by callbacks representing1744completions of asynchronous operations.1745 1746In such a case, a coroutine should be ready for resumption prior to a call to1747`async_op` function that may trigger resumption of a coroutine from the same or1748a different thread possibly prior to `async_op` call returning control back1749to the coroutine:1750 1751.. code-block:: llvm1752 1753 %save1 = call token @llvm.coro.save(ptr %hdl)1754 call void @async_op1(ptr %hdl)1755 %suspend1 = call i1 @llvm.coro.suspend(token %save1, i1 false)1756 switch i8 %suspend1, label %suspend [i8 0, label %resume11757 i8 1, label %cleanup]1758 1759.. _coro.suspend.async:1760 1761'llvm.coro.suspend.async' Intrinsic1762^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^1763::1764 1765 declare {ptr, ptr, ptr} @llvm.coro.suspend.async(1766 ptr <resume function>,1767 ptr <context projection function>,1768 ... <function to call>1769 ... <arguments to function>)1770 1771Overview:1772"""""""""1773 1774The '``llvm.coro.suspend.async``' intrinsic marks the point where1775execution of an async coroutine is suspended and control is passed to a callee.1776 1777Arguments:1778""""""""""1779 1780The first argument should be the result of the `llvm.coro.async.resume` intrinsic.1781Lowering will replace this intrinsic with the resume function for this suspend1782point.1783 1784The second argument is the `context projection function`. It should describe1785how-to restore the `async context` in the continuation function from the first1786argument of the continuation function. Its type is `ptr (ptr)`.1787 1788The third argument is the function that models transfer to the callee at the1789suspend point. It should take 3 arguments. Lowering will `musttail` call this1790function.1791 1792The fourth to six argument are the arguments for the third argument.1793 1794Semantics:1795""""""""""1796 1797The results of the intrinsic are mapped to the arguments of the resume function.1798Execution is suspended at this intrinsic and resumed when the resume function is1799called.1800 1801.. _coro.prepare.async:1802 1803'llvm.coro.prepare.async' Intrinsic1804^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^1805::1806 1807 declare ptr @llvm.coro.prepare.async(ptr <coroutine function>)1808 1809Overview:1810"""""""""1811 1812The '``llvm.coro.prepare.async``' intrinsic is used to block inlining of the1813async coroutine until after coroutine splitting.1814 1815Arguments:1816""""""""""1817 1818The first argument should be an async coroutine of type `void (ptr, ptr, ptr)`.1819Lowering will replace this intrinsic with its coroutine function argument.1820 1821.. _coro.suspend.retcon:1822 1823'llvm.coro.suspend.retcon' Intrinsic1824^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^1825::1826 1827 declare i1 @llvm.coro.suspend.retcon(...)1828 1829Overview:1830"""""""""1831 1832The '``llvm.coro.suspend.retcon``' intrinsic marks the point where1833execution of a returned-continuation coroutine is suspended and control1834is returned back to the caller.1835 1836`llvm.coro.suspend.retcon`` does not support separate save points;1837they are not useful when the continuation function is not locally1838accessible. That would be a more appropriate feature for a ``passcon``1839lowering that is not yet implemented.1840 1841Arguments:1842""""""""""1843 1844The types of the arguments must exactly match the yielded-types sequence1845of the coroutine. They will be turned into return values from the ramp1846and continuation functions, along with the next continuation function.1847 1848Semantics:1849""""""""""1850 1851The result of the intrinsic indicates whether the coroutine should resume1852abnormally (non-zero).1853 1854In a normal coroutine, it is undefined behavior if the coroutine executes1855a call to ``llvm.coro.suspend.retcon`` after resuming abnormally.1856 1857In a yield-once coroutine, it is undefined behavior if the coroutine1858executes a call to ``llvm.coro.suspend.retcon`` after resuming in any way.1859 1860.. _coro.await.suspend.void:1861 1862'llvm.coro.await.suspend.void' Intrinsic1863^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^1864::1865 1866 declare void @llvm.coro.await.suspend.void(1867 ptr <awaiter>,1868 ptr <handle>,1869 ptr <await_suspend_function>)1870 1871Overview:1872"""""""""1873 1874The '``llvm.coro.await.suspend.void``' intrinsic encapsulates C++ 1875`await-suspend` block until it can't interfere with coroutine transform.1876 1877The `await_suspend` block of `co_await` is essentially asynchronous1878to the execution of the coroutine. Inlining it normally into an unsplit1879coroutine can cause miscompilation because the coroutine CFG misrepresents1880the true control flow of the program: things that happen in the1881await_suspend are not guaranteed to happen prior to the resumption of the1882coroutine, and things that happen after the resumption of the coroutine1883(including its exit and the potential deallocation of the coroutine frame)1884are not guaranteed to happen only after the end of `await_suspend`.1885 1886This version of intrinsic corresponds to 1887'``void awaiter.await_suspend(...)``' variant.1888 1889Arguments:1890""""""""""1891 1892The first argument is a pointer to `awaiter` object.1893 1894The second argument is a pointer to the current coroutine's frame.1895 1896The third argument is a pointer to the wrapper function encapsulating1897`await-suspend` logic. Its signature must be1898 1899.. code-block:: llvm1900 1901 declare void @await_suspend_function(ptr %awaiter, ptr %hdl)1902 1903Semantics:1904""""""""""1905 1906The intrinsic must be used between corresponding `coro.save`_ and 1907`coro.suspend`_ calls. It is lowered to a direct 1908`await_suspend_function` call during `CoroSplit`_ pass.1909 1910Example:1911""""""""1912 1913.. code-block:: llvm1914 1915 ; before lowering1916 await.suspend:1917 %save = call token @llvm.coro.save(ptr %hdl)1918 call void @llvm.coro.await.suspend.void(1919 ptr %awaiter,1920 ptr %hdl,1921 ptr @await_suspend_function)1922 %suspend = call i8 @llvm.coro.suspend(token %save, i1 false)1923 ...1924 1925 ; after lowering1926 await.suspend:1927 %save = call token @llvm.coro.save(ptr %hdl)1928 ; the call to await_suspend_function can be inlined1929 call void @await_suspend_function(1930 ptr %awaiter,1931 ptr %hdl)1932 %suspend = call i8 @llvm.coro.suspend(token %save, i1 false) 1933 ...1934 1935 ; wrapper function example1936 define void @await_suspend_function(ptr %awaiter, ptr %hdl)1937 entry:1938 %hdl.arg = ... ; construct std::coroutine_handle from %hdl1939 call void @"Awaiter::await_suspend"(ptr %awaiter, ptr %hdl.arg)1940 ret void1941 1942.. _coro.await.suspend.bool:1943 1944'llvm.coro.await.suspend.bool' Intrinsic1945^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^1946::1947 1948 declare i1 @llvm.coro.await.suspend.bool(1949 ptr <awaiter>,1950 ptr <handle>,1951 ptr <await_suspend_function>)1952 1953Overview:1954"""""""""1955 1956The '``llvm.coro.await.suspend.bool``' intrinsic encapsulates C++1957`await-suspend` block until it can't interfere with coroutine transform.1958 1959The `await_suspend` block of `co_await` is essentially asynchronous1960to the execution of the coroutine. Inlining it normally into an unsplit1961coroutine can cause miscompilation because the coroutine CFG misrepresents1962the true control flow of the program: things that happen in the1963await_suspend are not guaranteed to happen prior to the resumption of the1964coroutine, and things that happen after the resumption of the coroutine1965(including its exit and the potential deallocation of the coroutine frame)1966are not guaranteed to happen only after the end of `await_suspend`.1967 1968This version of intrinsic corresponds to 1969'``bool awaiter.await_suspend(...)``' variant.1970 1971Arguments:1972""""""""""1973 1974The first argument is a pointer to `awaiter` object.1975 1976The second argument is a pointer to the current coroutine's frame.1977 1978The third argument is a pointer to the wrapper function encapsulating1979`await-suspend` logic. Its signature must be1980 1981.. code-block:: llvm1982 1983 declare i1 @await_suspend_function(ptr %awaiter, ptr %hdl)1984 1985Semantics:1986""""""""""1987 1988The intrinsic must be used between corresponding `coro.save`_ and 1989`coro.suspend`_ calls. It is lowered to a direct 1990`await_suspend_function` call during `CoroSplit`_ pass.1991 1992If `await_suspend_function` call returns `true`, the current coroutine is1993immediately resumed.1994 1995Example:1996""""""""1997 1998.. code-block:: llvm1999 2000 ; before lowering2001 await.suspend:2002 %save = call token @llvm.coro.save(ptr %hdl)2003 %resume = call i1 @llvm.coro.await.suspend.bool(2004 ptr %awaiter,2005 ptr %hdl,2006 ptr @await_suspend_function)2007 br i1 %resume, %await.suspend.bool, %await.ready2008 await.suspend.bool:2009 %suspend = call i8 @llvm.coro.suspend(token %save, i1 false)2010 ...2011 await.ready:2012 call void @"Awaiter::await_resume"(ptr %awaiter)2013 ...2014 2015 ; after lowering2016 await.suspend:2017 %save = call token @llvm.coro.save(ptr %hdl)2018 ; the call to await_suspend_function can inlined2019 %resume = call i1 @await_suspend_function(2020 ptr %awaiter,2021 ptr %hdl)2022 br i1 %resume, %await.suspend.bool, %await.ready2023 ...2024 2025 ; wrapper function example2026 define i1 @await_suspend_function(ptr %awaiter, ptr %hdl)2027 entry:2028 %hdl.arg = ... ; construct std::coroutine_handle from %hdl2029 %resume = call i1 @"Awaiter::await_suspend"(ptr %awaiter, ptr %hdl.arg)2030 ret i1 %resume2031 2032.. _coro.await.suspend.handle:2033 2034'llvm.coro.await.suspend.handle' Intrinsic2035^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^2036::2037 2038 declare void @llvm.coro.await.suspend.handle(2039 ptr <awaiter>,2040 ptr <handle>,2041 ptr <await_suspend_function>)2042 2043Overview:2044"""""""""2045 2046The '``llvm.coro.await.suspend.handle``' intrinsic encapsulates C++2047`await-suspend` block until it can't interfere with coroutine transform.2048 2049The `await_suspend` block of `co_await` is essentially asynchronous2050to the execution of the coroutine. Inlining it normally into an unsplit2051coroutine can cause miscompilation because the coroutine CFG misrepresents2052the true control flow of the program: things that happen in the2053await_suspend are not guaranteed to happen prior to the resumption of the2054coroutine, and things that happen after the resumption of the coroutine2055(including its exit and the potential deallocation of the coroutine frame)2056are not guaranteed to happen only after the end of `await_suspend`.2057 2058This version of intrinsic corresponds to 2059'``std::coroutine_handle<> awaiter.await_suspend(...)``' variant.2060 2061Arguments:2062""""""""""2063 2064The first argument is a pointer to `awaiter` object.2065 2066The second argument is a pointer to the current coroutine's frame.2067 2068The third argument is a pointer to the wrapper function encapsulating2069`await-suspend` logic. Its signature must be2070 2071.. code-block:: llvm2072 2073 declare ptr @await_suspend_function(ptr %awaiter, ptr %hdl)2074 2075Semantics:2076""""""""""2077 2078The intrinsic must be used between corresponding `coro.save`_ and 2079`coro.suspend`_ calls. It is lowered to a direct 2080`await_suspend_function` call during `CoroSplit`_ pass.2081 2082`await_suspend_function` must return a pointer to a valid2083coroutine frame. The intrinsic will be lowered to a tail call resuming the2084returned coroutine frame. It will be marked `musttail` on targets that support2085that. Instructions following the intrinsic will become unreachable.2086 2087Example:2088""""""""2089 2090.. code-block:: llvm2091 2092 ; before lowering2093 await.suspend:2094 %save = call token @llvm.coro.save(ptr %hdl)2095 call void @llvm.coro.await.suspend.handle(2096 ptr %awaiter,2097 ptr %hdl,2098 ptr @await_suspend_function)2099 %suspend = call i8 @llvm.coro.suspend(token %save, i1 false)2100 ...2101 2102 ; after lowering2103 await.suspend:2104 %save = call token @llvm.coro.save(ptr %hdl)2105 ; the call to await_suspend_function can be inlined2106 %next = call ptr @await_suspend_function(2107 ptr %awaiter,2108 ptr %hdl)2109 musttail call void @llvm.coro.resume(%next)2110 ret void2111 ...2112 2113 ; wrapper function example2114 define ptr @await_suspend_function(ptr %awaiter, ptr %hdl)2115 entry:2116 %hdl.arg = ... ; construct std::coroutine_handle from %hdl2117 %hdl.raw = call ptr @"Awaiter::await_suspend"(ptr %awaiter, ptr %hdl.arg)2118 %hdl.result = ... ; get address of returned coroutine handle2119 ret ptr %hdl.result2120 2121'llvm.coro.is_in_ramp' Intrinsic2122^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^2123::2124 2125 declare i1 @llvm.coro.is_in_ramp()2126 2127Overview:2128"""""""""2129 2130The '``llvm.coro.is_in_ramp``' intrinsic returns a bool value that marks coroutine ramp2131function and resume/destroy function.2132 2133Arguments:2134""""""""""2135 2136None2137 2138Semantics:2139""""""""""2140 2141The `CoroSpit` pass replaces `coro.is_in_ramp` with ``True`` ramp functions.2142Otherwise, it is replaced with ``False``, allowing the frontend to separate2143ramp function and resume/destroy function.2144 2145Coroutine Transformation Passes2146===============================2147CoroEarly2148---------2149The CoroEarly pass ensures later middle end passes correctly interpret coroutine 2150semantics and lowers coroutine intrinsics that not needed to be preserved to 2151help later coroutine passes. This pass lowers `coro.promise`_, `coro.frame`_ and 2152`coro.done`_ intrinsics. Afterwards, it replaces uses of promise alloca with 2153`coro.promise`_ intrinsic.2154 2155.. _CoroSplit:2156 2157CoroSplit2158---------2159The pass CoroSplit builds coroutine frame and outlines resume and destroy parts2160into separate functions. This pass also lowers `coro.await.suspend.void`_,2161`coro.await.suspend.bool`_ and `coro.await.suspend.handle`_ intrinsics.2162 2163CoroAnnotationElide2164-------------------2165This pass finds all usages of coroutines that are "must elide" and replaces2166`coro.begin` intrinsic with an address of a coroutine frame placed on its caller2167and replaces `coro.alloc` and `coro.free` intrinsics with `false` and `null`2168respectively to remove the deallocation code.2169 2170CoroElide2171---------2172The pass CoroElide examines if the inlined coroutine is eligible for heap2173allocation elision optimization. If so, it replaces2174`coro.begin` intrinsic with an address of a coroutine frame placed on its caller2175and replaces `coro.alloc` and `coro.free` intrinsics with `false` and `null`2176respectively to remove the deallocation code.2177This pass also replaces `coro.resume` and `coro.destroy` intrinsics with direct2178calls to resume and destroy functions for a particular coroutine where possible.2179 2180CoroCleanup2181-----------2182This pass runs late to lower all coroutine related intrinsics not replaced by2183earlier passes.2184 2185Attributes2186==========2187 2188coro_only_destroy_when_complete2189-------------------------------2190 2191When the coroutine is marked with coro_only_destroy_when_complete, it indicates2192the coroutine must reach the final suspend point when it get destroyed.2193 2194This attribute only works for switched-resume coroutines now.2195 2196coro_elide_safe2197---------------2198 2199When a Call or Invoke instruction to switch ABI coroutine `f` is marked with2200`coro_elide_safe`, CoroSplitPass generates a `f.noalloc` ramp function.2201`f.noalloc` has one more argument than its original ramp function `f`, which is2202the pointer to the allocated frame. `f.noalloc` also suppresses any allocations2203or deallocations that may be guarded by `@llvm.coro.alloc` and `@llvm.coro.free`.2204 2205CoroAnnotationElidePass performs the heap elision when possible. Note that for2206recursive or mutually recursive functions this elision is usually not possible.2207 2208Metadata2209========2210 2211'``coro.outside.frame``' Metadata2212---------------------------------2213 2214``coro.outside.frame`` metadata may be attached to an alloca instruction to2215to signify that it shouldn't be promoted to the coroutine frame, useful for2216filtering allocas out by the frontend when emitting internal control mechanisms.2217Additionally, this metadata is only used as a flag, so the associated2218node must be empty.2219 2220.. code-block:: text2221 2222 %__coro_gro = alloca %struct.GroType, align 1, !coro.outside.frame !02223 2224 ...2225 !0 = !{}2226 2227Areas Requiring Attention2228=========================2229#. When coro.suspend returns -1, the coroutine is suspended, and it's possible2230 that the coroutine has already been destroyed (hence the frame has been freed).2231 We cannot access anything on the frame on the suspend path.2232 However there is nothing that prevents the compiler from moving instructions2233 along that path (e.g. LICM), which can lead to use-after-free. At the moment2234 we disabled LICM for loops that have coro.suspend, but the general problem still2235 exists and requires a general solution.2236 2237#. Take advantage of the lifetime intrinsics for the data that goes into the2238 coroutine frame. Leave lifetime intrinsics as is for the data that stays in2239 allocas.2240 2241#. The CoroElide optimization pass relies on coroutine ramp function to be2242 inlined. It would be beneficial to split the ramp function further to2243 increase the chance that it will get inlined into its caller.2244 2245#. Design a convention that would make it possible to apply coroutine heap2246 elision optimization across ABI boundaries.2247 2248#. Cannot handle coroutines with `inalloca` parameters (used in x86 on Windows).2249 2250#. Alignment is ignored by coro.begin and coro.free intrinsics.2251 2252#. Make required changes to make sure that coroutine optimizations work with2253 LTO.2254 2255#. More tests, more tests, more tests2256