brintos / llvm-project-archived public Read only

1//===-- AMDGPULowerBufferFatPointers.cpp ---------------------------=//2//3// Part of the LLVM Project, under the Apache License v2.0 with LLVM Exceptions.4// See https://llvm.org/LICENSE.txt for license information.5// SPDX-License-Identifier: Apache-2.0 WITH LLVM-exception6//7//===----------------------------------------------------------------------===//8//9// This pass lowers operations on buffer fat pointers (addrspace 7) to10// operations on buffer resources (addrspace 8) and is needed for correct11// codegen.12//13// # Background14//15// Address space 7 (the buffer fat pointer) is a 160-bit pointer that consists16// of a 128-bit buffer descriptor and a 32-bit offset into that descriptor.17// The buffer resource part needs to be it needs to be a "raw" buffer resource18// (it must have a stride of 0 and bounds checks must be in raw buffer mode19// or disabled).20//21// When these requirements are met, a buffer resource can be treated as a22// typical (though quite wide) pointer that follows typical LLVM pointer23// semantics. This allows the frontend to reason about such buffers (which are24// often encountered in the context of SPIR-V kernels).25//26// However, because of their non-power-of-2 size, these fat pointers cannot be27// present during translation to MIR (though this restriction may be lifted28// during the transition to GlobalISel). Therefore, this pass is needed in order29// to correctly implement these fat pointers.30//31// The resource intrinsics take the resource part (the address space 8 pointer)32// and the offset part (the 32-bit integer) as separate arguments. In addition,33// many users of these buffers manipulate the offset while leaving the resource34// part alone. For these reasons, we want to typically separate the resource35// and offset parts into separate variables, but combine them together when36// encountering cases where this is required, such as by inserting these values37// into aggretates or moving them to memory.38//39// Therefore, at a high level, `ptr addrspace(7) %x` becomes `ptr addrspace(8)40// %x.rsrc` and `i32 %x.off`, which will be combined into `{ptr addrspace(8),41// i32} %x = {%x.rsrc, %x.off}` if needed. Similarly, `vector<Nxp7>` becomes42// `{vector<Nxp8>, vector<Nxi32 >}` and its component parts.43//44// # Implementation45//46// This pass proceeds in three main phases:47//48// ## Rewriting loads and stores of p7 and memcpy()-like handling49//50// The first phase is to rewrite away all loads and stors of `ptr addrspace(7)`,51// including aggregates containing such pointers, to ones that use `i160`. This52// is handled by `StoreFatPtrsAsIntsAndExpandMemcpyVisitor` , which visits53// loads, stores, and allocas and, if the loaded or stored type contains `ptr54// addrspace(7)`, rewrites that type to one where the p7s are replaced by i160s,55// copying other parts of aggregates as needed. In the case of a store, each56// pointer is `ptrtoint`d to i160 before storing, and load integers are57// `inttoptr`d back. This same transformation is applied to vectors of pointers.58//59// Such a transformation allows the later phases of the pass to not need60// to handle buffer fat pointers moving to and from memory, where we load61// have to handle the incompatibility between a `{Nxp8, Nxi32}` representation62// and `Nxi60` directly. Instead, that transposing action (where the vectors63// of resources and vectors of offsets are concatentated before being stored to64// memory) are handled through implementing `inttoptr` and `ptrtoint` only.65//66// Atomics operations on `ptr addrspace(7)` values are not suppported, as the67// hardware does not include a 160-bit atomic.68//69// In order to save on O(N) work and to ensure that the contents type70// legalizer correctly splits up wide loads, also unconditionally lower71// memcpy-like intrinsics into loops here.72//73// ## Buffer contents type legalization74//75// The underlying buffer intrinsics only support types up to 128 bits long,76// and don't support complex types. If buffer operations were77// standard pointer operations that could be represented as MIR-level loads,78// this would be handled by the various legalization schemes in instruction79// selection. However, because we have to do the conversion from `load` and80// `store` to intrinsics at LLVM IR level, we must perform that legalization81// ourselves.82//83// This involves a combination of84// - Converting arrays to vectors where possible85// - Otherwise, splitting loads and stores of aggregates into loads/stores of86//   each component.87// - Zero-extending things to fill a whole number of bytes88// - Casting values of types that don't neatly correspond to supported machine89// value90//   (for example, an i96 or i256) into ones that would work (91//    like <3 x i32> and <8 x i32>, respectively)92// - Splitting values that are too long (such as aforementioned <8 x i32>) into93//   multiple operations.94//95// ## Type remapping96//97// We use a `ValueMapper` to mangle uses of [vectors of] buffer fat pointers98// to the corresponding struct type, which has a resource part and an offset99// part.100//101// This uses a `BufferFatPtrToStructTypeMap` and a `FatPtrConstMaterializer`102// to, usually by way of `setType`ing values. Constants are handled here103// because there isn't a good way to fix them up later.104//105// This has the downside of leaving the IR in an invalid state (for example,106// the instruction `getelementptr {ptr addrspace(8), i32} %p, ...` will exist),107// but all such invalid states will be resolved by the third phase.108//109// Functions that don't take buffer fat pointers are modified in place. Those110// that do take such pointers have their basic blocks moved to a new function111// with arguments that are {ptr addrspace(8), i32} arguments and return values.112// This phase also records intrinsics so that they can be remangled or deleted113// later.114//115// ## Splitting pointer structs116//117// The meat of this pass consists of defining semantics for operations that118// produce or consume [vectors of] buffer fat pointers in terms of their119// resource and offset parts. This is accomplished throgh the `SplitPtrStructs`120// visitor.121//122// In the first pass through each function that is being lowered, the splitter123// inserts new instructions to implement the split-structures behavior, which is124// needed for correctness and performance. It records a list of "split users",125// instructions that are being replaced by operations on the resource and offset126// parts.127//128// Split users do not necessarily need to produce parts themselves (129// a `load float, ptr addrspace(7)` does not, for example), but, if they do not130// generate fat buffer pointers, they must RAUW in their replacement131// instructions during the initial visit.132//133// When these new instructions are created, they use the split parts recorded134// for their initial arguments in order to generate their replacements, creating135// a parallel set of instructions that does not refer to the original fat136// pointer values but instead to their resource and offset components.137//138// Instructions, such as `extractvalue`, that produce buffer fat pointers from139// sources that do not have split parts, have such parts generated using140// `extractvalue`. This is also the initial handling of PHI nodes, which141// are then cleaned up.142//143// ### Conditionals144//145// PHI nodes are initially given resource parts via `extractvalue`. However,146// this is not an efficient rewrite of such nodes, as, in most cases, the147// resource part in a conditional or loop remains constant throughout the loop148// and only the offset varies. Failing to optimize away these constant resources149// would cause additional registers to be sent around loops and might lead to150// waterfall loops being generated for buffer operations due to the151// "non-uniform" resource argument.152//153// Therefore, after all instructions have been visited, the pointer splitter154// post-processes all encountered conditionals. Given a PHI node or select,155// getPossibleRsrcRoots() collects all values that the resource parts of that156// conditional's input could come from as well as collecting all conditional157// instructions encountered during the search. If, after filtering out the158// initial node itself, the set of encountered conditionals is a subset of the159// potential roots and there is a single potential resource that isn't in the160// conditional set, that value is the only possible value the resource argument161// could have throughout the control flow.162//163// If that condition is met, then a PHI node can have its resource part changed164// to the singleton value and then be replaced by a PHI on the offsets.165// Otherwise, each PHI node is split into two, one for the resource part and one166// for the offset part, which replace the temporary `extractvalue` instructions167// that were added during the first pass.168//169// Similar logic applies to `select`, where170// `%z = select i1 %cond, %cond, ptr addrspace(7) %x, ptr addrspace(7) %y`171// can be split into `%z.rsrc = %x.rsrc` and172// `%z.off = select i1 %cond, ptr i32 %x.off, i32 %y.off`173// if both `%x` and `%y` have the same resource part, but two `select`174// operations will be needed if they do not.175//176// ### Final processing177//178// After conditionals have been cleaned up, the IR for each function is179// rewritten to remove all the old instructions that have been split up.180//181// Any instruction that used to produce a buffer fat pointer (and therefore now182// produces a resource-and-offset struct after type remapping) is183// replaced as follows:184// 1. All debug value annotations are cloned to reflect that the resource part185//    and offset parts are computed separately and constitute different186//    fragments of the underlying source language variable.187// 2. All uses that were themselves split are replaced by a `poison` of the188//    struct type, as they will themselves be erased soon. This rule, combined189//    with debug handling, should leave the use lists of split instructions190//    empty in almost all cases.191// 3. If a user of the original struct-valued result remains, the structure192//    needed for the new types to work is constructed out of the newly-defined193//    parts, and the original instruction is replaced by this structure194//    before being erased. Instructions requiring this construction include195//    `ret` and `insertvalue`.196//197// # Consequences198//199// This pass does not alter the CFG.200//201// Alias analysis information will become coarser, as the LLVM alias analyzer202// cannot handle the buffer intrinsics. Specifically, while we can determine203// that the following two loads do not alias:204// ```205//   %y = getelementptr i32, ptr addrspace(7) %x, i32 1206//   %a = load i32, ptr addrspace(7) %x207//   %b = load i32, ptr addrspace(7) %y208// ```209// we cannot (except through some code that runs during scheduling) determine210// that the rewritten loads below do not alias.211// ```212//   %y.off = add i32 %x.off, 1213//   %a = call @llvm.amdgcn.raw.ptr.buffer.load(ptr addrspace(8) %x.rsrc, i32214//     %x.off, ...)215//   %b = call @llvm.amdgcn.raw.ptr.buffer.load(ptr addrspace(8)216//     %x.rsrc, i32 %y.off, ...)217// ```218// However, existing alias information is preserved.219//===----------------------------------------------------------------------===//220 221#include "AMDGPU.h"222#include "AMDGPUTargetMachine.h"223#include "GCNSubtarget.h"224#include "SIDefines.h"225#include "llvm/ADT/SetOperations.h"226#include "llvm/ADT/SmallVector.h"227#include "llvm/Analysis/InstSimplifyFolder.h"228#include "llvm/Analysis/TargetTransformInfo.h"229#include "llvm/Analysis/Utils/Local.h"230#include "llvm/CodeGen/TargetPassConfig.h"231#include "llvm/IR/AttributeMask.h"232#include "llvm/IR/Constants.h"233#include "llvm/IR/DebugInfo.h"234#include "llvm/IR/DerivedTypes.h"235#include "llvm/IR/IRBuilder.h"236#include "llvm/IR/InstIterator.h"237#include "llvm/IR/InstVisitor.h"238#include "llvm/IR/Instructions.h"239#include "llvm/IR/IntrinsicInst.h"240#include "llvm/IR/Intrinsics.h"241#include "llvm/IR/IntrinsicsAMDGPU.h"242#include "llvm/IR/Metadata.h"243#include "llvm/IR/Operator.h"244#include "llvm/IR/PatternMatch.h"245#include "llvm/IR/ReplaceConstant.h"246#include "llvm/IR/ValueHandle.h"247#include "llvm/InitializePasses.h"248#include "llvm/Pass.h"249#include "llvm/Support/AMDGPUAddrSpace.h"250#include "llvm/Support/Alignment.h"251#include "llvm/Support/AtomicOrdering.h"252#include "llvm/Support/Debug.h"253#include "llvm/Support/ErrorHandling.h"254#include "llvm/Transforms/Utils/Cloning.h"255#include "llvm/Transforms/Utils/Local.h"256#include "llvm/Transforms/Utils/LowerMemIntrinsics.h"257#include "llvm/Transforms/Utils/ValueMapper.h"258 259#define DEBUG_TYPE "amdgpu-lower-buffer-fat-pointers"260 261using namespace llvm;262 263static constexpr unsigned BufferOffsetWidth = 32;264 265namespace {266/// Recursively replace instances of ptr addrspace(7) and vector<Nxptr267/// addrspace(7)> with some other type as defined by the relevant subclass.268class BufferFatPtrTypeLoweringBase : public ValueMapTypeRemapper {269  DenseMap<Type *, Type *> Map;270 271  Type *remapTypeImpl(Type *Ty);272 273protected:274  virtual Type *remapScalar(PointerType *PT) = 0;275  virtual Type *remapVector(VectorType *VT) = 0;276 277  const DataLayout &DL;278 279public:280  BufferFatPtrTypeLoweringBase(const DataLayout &DL) : DL(DL) {}281  Type *remapType(Type *SrcTy) override;282  void clear() { Map.clear(); }283};284 285/// Remap ptr addrspace(7) to i160 and vector<Nxptr addrspace(7)> to286/// vector<Nxi60> in order to correctly handling loading/storing these values287/// from memory.288class BufferFatPtrToIntTypeMap : public BufferFatPtrTypeLoweringBase {289  using BufferFatPtrTypeLoweringBase::BufferFatPtrTypeLoweringBase;290 291protected:292  Type *remapScalar(PointerType *PT) override { return DL.getIntPtrType(PT); }293  Type *remapVector(VectorType *VT) override { return DL.getIntPtrType(VT); }294};295 296/// Remap ptr addrspace(7) to {ptr addrspace(8), i32} (the resource and offset297/// parts of the pointer) so that we can easily rewrite operations on these298/// values that aren't loading them from or storing them to memory.299class BufferFatPtrToStructTypeMap : public BufferFatPtrTypeLoweringBase {300  using BufferFatPtrTypeLoweringBase::BufferFatPtrTypeLoweringBase;301 302protected:303  Type *remapScalar(PointerType *PT) override;304  Type *remapVector(VectorType *VT) override;305};306} // namespace307 308// This code is adapted from the type remapper in lib/Linker/IRMover.cpp309Type *BufferFatPtrTypeLoweringBase::remapTypeImpl(Type *Ty) {310  Type **Entry = &Map[Ty];311  if (*Entry)312    return *Entry;313  if (auto *PT = dyn_cast<PointerType>(Ty)) {314    if (PT->getAddressSpace() == AMDGPUAS::BUFFER_FAT_POINTER) {315      return *Entry = remapScalar(PT);316    }317  }318  if (auto *VT = dyn_cast<VectorType>(Ty)) {319    auto *PT = dyn_cast<PointerType>(VT->getElementType());320    if (PT && PT->getAddressSpace() == AMDGPUAS::BUFFER_FAT_POINTER) {321      return *Entry = remapVector(VT);322    }323    return *Entry = Ty;324  }325  // Whether the type is one that is structurally uniqued - that is, if it is326  // not a named struct (the only kind of type where multiple structurally327  // identical types that have a distinct `Type*`)328  StructType *TyAsStruct = dyn_cast<StructType>(Ty);329  bool IsUniqued = !TyAsStruct || TyAsStruct->isLiteral();330  // Base case for ints, floats, opaque pointers, and so on, which don't331  // require recursion.332  if (Ty->getNumContainedTypes() == 0 && IsUniqued)333    return *Entry = Ty;334  bool Changed = false;335  SmallVector<Type *> ElementTypes(Ty->getNumContainedTypes(), nullptr);336  for (unsigned int I = 0, E = Ty->getNumContainedTypes(); I < E; ++I) {337    Type *OldElem = Ty->getContainedType(I);338    Type *NewElem = remapTypeImpl(OldElem);339    ElementTypes[I] = NewElem;340    Changed |= (OldElem != NewElem);341  }342  // Recursive calls to remapTypeImpl() may have invalidated pointer.343  Entry = &Map[Ty];344  if (!Changed) {345    return *Entry = Ty;346  }347  if (auto *ArrTy = dyn_cast<ArrayType>(Ty))348    return *Entry = ArrayType::get(ElementTypes[0], ArrTy->getNumElements());349  if (auto *FnTy = dyn_cast<FunctionType>(Ty))350    return *Entry = FunctionType::get(ElementTypes[0],351                                      ArrayRef(ElementTypes).slice(1),352                                      FnTy->isVarArg());353  if (auto *STy = dyn_cast<StructType>(Ty)) {354    // Genuine opaque types don't have a remapping.355    if (STy->isOpaque())356      return *Entry = Ty;357    bool IsPacked = STy->isPacked();358    if (IsUniqued)359      return *Entry = StructType::get(Ty->getContext(), ElementTypes, IsPacked);360    SmallString<16> Name(STy->getName());361    STy->setName("");362    return *Entry = StructType::create(Ty->getContext(), ElementTypes, Name,363                                       IsPacked);364  }365  llvm_unreachable("Unknown type of type that contains elements");366}367 368Type *BufferFatPtrTypeLoweringBase::remapType(Type *SrcTy) {369  return remapTypeImpl(SrcTy);370}371 372Type *BufferFatPtrToStructTypeMap::remapScalar(PointerType *PT) {373  LLVMContext &Ctx = PT->getContext();374  return StructType::get(PointerType::get(Ctx, AMDGPUAS::BUFFER_RESOURCE),375                         IntegerType::get(Ctx, BufferOffsetWidth));376}377 378Type *BufferFatPtrToStructTypeMap::remapVector(VectorType *VT) {379  ElementCount EC = VT->getElementCount();380  LLVMContext &Ctx = VT->getContext();381  Type *RsrcVec =382      VectorType::get(PointerType::get(Ctx, AMDGPUAS::BUFFER_RESOURCE), EC);383  Type *OffVec = VectorType::get(IntegerType::get(Ctx, BufferOffsetWidth), EC);384  return StructType::get(RsrcVec, OffVec);385}386 387static bool isBufferFatPtrOrVector(Type *Ty) {388  if (auto *PT = dyn_cast<PointerType>(Ty->getScalarType()))389    return PT->getAddressSpace() == AMDGPUAS::BUFFER_FAT_POINTER;390  return false;391}392 393// True if the type is {ptr addrspace(8), i32} or a struct containing vectors of394// those types. Used to quickly skip instructions we don't need to process.395static bool isSplitFatPtr(Type *Ty) {396  auto *ST = dyn_cast<StructType>(Ty);397  if (!ST)398    return false;399  if (!ST->isLiteral() || ST->getNumElements() != 2)400    return false;401  auto *MaybeRsrc =402      dyn_cast<PointerType>(ST->getElementType(0)->getScalarType());403  auto *MaybeOff =404      dyn_cast<IntegerType>(ST->getElementType(1)->getScalarType());405  return MaybeRsrc && MaybeOff &&406         MaybeRsrc->getAddressSpace() == AMDGPUAS::BUFFER_RESOURCE &&407         MaybeOff->getBitWidth() == BufferOffsetWidth;408}409 410// True if the result type or any argument types are buffer fat pointers.411static bool isBufferFatPtrConst(Constant *C) {412  Type *T = C->getType();413  return isBufferFatPtrOrVector(T) || any_of(C->operands(), [](const Use &U) {414           return isBufferFatPtrOrVector(U.get()->getType());415         });416}417 418namespace {419/// Convert [vectors of] buffer fat pointers to integers when they are read from420/// or stored to memory. This ensures that these pointers will have the same421/// memory layout as before they are lowered, even though they will no longer422/// have their previous layout in registers/in the program (they'll be broken423/// down into resource and offset parts). This has the downside of imposing424/// marshalling costs when reading or storing these values, but since placing425/// such pointers into memory is an uncommon operation at best, we feel that426/// this cost is acceptable for better performance in the common case.427class StoreFatPtrsAsIntsAndExpandMemcpyVisitor428    : public InstVisitor<StoreFatPtrsAsIntsAndExpandMemcpyVisitor, bool> {429  BufferFatPtrToIntTypeMap *TypeMap;430 431  ValueToValueMapTy ConvertedForStore;432 433  IRBuilder<InstSimplifyFolder> IRB;434 435  const TargetMachine *TM;436 437  // Convert all the buffer fat pointers within the input value to inttegers438  // so that it can be stored in memory.439  Value *fatPtrsToInts(Value *V, Type *From, Type *To, const Twine &Name);440  // Convert all the i160s that need to be buffer fat pointers (as specified)441  // by the To type) into those pointers to preserve the semantics of the rest442  // of the program.443  Value *intsToFatPtrs(Value *V, Type *From, Type *To, const Twine &Name);444 445public:446  StoreFatPtrsAsIntsAndExpandMemcpyVisitor(BufferFatPtrToIntTypeMap *TypeMap,447                                           const DataLayout &DL,448                                           LLVMContext &Ctx,449                                           const TargetMachine *TM)450      : TypeMap(TypeMap), IRB(Ctx, InstSimplifyFolder(DL)), TM(TM) {}451  bool processFunction(Function &F);452 453  bool visitInstruction(Instruction &I) { return false; }454  bool visitAllocaInst(AllocaInst &I);455  bool visitLoadInst(LoadInst &LI);456  bool visitStoreInst(StoreInst &SI);457  bool visitGetElementPtrInst(GetElementPtrInst &I);458 459  bool visitMemCpyInst(MemCpyInst &MCI);460  bool visitMemMoveInst(MemMoveInst &MMI);461  bool visitMemSetInst(MemSetInst &MSI);462  bool visitMemSetPatternInst(MemSetPatternInst &MSPI);463};464} // namespace465 466Value *StoreFatPtrsAsIntsAndExpandMemcpyVisitor::fatPtrsToInts(467    Value *V, Type *From, Type *To, const Twine &Name) {468  if (From == To)469    return V;470  ValueToValueMapTy::iterator Find = ConvertedForStore.find(V);471  if (Find != ConvertedForStore.end())472    return Find->second;473  if (isBufferFatPtrOrVector(From)) {474    Value *Cast = IRB.CreatePtrToInt(V, To, Name + ".int");475    ConvertedForStore[V] = Cast;476    return Cast;477  }478  if (From->getNumContainedTypes() == 0)479    return V;480  // Structs, arrays, and other compound types.481  Value *Ret = PoisonValue::get(To);482  if (auto *AT = dyn_cast<ArrayType>(From)) {483    Type *FromPart = AT->getArrayElementType();484    Type *ToPart = cast<ArrayType>(To)->getElementType();485    for (uint64_t I = 0, E = AT->getArrayNumElements(); I < E; ++I) {486      Value *Field = IRB.CreateExtractValue(V, I);487      Value *NewField =488          fatPtrsToInts(Field, FromPart, ToPart, Name + "." + Twine(I));489      Ret = IRB.CreateInsertValue(Ret, NewField, I);490    }491  } else {492    for (auto [Idx, FromPart, ToPart] :493         enumerate(From->subtypes(), To->subtypes())) {494      Value *Field = IRB.CreateExtractValue(V, Idx);495      Value *NewField =496          fatPtrsToInts(Field, FromPart, ToPart, Name + "." + Twine(Idx));497      Ret = IRB.CreateInsertValue(Ret, NewField, Idx);498    }499  }500  ConvertedForStore[V] = Ret;501  return Ret;502}503 504Value *StoreFatPtrsAsIntsAndExpandMemcpyVisitor::intsToFatPtrs(505    Value *V, Type *From, Type *To, const Twine &Name) {506  if (From == To)507    return V;508  if (isBufferFatPtrOrVector(To)) {509    Value *Cast = IRB.CreateIntToPtr(V, To, Name + ".ptr");510    return Cast;511  }512  if (From->getNumContainedTypes() == 0)513    return V;514  // Structs, arrays, and other compound types.515  Value *Ret = PoisonValue::get(To);516  if (auto *AT = dyn_cast<ArrayType>(From)) {517    Type *FromPart = AT->getArrayElementType();518    Type *ToPart = cast<ArrayType>(To)->getElementType();519    for (uint64_t I = 0, E = AT->getArrayNumElements(); I < E; ++I) {520      Value *Field = IRB.CreateExtractValue(V, I);521      Value *NewField =522          intsToFatPtrs(Field, FromPart, ToPart, Name + "." + Twine(I));523      Ret = IRB.CreateInsertValue(Ret, NewField, I);524    }525  } else {526    for (auto [Idx, FromPart, ToPart] :527         enumerate(From->subtypes(), To->subtypes())) {528      Value *Field = IRB.CreateExtractValue(V, Idx);529      Value *NewField =530          intsToFatPtrs(Field, FromPart, ToPart, Name + "." + Twine(Idx));531      Ret = IRB.CreateInsertValue(Ret, NewField, Idx);532    }533  }534  return Ret;535}536 537bool StoreFatPtrsAsIntsAndExpandMemcpyVisitor::processFunction(Function &F) {538  bool Changed = false;539  // Process memcpy-like instructions after the main iteration because they can540  // invalidate iterators.541  SmallVector<WeakTrackingVH> CanBecomeLoops;542  for (Instruction &I : make_early_inc_range(instructions(F))) {543    if (isa<MemTransferInst, MemSetInst, MemSetPatternInst>(I))544      CanBecomeLoops.push_back(&I);545    else546      Changed |= visit(I);547  }548  for (WeakTrackingVH VH : make_early_inc_range(CanBecomeLoops)) {549    Changed |= visit(cast<Instruction>(VH));550  }551  ConvertedForStore.clear();552  return Changed;553}554 555bool StoreFatPtrsAsIntsAndExpandMemcpyVisitor::visitAllocaInst(AllocaInst &I) {556  Type *Ty = I.getAllocatedType();557  Type *NewTy = TypeMap->remapType(Ty);558  if (Ty == NewTy)559    return false;560  I.setAllocatedType(NewTy);561  return true;562}563 564bool StoreFatPtrsAsIntsAndExpandMemcpyVisitor::visitGetElementPtrInst(565    GetElementPtrInst &I) {566  Type *Ty = I.getSourceElementType();567  Type *NewTy = TypeMap->remapType(Ty);568  if (Ty == NewTy)569    return false;570  // We'll be rewriting the type `ptr addrspace(7)` out of existence soon, so571  // make sure GEPs don't have different semantics with the new type.572  I.setSourceElementType(NewTy);573  I.setResultElementType(TypeMap->remapType(I.getResultElementType()));574  return true;575}576 577bool StoreFatPtrsAsIntsAndExpandMemcpyVisitor::visitLoadInst(LoadInst &LI) {578  Type *Ty = LI.getType();579  Type *IntTy = TypeMap->remapType(Ty);580  if (Ty == IntTy)581    return false;582 583  IRB.SetInsertPoint(&LI);584  auto *NLI = cast<LoadInst>(LI.clone());585  NLI->mutateType(IntTy);586  NLI = IRB.Insert(NLI);587  NLI->takeName(&LI);588 589  Value *CastBack = intsToFatPtrs(NLI, IntTy, Ty, NLI->getName());590  LI.replaceAllUsesWith(CastBack);591  LI.eraseFromParent();592  return true;593}594 595bool StoreFatPtrsAsIntsAndExpandMemcpyVisitor::visitStoreInst(StoreInst &SI) {596  Value *V = SI.getValueOperand();597  Type *Ty = V->getType();598  Type *IntTy = TypeMap->remapType(Ty);599  if (Ty == IntTy)600    return false;601 602  IRB.SetInsertPoint(&SI);603  Value *IntV = fatPtrsToInts(V, Ty, IntTy, V->getName());604  for (auto *Dbg : at::getDVRAssignmentMarkers(&SI))605    Dbg->setRawLocation(ValueAsMetadata::get(IntV));606 607  SI.setOperand(0, IntV);608  return true;609}610 611bool StoreFatPtrsAsIntsAndExpandMemcpyVisitor::visitMemCpyInst(612    MemCpyInst &MCI) {613  // TODO: Allow memcpy.p7.p3 as a synonym for the direct-to-LDS copy, which'll614  // need loop expansion here.615  if (MCI.getSourceAddressSpace() != AMDGPUAS::BUFFER_FAT_POINTER &&616      MCI.getDestAddressSpace() != AMDGPUAS::BUFFER_FAT_POINTER)617    return false;618  llvm::expandMemCpyAsLoop(&MCI,619                           TM->getTargetTransformInfo(*MCI.getFunction()));620  MCI.eraseFromParent();621  return true;622}623 624bool StoreFatPtrsAsIntsAndExpandMemcpyVisitor::visitMemMoveInst(625    MemMoveInst &MMI) {626  if (MMI.getSourceAddressSpace() != AMDGPUAS::BUFFER_FAT_POINTER &&627      MMI.getDestAddressSpace() != AMDGPUAS::BUFFER_FAT_POINTER)628    return false;629  reportFatalUsageError(630      "memmove() on buffer descriptors is not implemented because pointer "631      "comparison on buffer descriptors isn't implemented\n");632}633 634bool StoreFatPtrsAsIntsAndExpandMemcpyVisitor::visitMemSetInst(635    MemSetInst &MSI) {636  if (MSI.getDestAddressSpace() != AMDGPUAS::BUFFER_FAT_POINTER)637    return false;638  llvm::expandMemSetAsLoop(&MSI);639  MSI.eraseFromParent();640  return true;641}642 643bool StoreFatPtrsAsIntsAndExpandMemcpyVisitor::visitMemSetPatternInst(644    MemSetPatternInst &MSPI) {645  if (MSPI.getDestAddressSpace() != AMDGPUAS::BUFFER_FAT_POINTER)646    return false;647  llvm::expandMemSetPatternAsLoop(&MSPI);648  MSPI.eraseFromParent();649  return true;650}651 652namespace {653/// Convert loads/stores of types that the buffer intrinsics can't handle into654/// one ore more such loads/stores that consist of legal types.655///656/// Do this by657/// 1. Recursing into structs (and arrays that don't share a memory layout with658/// vectors) since the intrinsics can't handle complex types.659/// 2. Converting arrays of non-aggregate, byte-sized types into their660/// corresponding vectors661/// 3. Bitcasting unsupported types, namely overly-long scalars and byte662/// vectors, into vectors of supported types.663/// 4. Splitting up excessively long reads/writes into multiple operations.664///665/// Note that this doesn't handle complex data strucures, but, in the future,666/// the aggregate load splitter from SROA could be refactored to allow for that667/// case.668class LegalizeBufferContentTypesVisitor669    : public InstVisitor<LegalizeBufferContentTypesVisitor, bool> {670  friend class InstVisitor<LegalizeBufferContentTypesVisitor, bool>;671 672  IRBuilder<InstSimplifyFolder> IRB;673 674  const DataLayout &DL;675 676  /// If T is [N x U], where U is a scalar type, return the vector type677  /// <N x U>, otherwise, return T.678  Type *scalarArrayTypeAsVector(Type *MaybeArrayType);679  Value *arrayToVector(Value *V, Type *TargetType, const Twine &Name);680  Value *vectorToArray(Value *V, Type *OrigType, const Twine &Name);681 682  /// Break up the loads of a struct into the loads of its components683 684  /// Convert a vector or scalar type that can't be operated on by buffer685  /// intrinsics to one that would be legal through bitcasts and/or truncation.686  /// Uses the wider of i32, i16, or i8 where possible.687  Type *legalNonAggregateFor(Type *T);688  Value *makeLegalNonAggregate(Value *V, Type *TargetType, const Twine &Name);689  Value *makeIllegalNonAggregate(Value *V, Type *OrigType, const Twine &Name);690 691  struct VecSlice {692    uint64_t Index = 0;693    uint64_t Length = 0;694    VecSlice() = delete;695    // Needed for some Clangs696    VecSlice(uint64_t Index, uint64_t Length) : Index(Index), Length(Length) {}697  };698  /// Return the [index, length] pairs into which `T` needs to be cut to form699  /// legal buffer load or store operations. Clears `Slices`. Creates an empty700  /// `Slices` for non-vector inputs and creates one slice if no slicing will be701  /// needed.702  void getVecSlices(Type *T, SmallVectorImpl<VecSlice> &Slices);703 704  Value *extractSlice(Value *Vec, VecSlice S, const Twine &Name);705  Value *insertSlice(Value *Whole, Value *Part, VecSlice S, const Twine &Name);706 707  /// In most cases, return `LegalType`. However, when given an input that would708  /// normally be a legal type for the buffer intrinsics to return but that709  /// isn't hooked up through SelectionDAG, return a type of the same width that710  /// can be used with the relevant intrinsics. Specifically, handle the cases:711  /// - <1 x T> => T for all T712  /// - <N x i8> <=> i16, i32, 2xi32, 4xi32 (as needed)713  /// - <N x T> where T is under 32 bits and the total size is 96 bits <=> <3 x714  /// i32>715  Type *intrinsicTypeFor(Type *LegalType);716 717  bool visitLoadImpl(LoadInst &OrigLI, Type *PartType,718                     SmallVectorImpl<uint32_t> &AggIdxs, uint64_t AggByteOffset,719                     Value *&Result, const Twine &Name);720  /// Return value is (Changed, ModifiedInPlace)721  std::pair<bool, bool> visitStoreImpl(StoreInst &OrigSI, Type *PartType,722                                       SmallVectorImpl<uint32_t> &AggIdxs,723                                       uint64_t AggByteOffset,724                                       const Twine &Name);725 726  bool visitInstruction(Instruction &I) { return false; }727  bool visitLoadInst(LoadInst &LI);728  bool visitStoreInst(StoreInst &SI);729 730public:731  LegalizeBufferContentTypesVisitor(const DataLayout &DL, LLVMContext &Ctx)732      : IRB(Ctx, InstSimplifyFolder(DL)), DL(DL) {}733  bool processFunction(Function &F);734};735} // namespace736 737Type *LegalizeBufferContentTypesVisitor::scalarArrayTypeAsVector(Type *T) {738  ArrayType *AT = dyn_cast<ArrayType>(T);739  if (!AT)740    return T;741  Type *ET = AT->getElementType();742  if (!ET->isSingleValueType() || isa<VectorType>(ET))743    reportFatalUsageError("loading non-scalar arrays from buffer fat pointers "744                          "should have recursed");745  if (!DL.typeSizeEqualsStoreSize(AT))746    reportFatalUsageError(747        "loading padded arrays from buffer fat pinters should have recursed");748  return FixedVectorType::get(ET, AT->getNumElements());749}750 751Value *LegalizeBufferContentTypesVisitor::arrayToVector(Value *V,752                                                        Type *TargetType,753                                                        const Twine &Name) {754  Value *VectorRes = PoisonValue::get(TargetType);755  auto *VT = cast<FixedVectorType>(TargetType);756  unsigned EC = VT->getNumElements();757  for (auto I : iota_range<unsigned>(0, EC, /*Inclusive=*/false)) {758    Value *Elem = IRB.CreateExtractValue(V, I, Name + ".elem." + Twine(I));759    VectorRes = IRB.CreateInsertElement(VectorRes, Elem, I,760                                        Name + ".as.vec." + Twine(I));761  }762  return VectorRes;763}764 765Value *LegalizeBufferContentTypesVisitor::vectorToArray(Value *V,766                                                        Type *OrigType,767                                                        const Twine &Name) {768  Value *ArrayRes = PoisonValue::get(OrigType);769  ArrayType *AT = cast<ArrayType>(OrigType);770  unsigned EC = AT->getNumElements();771  for (auto I : iota_range<unsigned>(0, EC, /*Inclusive=*/false)) {772    Value *Elem = IRB.CreateExtractElement(V, I, Name + ".elem." + Twine(I));773    ArrayRes = IRB.CreateInsertValue(ArrayRes, Elem, I,774                                     Name + ".as.array." + Twine(I));775  }776  return ArrayRes;777}778 779Type *LegalizeBufferContentTypesVisitor::legalNonAggregateFor(Type *T) {780  TypeSize Size = DL.getTypeStoreSizeInBits(T);781  // Implicitly zero-extend to the next byte if needed782  if (!DL.typeSizeEqualsStoreSize(T))783    T = IRB.getIntNTy(Size.getFixedValue());784  Type *ElemTy = T->getScalarType();785  if (isa<PointerType, ScalableVectorType>(ElemTy)) {786    // Pointers are always big enough, and we'll let scalable vectors through to787    // fail in codegen.788    return T;789  }790  unsigned ElemSize = DL.getTypeSizeInBits(ElemTy).getFixedValue();791  if (isPowerOf2_32(ElemSize) && ElemSize >= 16 && ElemSize <= 128) {792    // [vectors of] anything that's 16/32/64/128 bits can be cast and split into793    // legal buffer operations.794    return T;795  }796  Type *BestVectorElemType = nullptr;797  if (Size.isKnownMultipleOf(32))798    BestVectorElemType = IRB.getInt32Ty();799  else if (Size.isKnownMultipleOf(16))800    BestVectorElemType = IRB.getInt16Ty();801  else802    BestVectorElemType = IRB.getInt8Ty();803  unsigned NumCastElems =804      Size.getFixedValue() / BestVectorElemType->getIntegerBitWidth();805  if (NumCastElems == 1)806    return BestVectorElemType;807  return FixedVectorType::get(BestVectorElemType, NumCastElems);808}809 810Value *LegalizeBufferContentTypesVisitor::makeLegalNonAggregate(811    Value *V, Type *TargetType, const Twine &Name) {812  Type *SourceType = V->getType();813  TypeSize SourceSize = DL.getTypeSizeInBits(SourceType);814  TypeSize TargetSize = DL.getTypeSizeInBits(TargetType);815  if (SourceSize != TargetSize) {816    Type *ShortScalarTy = IRB.getIntNTy(SourceSize.getFixedValue());817    Type *ByteScalarTy = IRB.getIntNTy(TargetSize.getFixedValue());818    Value *AsScalar = IRB.CreateBitCast(V, ShortScalarTy, Name + ".as.scalar");819    Value *Zext = IRB.CreateZExt(AsScalar, ByteScalarTy, Name + ".zext");820    V = Zext;821    SourceType = ByteScalarTy;822  }823  return IRB.CreateBitCast(V, TargetType, Name + ".legal");824}825 826Value *LegalizeBufferContentTypesVisitor::makeIllegalNonAggregate(827    Value *V, Type *OrigType, const Twine &Name) {828  Type *LegalType = V->getType();829  TypeSize LegalSize = DL.getTypeSizeInBits(LegalType);830  TypeSize OrigSize = DL.getTypeSizeInBits(OrigType);831  if (LegalSize != OrigSize) {832    Type *ShortScalarTy = IRB.getIntNTy(OrigSize.getFixedValue());833    Type *ByteScalarTy = IRB.getIntNTy(LegalSize.getFixedValue());834    Value *AsScalar = IRB.CreateBitCast(V, ByteScalarTy, Name + ".bytes.cast");835    Value *Trunc = IRB.CreateTrunc(AsScalar, ShortScalarTy, Name + ".trunc");836    return IRB.CreateBitCast(Trunc, OrigType, Name + ".orig");837  }838  return IRB.CreateBitCast(V, OrigType, Name + ".real.ty");839}840 841Type *LegalizeBufferContentTypesVisitor::intrinsicTypeFor(Type *LegalType) {842  auto *VT = dyn_cast<FixedVectorType>(LegalType);843  if (!VT)844    return LegalType;845  Type *ET = VT->getElementType();846  // Explicitly return the element type of 1-element vectors because the847  // underlying intrinsics don't like <1 x T> even though it's a synonym for T.848  if (VT->getNumElements() == 1)849    return ET;850  if (DL.getTypeSizeInBits(LegalType) == 96 && DL.getTypeSizeInBits(ET) < 32)851    return FixedVectorType::get(IRB.getInt32Ty(), 3);852  if (ET->isIntegerTy(8)) {853    switch (VT->getNumElements()) {854    default:855      return LegalType; // Let it crash later856    case 1:857      return IRB.getInt8Ty();858    case 2:859      return IRB.getInt16Ty();860    case 4:861      return IRB.getInt32Ty();862    case 8:863      return FixedVectorType::get(IRB.getInt32Ty(), 2);864    case 16:865      return FixedVectorType::get(IRB.getInt32Ty(), 4);866    }867  }868  return LegalType;869}870 871void LegalizeBufferContentTypesVisitor::getVecSlices(872    Type *T, SmallVectorImpl<VecSlice> &Slices) {873  Slices.clear();874  auto *VT = dyn_cast<FixedVectorType>(T);875  if (!VT)876    return;877 878  uint64_t ElemBitWidth =879      DL.getTypeSizeInBits(VT->getElementType()).getFixedValue();880 881  uint64_t ElemsPer4Words = 128 / ElemBitWidth;882  uint64_t ElemsPer2Words = ElemsPer4Words / 2;883  uint64_t ElemsPerWord = ElemsPer2Words / 2;884  uint64_t ElemsPerShort = ElemsPerWord / 2;885  uint64_t ElemsPerByte = ElemsPerShort / 2;886  // If the elements evenly pack into 32-bit words, we can use 3-word stores,887  // such as for <6 x bfloat> or <3 x i32>, but we can't dot his for, for888  // example, <3 x i64>, since that's not slicing.889  uint64_t ElemsPer3Words = ElemsPerWord * 3;890 891  uint64_t TotalElems = VT->getNumElements();892  uint64_t Index = 0;893  auto TrySlice = [&](unsigned MaybeLen) {894    if (MaybeLen > 0 && Index + MaybeLen <= TotalElems) {895      VecSlice Slice{/*Index=*/Index, /*Length=*/MaybeLen};896      Slices.push_back(Slice);897      Index += MaybeLen;898      return true;899    }900    return false;901  };902  while (Index < TotalElems) {903    TrySlice(ElemsPer4Words) || TrySlice(ElemsPer3Words) ||904        TrySlice(ElemsPer2Words) || TrySlice(ElemsPerWord) ||905        TrySlice(ElemsPerShort) || TrySlice(ElemsPerByte);906  }907}908 909Value *LegalizeBufferContentTypesVisitor::extractSlice(Value *Vec, VecSlice S,910                                                       const Twine &Name) {911  auto *VecVT = dyn_cast<FixedVectorType>(Vec->getType());912  if (!VecVT)913    return Vec;914  if (S.Length == VecVT->getNumElements() && S.Index == 0)915    return Vec;916  if (S.Length == 1)917    return IRB.CreateExtractElement(Vec, S.Index,918                                    Name + ".slice." + Twine(S.Index));919  SmallVector<int> Mask = llvm::to_vector(920      llvm::iota_range<int>(S.Index, S.Index + S.Length, /*Inclusive=*/false));921  return IRB.CreateShuffleVector(Vec, Mask, Name + ".slice." + Twine(S.Index));922}923 924Value *LegalizeBufferContentTypesVisitor::insertSlice(Value *Whole, Value *Part,925                                                      VecSlice S,926                                                      const Twine &Name) {927  auto *WholeVT = dyn_cast<FixedVectorType>(Whole->getType());928  if (!WholeVT)929    return Part;930  if (S.Length == WholeVT->getNumElements() && S.Index == 0)931    return Part;932  if (S.Length == 1) {933    return IRB.CreateInsertElement(Whole, Part, S.Index,934                                   Name + ".slice." + Twine(S.Index));935  }936  int NumElems = cast<FixedVectorType>(Whole->getType())->getNumElements();937 938  // Extend the slice with poisons to make the main shufflevector happy.939  SmallVector<int> ExtPartMask(NumElems, -1);940  for (auto [I, E] : llvm::enumerate(941           MutableArrayRef<int>(ExtPartMask).take_front(S.Length))) {942    E = I;943  }944  Value *ExtPart = IRB.CreateShuffleVector(Part, ExtPartMask,945                                           Name + ".ext." + Twine(S.Index));946 947  SmallVector<int> Mask =948      llvm::to_vector(llvm::iota_range<int>(0, NumElems, /*Inclusive=*/false));949  for (auto [I, E] :950       llvm::enumerate(MutableArrayRef<int>(Mask).slice(S.Index, S.Length)))951    E = I + NumElems;952  return IRB.CreateShuffleVector(Whole, ExtPart, Mask,953                                 Name + ".parts." + Twine(S.Index));954}955 956bool LegalizeBufferContentTypesVisitor::visitLoadImpl(957    LoadInst &OrigLI, Type *PartType, SmallVectorImpl<uint32_t> &AggIdxs,958    uint64_t AggByteOff, Value *&Result, const Twine &Name) {959  if (auto *ST = dyn_cast<StructType>(PartType)) {960    const StructLayout *Layout = DL.getStructLayout(ST);961    bool Changed = false;962    for (auto [I, ElemTy, Offset] :963         llvm::enumerate(ST->elements(), Layout->getMemberOffsets())) {964      AggIdxs.push_back(I);965      Changed |= visitLoadImpl(OrigLI, ElemTy, AggIdxs,966                               AggByteOff + Offset.getFixedValue(), Result,967                               Name + "." + Twine(I));968      AggIdxs.pop_back();969    }970    return Changed;971  }972  if (auto *AT = dyn_cast<ArrayType>(PartType)) {973    Type *ElemTy = AT->getElementType();974    if (!ElemTy->isSingleValueType() || !DL.typeSizeEqualsStoreSize(ElemTy) ||975        ElemTy->isVectorTy()) {976      TypeSize ElemStoreSize = DL.getTypeStoreSize(ElemTy);977      bool Changed = false;978      for (auto I : llvm::iota_range<uint32_t>(0, AT->getNumElements(),979                                               /*Inclusive=*/false)) {980        AggIdxs.push_back(I);981        Changed |= visitLoadImpl(OrigLI, ElemTy, AggIdxs,982                                 AggByteOff + I * ElemStoreSize.getFixedValue(),983                                 Result, Name + Twine(I));984        AggIdxs.pop_back();985      }986      return Changed;987    }988  }989 990  // Typical case991 992  Type *ArrayAsVecType = scalarArrayTypeAsVector(PartType);993  Type *LegalType = legalNonAggregateFor(ArrayAsVecType);994 995  SmallVector<VecSlice> Slices;996  getVecSlices(LegalType, Slices);997  bool HasSlices = Slices.size() > 1;998  bool IsAggPart = !AggIdxs.empty();999  Value *LoadsRes;1000  if (!HasSlices && !IsAggPart) {1001    Type *LoadableType = intrinsicTypeFor(LegalType);1002    if (LoadableType == PartType)1003      return false;1004 1005    IRB.SetInsertPoint(&OrigLI);1006    auto *NLI = cast<LoadInst>(OrigLI.clone());1007    NLI->mutateType(LoadableType);1008    NLI = IRB.Insert(NLI);1009    NLI->setName(Name + ".loadable");1010 1011    LoadsRes = IRB.CreateBitCast(NLI, LegalType, Name + ".from.loadable");1012  } else {1013    IRB.SetInsertPoint(&OrigLI);1014    LoadsRes = PoisonValue::get(LegalType);1015    Value *OrigPtr = OrigLI.getPointerOperand();1016    // If we're needing to spill something into more than one load, its legal1017    // type will be a vector (ex. an i256 load will have LegalType = <8 x i32>).1018    // But if we're already a scalar (which can happen if we're splitting up a1019    // struct), the element type will be the legal type itself.1020    Type *ElemType = LegalType->getScalarType();1021    unsigned ElemBytes = DL.getTypeStoreSize(ElemType);1022    AAMDNodes AANodes = OrigLI.getAAMetadata();1023    if (IsAggPart && Slices.empty())1024      Slices.push_back(VecSlice{/*Index=*/0, /*Length=*/1});1025    for (VecSlice S : Slices) {1026      Type *SliceType =1027          S.Length != 1 ? FixedVectorType::get(ElemType, S.Length) : ElemType;1028      int64_t ByteOffset = AggByteOff + S.Index * ElemBytes;1029      // You can't reasonably expect loads to wrap around the edge of memory.1030      Value *NewPtr = IRB.CreateGEP(1031          IRB.getInt8Ty(), OrigLI.getPointerOperand(), IRB.getInt32(ByteOffset),1032          OrigPtr->getName() + ".off.ptr." + Twine(ByteOffset),1033          GEPNoWrapFlags::noUnsignedWrap());1034      Type *LoadableType = intrinsicTypeFor(SliceType);1035      LoadInst *NewLI = IRB.CreateAlignedLoad(1036          LoadableType, NewPtr, commonAlignment(OrigLI.getAlign(), ByteOffset),1037          Name + ".off." + Twine(ByteOffset));1038      copyMetadataForLoad(*NewLI, OrigLI);1039      NewLI->setAAMetadata(1040          AANodes.adjustForAccess(ByteOffset, LoadableType, DL));1041      NewLI->setAtomic(OrigLI.getOrdering(), OrigLI.getSyncScopeID());1042      NewLI->setVolatile(OrigLI.isVolatile());1043      Value *Loaded = IRB.CreateBitCast(NewLI, SliceType,1044                                        NewLI->getName() + ".from.loadable");1045      LoadsRes = insertSlice(LoadsRes, Loaded, S, Name);1046    }1047  }1048  if (LegalType != ArrayAsVecType)1049    LoadsRes = makeIllegalNonAggregate(LoadsRes, ArrayAsVecType, Name);1050  if (ArrayAsVecType != PartType)1051    LoadsRes = vectorToArray(LoadsRes, PartType, Name);1052 1053  if (IsAggPart)1054    Result = IRB.CreateInsertValue(Result, LoadsRes, AggIdxs, Name);1055  else1056    Result = LoadsRes;1057  return true;1058}1059 1060bool LegalizeBufferContentTypesVisitor::visitLoadInst(LoadInst &LI) {1061  if (LI.getPointerAddressSpace() != AMDGPUAS::BUFFER_FAT_POINTER)1062    return false;1063 1064  SmallVector<uint32_t> AggIdxs;1065  Type *OrigType = LI.getType();1066  Value *Result = PoisonValue::get(OrigType);1067  bool Changed = visitLoadImpl(LI, OrigType, AggIdxs, 0, Result, LI.getName());1068  if (!Changed)1069    return false;1070  Result->takeName(&LI);1071  LI.replaceAllUsesWith(Result);1072  LI.eraseFromParent();1073  return Changed;1074}1075 1076std::pair<bool, bool> LegalizeBufferContentTypesVisitor::visitStoreImpl(1077    StoreInst &OrigSI, Type *PartType, SmallVectorImpl<uint32_t> &AggIdxs,1078    uint64_t AggByteOff, const Twine &Name) {1079  if (auto *ST = dyn_cast<StructType>(PartType)) {1080    const StructLayout *Layout = DL.getStructLayout(ST);1081    bool Changed = false;1082    for (auto [I, ElemTy, Offset] :1083         llvm::enumerate(ST->elements(), Layout->getMemberOffsets())) {1084      AggIdxs.push_back(I);1085      Changed |= std::get<0>(visitStoreImpl(OrigSI, ElemTy, AggIdxs,1086                                            AggByteOff + Offset.getFixedValue(),1087                                            Name + "." + Twine(I)));1088      AggIdxs.pop_back();1089    }1090    return std::make_pair(Changed, /*ModifiedInPlace=*/false);1091  }1092  if (auto *AT = dyn_cast<ArrayType>(PartType)) {1093    Type *ElemTy = AT->getElementType();1094    if (!ElemTy->isSingleValueType() || !DL.typeSizeEqualsStoreSize(ElemTy) ||1095        ElemTy->isVectorTy()) {1096      TypeSize ElemStoreSize = DL.getTypeStoreSize(ElemTy);1097      bool Changed = false;1098      for (auto I : llvm::iota_range<uint32_t>(0, AT->getNumElements(),1099                                               /*Inclusive=*/false)) {1100        AggIdxs.push_back(I);1101        Changed |= std::get<0>(visitStoreImpl(1102            OrigSI, ElemTy, AggIdxs,1103            AggByteOff + I * ElemStoreSize.getFixedValue(), Name + Twine(I)));1104        AggIdxs.pop_back();1105      }1106      return std::make_pair(Changed, /*ModifiedInPlace=*/false);1107    }1108  }1109 1110  Value *OrigData = OrigSI.getValueOperand();1111  Value *NewData = OrigData;1112 1113  bool IsAggPart = !AggIdxs.empty();1114  if (IsAggPart)1115    NewData = IRB.CreateExtractValue(NewData, AggIdxs, Name);1116 1117  Type *ArrayAsVecType = scalarArrayTypeAsVector(PartType);1118  if (ArrayAsVecType != PartType) {1119    NewData = arrayToVector(NewData, ArrayAsVecType, Name);1120  }1121 1122  Type *LegalType = legalNonAggregateFor(ArrayAsVecType);1123  if (LegalType != ArrayAsVecType) {1124    NewData = makeLegalNonAggregate(NewData, LegalType, Name);1125  }1126 1127  SmallVector<VecSlice> Slices;1128  getVecSlices(LegalType, Slices);1129  bool NeedToSplit = Slices.size() > 1 || IsAggPart;1130  if (!NeedToSplit) {1131    Type *StorableType = intrinsicTypeFor(LegalType);1132    if (StorableType == PartType)1133      return std::make_pair(/*Changed=*/false, /*ModifiedInPlace=*/false);1134    NewData = IRB.CreateBitCast(NewData, StorableType, Name + ".storable");1135    OrigSI.setOperand(0, NewData);1136    return std::make_pair(/*Changed=*/true, /*ModifiedInPlace=*/true);1137  }1138 1139  Value *OrigPtr = OrigSI.getPointerOperand();1140  Type *ElemType = LegalType->getScalarType();1141  if (IsAggPart && Slices.empty())1142    Slices.push_back(VecSlice{/*Index=*/0, /*Length=*/1});1143  unsigned ElemBytes = DL.getTypeStoreSize(ElemType);1144  AAMDNodes AANodes = OrigSI.getAAMetadata();1145  for (VecSlice S : Slices) {1146    Type *SliceType =1147        S.Length != 1 ? FixedVectorType::get(ElemType, S.Length) : ElemType;1148    int64_t ByteOffset = AggByteOff + S.Index * ElemBytes;1149    Value *NewPtr =1150        IRB.CreateGEP(IRB.getInt8Ty(), OrigPtr, IRB.getInt32(ByteOffset),1151                      OrigPtr->getName() + ".part." + Twine(S.Index),1152                      GEPNoWrapFlags::noUnsignedWrap());1153    Value *DataSlice = extractSlice(NewData, S, Name);1154    Type *StorableType = intrinsicTypeFor(SliceType);1155    DataSlice = IRB.CreateBitCast(DataSlice, StorableType,1156                                  DataSlice->getName() + ".storable");1157    auto *NewSI = cast<StoreInst>(OrigSI.clone());1158    NewSI->setAlignment(commonAlignment(OrigSI.getAlign(), ByteOffset));1159    IRB.Insert(NewSI);1160    NewSI->setOperand(0, DataSlice);1161    NewSI->setOperand(1, NewPtr);1162    NewSI->setAAMetadata(AANodes.adjustForAccess(ByteOffset, StorableType, DL));1163  }1164  return std::make_pair(/*Changed=*/true, /*ModifiedInPlace=*/false);1165}1166 1167bool LegalizeBufferContentTypesVisitor::visitStoreInst(StoreInst &SI) {1168  if (SI.getPointerAddressSpace() != AMDGPUAS::BUFFER_FAT_POINTER)1169    return false;1170  IRB.SetInsertPoint(&SI);1171  SmallVector<uint32_t> AggIdxs;1172  Value *OrigData = SI.getValueOperand();1173  auto [Changed, ModifiedInPlace] =1174      visitStoreImpl(SI, OrigData->getType(), AggIdxs, 0, OrigData->getName());1175  if (Changed && !ModifiedInPlace)1176    SI.eraseFromParent();1177  return Changed;1178}1179 1180bool LegalizeBufferContentTypesVisitor::processFunction(Function &F) {1181  bool Changed = false;1182  // Note, memory transfer intrinsics won't1183  for (Instruction &I : make_early_inc_range(instructions(F))) {1184    Changed |= visit(I);1185  }1186  return Changed;1187}1188 1189/// Return the ptr addrspace(8) and i32 (resource and offset parts) in a lowered1190/// buffer fat pointer constant.1191static std::pair<Constant *, Constant *>1192splitLoweredFatBufferConst(Constant *C) {1193  assert(isSplitFatPtr(C->getType()) && "Not a split fat buffer pointer");1194  return std::make_pair(C->getAggregateElement(0u), C->getAggregateElement(1u));1195}1196 1197namespace {1198/// Handle the remapping of ptr addrspace(7) constants.1199class FatPtrConstMaterializer final : public ValueMaterializer {1200  BufferFatPtrToStructTypeMap *TypeMap;1201  // An internal mapper that is used to recurse into the arguments of constants.1202  // While the documentation for `ValueMapper` specifies not to use it1203  // recursively, examination of the logic in mapValue() shows that it can1204  // safely be used recursively when handling constants, like it does in its own1205  // logic.1206  ValueMapper InternalMapper;1207 1208  Constant *materializeBufferFatPtrConst(Constant *C);1209 1210public:1211  // UnderlyingMap is the value map this materializer will be filling.1212  FatPtrConstMaterializer(BufferFatPtrToStructTypeMap *TypeMap,1213                          ValueToValueMapTy &UnderlyingMap)1214      : TypeMap(TypeMap),1215        InternalMapper(UnderlyingMap, RF_None, TypeMap, this) {}1216  ~FatPtrConstMaterializer() = default;1217 1218  Value *materialize(Value *V) override;1219};1220} // namespace1221 1222Constant *FatPtrConstMaterializer::materializeBufferFatPtrConst(Constant *C) {1223  Type *SrcTy = C->getType();1224  auto *NewTy = dyn_cast<StructType>(TypeMap->remapType(SrcTy));1225  if (C->isNullValue())1226    return ConstantAggregateZero::getNullValue(NewTy);1227  if (isa<PoisonValue>(C)) {1228    return ConstantStruct::get(NewTy,1229                               {PoisonValue::get(NewTy->getElementType(0)),1230                                PoisonValue::get(NewTy->getElementType(1))});1231  }1232  if (isa<UndefValue>(C)) {1233    return ConstantStruct::get(NewTy,1234                               {UndefValue::get(NewTy->getElementType(0)),1235                                UndefValue::get(NewTy->getElementType(1))});1236  }1237 1238  if (auto *VC = dyn_cast<ConstantVector>(C)) {1239    if (Constant *S = VC->getSplatValue()) {1240      Constant *NewS = InternalMapper.mapConstant(*S);1241      if (!NewS)1242        return nullptr;1243      auto [Rsrc, Off] = splitLoweredFatBufferConst(NewS);1244      auto EC = VC->getType()->getElementCount();1245      return ConstantStruct::get(NewTy, {ConstantVector::getSplat(EC, Rsrc),1246                                         ConstantVector::getSplat(EC, Off)});1247    }1248    SmallVector<Constant *> Rsrcs;1249    SmallVector<Constant *> Offs;1250    for (Value *Op : VC->operand_values()) {1251      auto *NewOp = dyn_cast_or_null<Constant>(InternalMapper.mapValue(*Op));1252      if (!NewOp)1253        return nullptr;1254      auto [Rsrc, Off] = splitLoweredFatBufferConst(NewOp);1255      Rsrcs.push_back(Rsrc);1256      Offs.push_back(Off);1257    }1258    Constant *RsrcVec = ConstantVector::get(Rsrcs);1259    Constant *OffVec = ConstantVector::get(Offs);1260    return ConstantStruct::get(NewTy, {RsrcVec, OffVec});1261  }1262 1263  if (isa<GlobalValue>(C))1264    reportFatalUsageError("global values containing ptr addrspace(7) (buffer "1265                          "fat pointer) values are not supported");1266 1267  if (isa<ConstantExpr>(C))1268    reportFatalUsageError(1269        "constant exprs containing ptr addrspace(7) (buffer "1270        "fat pointer) values should have been expanded earlier");1271 1272  return nullptr;1273}1274 1275Value *FatPtrConstMaterializer::materialize(Value *V) {1276  Constant *C = dyn_cast<Constant>(V);1277  if (!C)1278    return nullptr;1279  // Structs and other types that happen to contain fat pointers get remapped1280  // by the mapValue() logic.1281  if (!isBufferFatPtrConst(C))1282    return nullptr;1283  return materializeBufferFatPtrConst(C);1284}1285 1286using PtrParts = std::pair<Value *, Value *>;1287namespace {1288// The visitor returns the resource and offset parts for an instruction if they1289// can be computed, or (nullptr, nullptr) for cases that don't have a meaningful1290// value mapping.1291class SplitPtrStructs : public InstVisitor<SplitPtrStructs, PtrParts> {1292  ValueToValueMapTy RsrcParts;1293  ValueToValueMapTy OffParts;1294 1295  // Track instructions that have been rewritten into a user of the component1296  // parts of their ptr addrspace(7) input. Instructions that produced1297  // ptr addrspace(7) parts should **not** be RAUW'd before being added to this1298  // set, as that replacement will be handled in a post-visit step. However,1299  // instructions that yield values that aren't fat pointers (ex. ptrtoint)1300  // should RAUW themselves with new instructions that use the split parts1301  // of their arguments during processing.1302  DenseSet<Instruction *> SplitUsers;1303 1304  // Nodes that need a second look once we've computed the parts for all other1305  // instructions to see if, for example, we really need to phi on the resource1306  // part.1307  SmallVector<Instruction *> Conditionals;1308  // Temporary instructions produced while lowering conditionals that should be1309  // killed.1310  SmallVector<Instruction *> ConditionalTemps;1311 1312  // Subtarget info, needed for determining what cache control bits to set.1313  const TargetMachine *TM;1314  const GCNSubtarget *ST = nullptr;1315 1316  IRBuilder<InstSimplifyFolder> IRB;1317 1318  // Copy metadata between instructions if applicable.1319  void copyMetadata(Value *Dest, Value *Src);1320 1321  // Get the resource and offset parts of the value V, inserting appropriate1322  // extractvalue calls if needed.1323  PtrParts getPtrParts(Value *V);1324 1325  // Given an instruction that could produce multiple resource parts (a PHI or1326  // select), collect the set of possible instructions that could have provided1327  // its resource parts  that it could have (the `Roots`) and the set of1328  // conditional instructions visited during the search (`Seen`). If, after1329  // removing the root of the search from `Seen` and `Roots`, `Seen` is a subset1330  // of `Roots` and `Roots - Seen` contains one element, the resource part of1331  // that element can replace the resource part of all other elements in `Seen`.1332  void getPossibleRsrcRoots(Instruction *I, SmallPtrSetImpl<Value *> &Roots,1333                            SmallPtrSetImpl<Value *> &Seen);1334  void processConditionals();1335 1336  // If an instruction hav been split into resource and offset parts,1337  // delete that instruction. If any of its uses have not themselves been split1338  // into parts (for example, an insertvalue), construct the structure1339  // that the type rewrites declared should be produced by the dying instruction1340  // and use that.1341  // Also, kill the temporary extractvalue operations produced by the two-stage1342  // lowering of PHIs and conditionals.1343  void killAndReplaceSplitInstructions(SmallVectorImpl<Instruction *> &Origs);1344 1345  void setAlign(CallInst *Intr, Align A, unsigned RsrcArgIdx);1346  void insertPreMemOpFence(AtomicOrdering Order, SyncScope::ID SSID);1347  void insertPostMemOpFence(AtomicOrdering Order, SyncScope::ID SSID);1348  Value *handleMemoryInst(Instruction *I, Value *Arg, Value *Ptr, Type *Ty,1349                          Align Alignment, AtomicOrdering Order,1350                          bool IsVolatile, SyncScope::ID SSID);1351 1352public:1353  SplitPtrStructs(const DataLayout &DL, LLVMContext &Ctx,1354                  const TargetMachine *TM)1355      : TM(TM), IRB(Ctx, InstSimplifyFolder(DL)) {}1356 1357  void processFunction(Function &F);1358 1359  PtrParts visitInstruction(Instruction &I);1360  PtrParts visitLoadInst(LoadInst &LI);1361  PtrParts visitStoreInst(StoreInst &SI);1362  PtrParts visitAtomicRMWInst(AtomicRMWInst &AI);1363  PtrParts visitAtomicCmpXchgInst(AtomicCmpXchgInst &AI);1364  PtrParts visitGetElementPtrInst(GetElementPtrInst &GEP);1365 1366  PtrParts visitPtrToAddrInst(PtrToAddrInst &PA);1367  PtrParts visitPtrToIntInst(PtrToIntInst &PI);1368  PtrParts visitIntToPtrInst(IntToPtrInst &IP);1369  PtrParts visitAddrSpaceCastInst(AddrSpaceCastInst &I);1370  PtrParts visitICmpInst(ICmpInst &Cmp);1371  PtrParts visitFreezeInst(FreezeInst &I);1372 1373  PtrParts visitExtractElementInst(ExtractElementInst &I);1374  PtrParts visitInsertElementInst(InsertElementInst &I);1375  PtrParts visitShuffleVectorInst(ShuffleVectorInst &I);1376 1377  PtrParts visitPHINode(PHINode &PHI);1378  PtrParts visitSelectInst(SelectInst &SI);1379 1380  PtrParts visitIntrinsicInst(IntrinsicInst &II);1381};1382} // namespace1383 1384void SplitPtrStructs::copyMetadata(Value *Dest, Value *Src) {1385  auto *DestI = dyn_cast<Instruction>(Dest);1386  auto *SrcI = dyn_cast<Instruction>(Src);1387 1388  if (!DestI || !SrcI)1389    return;1390 1391  DestI->copyMetadata(*SrcI);1392}1393 1394PtrParts SplitPtrStructs::getPtrParts(Value *V) {1395  assert(isSplitFatPtr(V->getType()) && "it's not meaningful to get the parts "1396                                        "of something that wasn't rewritten");1397  auto *RsrcEntry = &RsrcParts[V];1398  auto *OffEntry = &OffParts[V];1399  if (*RsrcEntry && *OffEntry)1400    return {*RsrcEntry, *OffEntry};1401 1402  if (auto *C = dyn_cast<Constant>(V)) {1403    auto [Rsrc, Off] = splitLoweredFatBufferConst(C);1404    return {*RsrcEntry = Rsrc, *OffEntry = Off};1405  }1406 1407  IRBuilder<InstSimplifyFolder>::InsertPointGuard Guard(IRB);1408  if (auto *I = dyn_cast<Instruction>(V)) {1409    LLVM_DEBUG(dbgs() << "Recursing to split parts of " << *I << "\n");1410    auto [Rsrc, Off] = visit(*I);1411    if (Rsrc && Off)1412      return {*RsrcEntry = Rsrc, *OffEntry = Off};1413    // We'll be creating the new values after the relevant instruction.1414    // This instruction generates a value and so isn't a terminator.1415    IRB.SetInsertPoint(*I->getInsertionPointAfterDef());1416    IRB.SetCurrentDebugLocation(I->getDebugLoc());1417  } else if (auto *A = dyn_cast<Argument>(V)) {1418    IRB.SetInsertPointPastAllocas(A->getParent());1419    IRB.SetCurrentDebugLocation(DebugLoc());1420  }1421  Value *Rsrc = IRB.CreateExtractValue(V, 0, V->getName() + ".rsrc");1422  Value *Off = IRB.CreateExtractValue(V, 1, V->getName() + ".off");1423  return {*RsrcEntry = Rsrc, *OffEntry = Off};1424}1425 1426/// Returns the instruction that defines the resource part of the value V.1427/// Note that this is not getUnderlyingObject(), since that looks through1428/// operations like ptrmask which might modify the resource part.1429///1430/// We can limit ourselves to just looking through GEPs followed by looking1431/// through addrspacecasts because only those two operations preserve the1432/// resource part, and because operations on an `addrspace(8)` (which is the1433/// legal input to this addrspacecast) would produce a different resource part.1434static Value *rsrcPartRoot(Value *V) {1435  while (auto *GEP = dyn_cast<GEPOperator>(V))1436    V = GEP->getPointerOperand();1437  while (auto *ASC = dyn_cast<AddrSpaceCastOperator>(V))1438    V = ASC->getPointerOperand();1439  return V;1440}1441 1442void SplitPtrStructs::getPossibleRsrcRoots(Instruction *I,1443                                           SmallPtrSetImpl<Value *> &Roots,1444                                           SmallPtrSetImpl<Value *> &Seen) {1445  if (auto *PHI = dyn_cast<PHINode>(I)) {1446    if (!Seen.insert(I).second)1447      return;1448    for (Value *In : PHI->incoming_values()) {1449      In = rsrcPartRoot(In);1450      Roots.insert(In);1451      if (isa<PHINode, SelectInst>(In))1452        getPossibleRsrcRoots(cast<Instruction>(In), Roots, Seen);1453    }1454  } else if (auto *SI = dyn_cast<SelectInst>(I)) {1455    if (!Seen.insert(SI).second)1456      return;1457    Value *TrueVal = rsrcPartRoot(SI->getTrueValue());1458    Value *FalseVal = rsrcPartRoot(SI->getFalseValue());1459    Roots.insert(TrueVal);1460    Roots.insert(FalseVal);1461    if (isa<PHINode, SelectInst>(TrueVal))1462      getPossibleRsrcRoots(cast<Instruction>(TrueVal), Roots, Seen);1463    if (isa<PHINode, SelectInst>(FalseVal))1464      getPossibleRsrcRoots(cast<Instruction>(FalseVal), Roots, Seen);1465  } else {1466    llvm_unreachable("getPossibleRsrcParts() only works on phi and select");1467  }1468}1469 1470void SplitPtrStructs::processConditionals() {1471  SmallDenseMap<Value *, Value *> FoundRsrcs;1472  SmallPtrSet<Value *, 4> Roots;1473  SmallPtrSet<Value *, 4> Seen;1474  for (Instruction *I : Conditionals) {1475    // These have to exist by now because we've visited these nodes.1476    Value *Rsrc = RsrcParts[I];1477    Value *Off = OffParts[I];1478    assert(Rsrc && Off && "must have visited conditionals by now");1479 1480    std::optional<Value *> MaybeRsrc;1481    auto MaybeFoundRsrc = FoundRsrcs.find(I);1482    if (MaybeFoundRsrc != FoundRsrcs.end()) {1483      MaybeRsrc = MaybeFoundRsrc->second;1484    } else {1485      IRBuilder<InstSimplifyFolder>::InsertPointGuard Guard(IRB);1486      Roots.clear();1487      Seen.clear();1488      getPossibleRsrcRoots(I, Roots, Seen);1489      LLVM_DEBUG(dbgs() << "Processing conditional: " << *I << "\n");1490#ifndef NDEBUG1491      for (Value *V : Roots)1492        LLVM_DEBUG(dbgs() << "Root: " << *V << "\n");1493      for (Value *V : Seen)1494        LLVM_DEBUG(dbgs() << "Seen: " << *V << "\n");1495#endif1496      // If we are our own possible root, then we shouldn't block our1497      // replacement with a valid incoming value.1498      Roots.erase(I);1499      // We don't want to block the optimization for conditionals that don't1500      // refer to themselves but did see themselves during the traversal.1501      Seen.erase(I);1502 1503      if (set_is_subset(Seen, Roots)) {1504        auto Diff = set_difference(Roots, Seen);1505        if (Diff.size() == 1) {1506          Value *RootVal = *Diff.begin();1507          // Handle the case where previous loops already looked through1508          // an addrspacecast.1509          if (isSplitFatPtr(RootVal->getType()))1510            MaybeRsrc = std::get<0>(getPtrParts(RootVal));1511          else1512            MaybeRsrc = RootVal;1513        }1514      }1515    }1516 1517    if (auto *PHI = dyn_cast<PHINode>(I)) {1518      Value *NewRsrc;1519      StructType *PHITy = cast<StructType>(PHI->getType());1520      IRB.SetInsertPoint(*PHI->getInsertionPointAfterDef());1521      IRB.SetCurrentDebugLocation(PHI->getDebugLoc());1522      if (MaybeRsrc) {1523        NewRsrc = *MaybeRsrc;1524      } else {1525        Type *RsrcTy = PHITy->getElementType(0);1526        auto *RsrcPHI = IRB.CreatePHI(RsrcTy, PHI->getNumIncomingValues());1527        RsrcPHI->takeName(Rsrc);1528        for (auto [V, BB] : llvm::zip(PHI->incoming_values(), PHI->blocks())) {1529          Value *VRsrc = std::get<0>(getPtrParts(V));1530          RsrcPHI->addIncoming(VRsrc, BB);1531        }1532        copyMetadata(RsrcPHI, PHI);1533        NewRsrc = RsrcPHI;1534      }1535 1536      Type *OffTy = PHITy->getElementType(1);1537      auto *NewOff = IRB.CreatePHI(OffTy, PHI->getNumIncomingValues());1538      NewOff->takeName(Off);1539      for (auto [V, BB] : llvm::zip(PHI->incoming_values(), PHI->blocks())) {1540        assert(OffParts.count(V) && "An offset part had to be created by now");1541        Value *VOff = std::get<1>(getPtrParts(V));1542        NewOff->addIncoming(VOff, BB);1543      }1544      copyMetadata(NewOff, PHI);1545 1546      // Note: We don't eraseFromParent() the temporaries because we don't want1547      // to put the corrections maps in an inconstent state. That'll be handed1548      // during the rest of the killing. Also, `ValueToValueMapTy` guarantees1549      // that references in that map will be updated as well.1550      // Note that if the temporary instruction got `InstSimplify`'d away, it1551      // might be something like a block argument.1552      if (auto *RsrcInst = dyn_cast<Instruction>(Rsrc)) {1553        ConditionalTemps.push_back(RsrcInst);1554        RsrcInst->replaceAllUsesWith(NewRsrc);1555      }1556      if (auto *OffInst = dyn_cast<Instruction>(Off)) {1557        ConditionalTemps.push_back(OffInst);1558        OffInst->replaceAllUsesWith(NewOff);1559      }1560 1561      // Save on recomputing the cycle traversals in known-root cases.1562      if (MaybeRsrc)1563        for (Value *V : Seen)1564          FoundRsrcs[V] = NewRsrc;1565    } else if (isa<SelectInst>(I)) {1566      if (MaybeRsrc) {1567        if (auto *RsrcInst = dyn_cast<Instruction>(Rsrc)) {1568          // Guard against conditionals that were already folded away.1569          if (RsrcInst != *MaybeRsrc) {1570            ConditionalTemps.push_back(RsrcInst);1571            RsrcInst->replaceAllUsesWith(*MaybeRsrc);1572          }1573        }1574        for (Value *V : Seen)1575          FoundRsrcs[V] = *MaybeRsrc;1576      }1577    } else {1578      llvm_unreachable("Only PHIs and selects go in the conditionals list");1579    }1580  }1581}1582 1583void SplitPtrStructs::killAndReplaceSplitInstructions(1584    SmallVectorImpl<Instruction *> &Origs) {1585  for (Instruction *I : ConditionalTemps)1586    I->eraseFromParent();1587 1588  for (Instruction *I : Origs) {1589    if (!SplitUsers.contains(I))1590      continue;1591 1592    SmallVector<DbgVariableRecord *> Dbgs;1593    findDbgValues(I, Dbgs);1594    for (DbgVariableRecord *Dbg : Dbgs) {1595      auto &DL = I->getDataLayout();1596      assert(isSplitFatPtr(I->getType()) &&1597             "We should've RAUW'd away loads, stores, etc. at this point");1598      DbgVariableRecord *OffDbg = Dbg->clone();1599      auto [Rsrc, Off] = getPtrParts(I);1600 1601      int64_t RsrcSz = DL.getTypeSizeInBits(Rsrc->getType());1602      int64_t OffSz = DL.getTypeSizeInBits(Off->getType());1603 1604      std::optional<DIExpression *> RsrcExpr =1605          DIExpression::createFragmentExpression(Dbg->getExpression(), 0,1606                                                 RsrcSz);1607      std::optional<DIExpression *> OffExpr =1608          DIExpression::createFragmentExpression(Dbg->getExpression(), RsrcSz,1609                                                 OffSz);1610      if (OffExpr) {1611        OffDbg->setExpression(*OffExpr);1612        OffDbg->replaceVariableLocationOp(I, Off);1613        OffDbg->insertBefore(Dbg);1614      } else {1615        OffDbg->eraseFromParent();1616      }1617      if (RsrcExpr) {1618        Dbg->setExpression(*RsrcExpr);1619        Dbg->replaceVariableLocationOp(I, Rsrc);1620      } else {1621        Dbg->replaceVariableLocationOp(I, PoisonValue::get(I->getType()));1622      }1623    }1624 1625    Value *Poison = PoisonValue::get(I->getType());1626    I->replaceUsesWithIf(Poison, [&](const Use &U) -> bool {1627      if (const auto *UI = dyn_cast<Instruction>(U.getUser()))1628        return SplitUsers.contains(UI);1629      return false;1630    });1631 1632    if (I->use_empty()) {1633      I->eraseFromParent();1634      continue;1635    }1636    IRB.SetInsertPoint(*I->getInsertionPointAfterDef());1637    IRB.SetCurrentDebugLocation(I->getDebugLoc());1638    auto [Rsrc, Off] = getPtrParts(I);1639    Value *Struct = PoisonValue::get(I->getType());1640    Struct = IRB.CreateInsertValue(Struct, Rsrc, 0);1641    Struct = IRB.CreateInsertValue(Struct, Off, 1);1642    copyMetadata(Struct, I);1643    Struct->takeName(I);1644    I->replaceAllUsesWith(Struct);1645    I->eraseFromParent();1646  }1647}1648 1649void SplitPtrStructs::setAlign(CallInst *Intr, Align A, unsigned RsrcArgIdx) {1650  LLVMContext &Ctx = Intr->getContext();1651  Intr->addParamAttr(RsrcArgIdx, Attribute::getWithAlignment(Ctx, A));1652}1653 1654void SplitPtrStructs::insertPreMemOpFence(AtomicOrdering Order,1655                                          SyncScope::ID SSID) {1656  switch (Order) {1657  case AtomicOrdering::Release:1658  case AtomicOrdering::AcquireRelease:1659  case AtomicOrdering::SequentiallyConsistent:1660    IRB.CreateFence(AtomicOrdering::Release, SSID);1661    break;1662  default:1663    break;1664  }1665}1666 1667void SplitPtrStructs::insertPostMemOpFence(AtomicOrdering Order,1668                                           SyncScope::ID SSID) {1669  switch (Order) {1670  case AtomicOrdering::Acquire:1671  case AtomicOrdering::AcquireRelease:1672  case AtomicOrdering::SequentiallyConsistent:1673    IRB.CreateFence(AtomicOrdering::Acquire, SSID);1674    break;1675  default:1676    break;1677  }1678}1679 1680Value *SplitPtrStructs::handleMemoryInst(Instruction *I, Value *Arg, Value *Ptr,1681                                         Type *Ty, Align Alignment,1682                                         AtomicOrdering Order, bool IsVolatile,1683                                         SyncScope::ID SSID) {1684  IRB.SetInsertPoint(I);1685 1686  auto [Rsrc, Off] = getPtrParts(Ptr);1687  SmallVector<Value *, 5> Args;1688  if (Arg)1689    Args.push_back(Arg);1690  Args.push_back(Rsrc);1691  Args.push_back(Off);1692  insertPreMemOpFence(Order, SSID);1693  // soffset is always 0 for these cases, where we always want any offset to be1694  // part of bounds checking and we don't know which parts of the GEPs is1695  // uniform.1696  Args.push_back(IRB.getInt32(0));1697 1698  uint32_t Aux = 0;1699  if (IsVolatile)1700    Aux |= AMDGPU::CPol::VOLATILE;1701  Args.push_back(IRB.getInt32(Aux));1702 1703  Intrinsic::ID IID = Intrinsic::not_intrinsic;1704  if (isa<LoadInst>(I))1705    IID = Order == AtomicOrdering::NotAtomic1706              ? Intrinsic::amdgcn_raw_ptr_buffer_load1707              : Intrinsic::amdgcn_raw_ptr_atomic_buffer_load;1708  else if (isa<StoreInst>(I))1709    IID = Intrinsic::amdgcn_raw_ptr_buffer_store;1710  else if (auto *RMW = dyn_cast<AtomicRMWInst>(I)) {1711    switch (RMW->getOperation()) {1712    case AtomicRMWInst::Xchg:1713      IID = Intrinsic::amdgcn_raw_ptr_buffer_atomic_swap;1714      break;1715    case AtomicRMWInst::Add:1716      IID = Intrinsic::amdgcn_raw_ptr_buffer_atomic_add;1717      break;1718    case AtomicRMWInst::Sub:1719      IID = Intrinsic::amdgcn_raw_ptr_buffer_atomic_sub;1720      break;1721    case AtomicRMWInst::And:1722      IID = Intrinsic::amdgcn_raw_ptr_buffer_atomic_and;1723      break;1724    case AtomicRMWInst::Or:1725      IID = Intrinsic::amdgcn_raw_ptr_buffer_atomic_or;1726      break;1727    case AtomicRMWInst::Xor:1728      IID = Intrinsic::amdgcn_raw_ptr_buffer_atomic_xor;1729      break;1730    case AtomicRMWInst::Max:1731      IID = Intrinsic::amdgcn_raw_ptr_buffer_atomic_smax;1732      break;1733    case AtomicRMWInst::Min:1734      IID = Intrinsic::amdgcn_raw_ptr_buffer_atomic_smin;1735      break;1736    case AtomicRMWInst::UMax:1737      IID = Intrinsic::amdgcn_raw_ptr_buffer_atomic_umax;1738      break;1739    case AtomicRMWInst::UMin:1740      IID = Intrinsic::amdgcn_raw_ptr_buffer_atomic_umin;1741      break;1742    case AtomicRMWInst::FAdd:1743      IID = Intrinsic::amdgcn_raw_ptr_buffer_atomic_fadd;1744      break;1745    case AtomicRMWInst::FMax:1746      IID = Intrinsic::amdgcn_raw_ptr_buffer_atomic_fmax;1747      break;1748    case AtomicRMWInst::FMin:1749      IID = Intrinsic::amdgcn_raw_ptr_buffer_atomic_fmin;1750      break;1751    case AtomicRMWInst::FSub: {1752      reportFatalUsageError(1753          "atomic floating point subtraction not supported for "1754          "buffer resources and should've been expanded away");1755      break;1756    }1757    case AtomicRMWInst::FMaximum: {1758      reportFatalUsageError(1759          "atomic floating point fmaximum not supported for "1760          "buffer resources and should've been expanded away");1761      break;1762    }1763    case AtomicRMWInst::FMinimum: {1764      reportFatalUsageError(1765          "atomic floating point fminimum not supported for "1766          "buffer resources and should've been expanded away");1767      break;1768    }1769    case AtomicRMWInst::Nand:1770      reportFatalUsageError(1771          "atomic nand not supported for buffer resources and "1772          "should've been expanded away");1773      break;1774    case AtomicRMWInst::UIncWrap:1775    case AtomicRMWInst::UDecWrap:1776      reportFatalUsageError("wrapping increment/decrement not supported for "1777                            "buffer resources and should've ben expanded away");1778      break;1779    case AtomicRMWInst::BAD_BINOP:1780      llvm_unreachable("Not sure how we got a bad binop");1781    case AtomicRMWInst::USubCond:1782    case AtomicRMWInst::USubSat:1783      break;1784    }1785  }1786 1787  auto *Call = IRB.CreateIntrinsic(IID, Ty, Args);1788  copyMetadata(Call, I);1789  setAlign(Call, Alignment, Arg ? 1 : 0);1790  Call->takeName(I);1791 1792  insertPostMemOpFence(Order, SSID);1793  // The "no moving p7 directly" rewrites ensure that this load or store won't1794  // itself need to be split into parts.1795  SplitUsers.insert(I);1796  I->replaceAllUsesWith(Call);1797  return Call;1798}1799 1800PtrParts SplitPtrStructs::visitInstruction(Instruction &I) {1801  return {nullptr, nullptr};1802}1803 1804PtrParts SplitPtrStructs::visitLoadInst(LoadInst &LI) {1805  if (!isSplitFatPtr(LI.getPointerOperandType()))1806    return {nullptr, nullptr};1807  handleMemoryInst(&LI, nullptr, LI.getPointerOperand(), LI.getType(),1808                   LI.getAlign(), LI.getOrdering(), LI.isVolatile(),1809                   LI.getSyncScopeID());1810  return {nullptr, nullptr};1811}1812 1813PtrParts SplitPtrStructs::visitStoreInst(StoreInst &SI) {1814  if (!isSplitFatPtr(SI.getPointerOperandType()))1815    return {nullptr, nullptr};1816  Value *Arg = SI.getValueOperand();1817  handleMemoryInst(&SI, Arg, SI.getPointerOperand(), Arg->getType(),1818                   SI.getAlign(), SI.getOrdering(), SI.isVolatile(),1819                   SI.getSyncScopeID());1820  return {nullptr, nullptr};1821}1822 1823PtrParts SplitPtrStructs::visitAtomicRMWInst(AtomicRMWInst &AI) {1824  if (!isSplitFatPtr(AI.getPointerOperand()->getType()))1825    return {nullptr, nullptr};1826  Value *Arg = AI.getValOperand();1827  handleMemoryInst(&AI, Arg, AI.getPointerOperand(), Arg->getType(),1828                   AI.getAlign(), AI.getOrdering(), AI.isVolatile(),1829                   AI.getSyncScopeID());1830  return {nullptr, nullptr};1831}1832 1833// Unlike load, store, and RMW, cmpxchg needs special handling to account1834// for the boolean argument.1835PtrParts SplitPtrStructs::visitAtomicCmpXchgInst(AtomicCmpXchgInst &AI) {1836  Value *Ptr = AI.getPointerOperand();1837  if (!isSplitFatPtr(Ptr->getType()))1838    return {nullptr, nullptr};1839  IRB.SetInsertPoint(&AI);1840 1841  Type *Ty = AI.getNewValOperand()->getType();1842  AtomicOrdering Order = AI.getMergedOrdering();1843  SyncScope::ID SSID = AI.getSyncScopeID();1844  bool IsNonTemporal = AI.getMetadata(LLVMContext::MD_nontemporal);1845 1846  auto [Rsrc, Off] = getPtrParts(Ptr);1847  insertPreMemOpFence(Order, SSID);1848 1849  uint32_t Aux = 0;1850  if (IsNonTemporal)1851    Aux |= AMDGPU::CPol::SLC;1852  if (AI.isVolatile())1853    Aux |= AMDGPU::CPol::VOLATILE;1854  auto *Call =1855      IRB.CreateIntrinsic(Intrinsic::amdgcn_raw_ptr_buffer_atomic_cmpswap, Ty,1856                          {AI.getNewValOperand(), AI.getCompareOperand(), Rsrc,1857                           Off, IRB.getInt32(0), IRB.getInt32(Aux)});1858  copyMetadata(Call, &AI);1859  setAlign(Call, AI.getAlign(), 2);1860  Call->takeName(&AI);1861  insertPostMemOpFence(Order, SSID);1862 1863  Value *Res = PoisonValue::get(AI.getType());1864  Res = IRB.CreateInsertValue(Res, Call, 0);1865  if (!AI.isWeak()) {1866    Value *Succeeded = IRB.CreateICmpEQ(Call, AI.getCompareOperand());1867    Res = IRB.CreateInsertValue(Res, Succeeded, 1);1868  }1869  SplitUsers.insert(&AI);1870  AI.replaceAllUsesWith(Res);1871  return {nullptr, nullptr};1872}1873 1874PtrParts SplitPtrStructs::visitGetElementPtrInst(GetElementPtrInst &GEP) {1875  using namespace llvm::PatternMatch;1876  Value *Ptr = GEP.getPointerOperand();1877  if (!isSplitFatPtr(Ptr->getType()))1878    return {nullptr, nullptr};1879  IRB.SetInsertPoint(&GEP);1880 1881  auto [Rsrc, Off] = getPtrParts(Ptr);1882  const DataLayout &DL = GEP.getDataLayout();1883  bool IsNUW = GEP.hasNoUnsignedWrap();1884  bool IsNUSW = GEP.hasNoUnsignedSignedWrap();1885 1886  StructType *ResTy = cast<StructType>(GEP.getType());1887  Type *ResRsrcTy = ResTy->getElementType(0);1888  VectorType *ResRsrcVecTy = dyn_cast<VectorType>(ResRsrcTy);1889  bool BroadcastsPtr = ResRsrcVecTy && !isa<VectorType>(Off->getType());1890 1891  // In order to call emitGEPOffset() and thus not have to reimplement it,1892  // we need the GEP result to have ptr addrspace(7) type.1893  Type *FatPtrTy =1894      ResRsrcTy->getWithNewType(IRB.getPtrTy(AMDGPUAS::BUFFER_FAT_POINTER));1895  GEP.mutateType(FatPtrTy);1896  Value *OffAccum = emitGEPOffset(&IRB, DL, &GEP);1897  GEP.mutateType(ResTy);1898 1899  if (BroadcastsPtr) {1900    Rsrc = IRB.CreateVectorSplat(ResRsrcVecTy->getElementCount(), Rsrc,1901                                 Rsrc->getName());1902    Off = IRB.CreateVectorSplat(ResRsrcVecTy->getElementCount(), Off,1903                                Off->getName());1904  }1905  if (match(OffAccum, m_Zero())) { // Constant-zero offset1906    SplitUsers.insert(&GEP);1907    return {Rsrc, Off};1908  }1909 1910  bool HasNonNegativeOff = false;1911  if (auto *CI = dyn_cast<ConstantInt>(OffAccum)) {1912    HasNonNegativeOff = !CI->isNegative();1913  }1914  Value *NewOff;1915  if (match(Off, m_Zero())) {1916    NewOff = OffAccum;1917  } else {1918    NewOff = IRB.CreateAdd(Off, OffAccum, "",1919                           /*hasNUW=*/IsNUW || (IsNUSW && HasNonNegativeOff),1920                           /*hasNSW=*/false);1921  }1922  copyMetadata(NewOff, &GEP);1923  NewOff->takeName(&GEP);1924  SplitUsers.insert(&GEP);1925  return {Rsrc, NewOff};1926}1927 1928PtrParts SplitPtrStructs::visitPtrToIntInst(PtrToIntInst &PI) {1929  Value *Ptr = PI.getPointerOperand();1930  if (!isSplitFatPtr(Ptr->getType()))1931    return {nullptr, nullptr};1932  IRB.SetInsertPoint(&PI);1933 1934  Type *ResTy = PI.getType();1935  unsigned Width = ResTy->getScalarSizeInBits();1936 1937  auto [Rsrc, Off] = getPtrParts(Ptr);1938  const DataLayout &DL = PI.getDataLayout();1939  unsigned FatPtrWidth = DL.getPointerSizeInBits(AMDGPUAS::BUFFER_FAT_POINTER);1940 1941  Value *Res;1942  if (Width <= BufferOffsetWidth) {1943    Res = IRB.CreateIntCast(Off, ResTy, /*isSigned=*/false,1944                            PI.getName() + ".off");1945  } else {1946    Value *RsrcInt = IRB.CreatePtrToInt(Rsrc, ResTy, PI.getName() + ".rsrc");1947    Value *Shl = IRB.CreateShl(1948        RsrcInt,1949        ConstantExpr::getIntegerValue(ResTy, APInt(Width, BufferOffsetWidth)),1950        "", Width >= FatPtrWidth, Width > FatPtrWidth);1951    Value *OffCast = IRB.CreateIntCast(Off, ResTy, /*isSigned=*/false,1952                                       PI.getName() + ".off");1953    Res = IRB.CreateOr(Shl, OffCast);1954  }1955 1956  copyMetadata(Res, &PI);1957  Res->takeName(&PI);1958  SplitUsers.insert(&PI);1959  PI.replaceAllUsesWith(Res);1960  return {nullptr, nullptr};1961}1962 1963PtrParts SplitPtrStructs::visitPtrToAddrInst(PtrToAddrInst &PA) {1964  Value *Ptr = PA.getPointerOperand();1965  if (!isSplitFatPtr(Ptr->getType()))1966    return {nullptr, nullptr};1967  IRB.SetInsertPoint(&PA);1968 1969  auto [Rsrc, Off] = getPtrParts(Ptr);1970  Value *Res = IRB.CreateIntCast(Off, PA.getType(), /*isSigned=*/false);1971  copyMetadata(Res, &PA);1972  Res->takeName(&PA);1973  SplitUsers.insert(&PA);1974  PA.replaceAllUsesWith(Res);1975  return {nullptr, nullptr};1976}1977 1978PtrParts SplitPtrStructs::visitIntToPtrInst(IntToPtrInst &IP) {1979  if (!isSplitFatPtr(IP.getType()))1980    return {nullptr, nullptr};1981  IRB.SetInsertPoint(&IP);1982  const DataLayout &DL = IP.getDataLayout();1983  unsigned RsrcPtrWidth = DL.getPointerSizeInBits(AMDGPUAS::BUFFER_RESOURCE);1984  Value *Int = IP.getOperand(0);1985  Type *IntTy = Int->getType();1986  Type *RsrcIntTy = IntTy->getWithNewBitWidth(RsrcPtrWidth);1987  unsigned Width = IntTy->getScalarSizeInBits();1988 1989  auto *RetTy = cast<StructType>(IP.getType());1990  Type *RsrcTy = RetTy->getElementType(0);1991  Type *OffTy = RetTy->getElementType(1);1992  Value *RsrcPart = IRB.CreateLShr(1993      Int,1994      ConstantExpr::getIntegerValue(IntTy, APInt(Width, BufferOffsetWidth)));1995  Value *RsrcInt = IRB.CreateIntCast(RsrcPart, RsrcIntTy, /*isSigned=*/false);1996  Value *Rsrc = IRB.CreateIntToPtr(RsrcInt, RsrcTy, IP.getName() + ".rsrc");1997  Value *Off =1998      IRB.CreateIntCast(Int, OffTy, /*IsSigned=*/false, IP.getName() + ".off");1999 2000  copyMetadata(Rsrc, &IP);2001  SplitUsers.insert(&IP);2002  return {Rsrc, Off};2003}2004 2005PtrParts SplitPtrStructs::visitAddrSpaceCastInst(AddrSpaceCastInst &I) {2006  // TODO(krzysz00): handle casts from ptr addrspace(7) to global pointers2007  // by computing the effective address.2008  if (!isSplitFatPtr(I.getType()))2009    return {nullptr, nullptr};2010  IRB.SetInsertPoint(&I);2011  Value *In = I.getPointerOperand();2012  // No-op casts preserve parts2013  if (In->getType() == I.getType()) {2014    auto [Rsrc, Off] = getPtrParts(In);2015    SplitUsers.insert(&I);2016    return {Rsrc, Off};2017  }2018 2019  auto *ResTy = cast<StructType>(I.getType());2020  Type *RsrcTy = ResTy->getElementType(0);2021  Type *OffTy = ResTy->getElementType(1);2022  Value *ZeroOff = Constant::getNullValue(OffTy);2023 2024  // Special case for null pointers, undef, and poison, which can be created by2025  // address space propagation.2026  auto *InConst = dyn_cast<Constant>(In);2027  if (InConst && InConst->isNullValue()) {2028    Value *NullRsrc = Constant::getNullValue(RsrcTy);2029    SplitUsers.insert(&I);2030    return {NullRsrc, ZeroOff};2031  }2032  if (isa<PoisonValue>(In)) {2033    Value *PoisonRsrc = PoisonValue::get(RsrcTy);2034    Value *PoisonOff = PoisonValue::get(OffTy);2035    SplitUsers.insert(&I);2036    return {PoisonRsrc, PoisonOff};2037  }2038  if (isa<UndefValue>(In)) {2039    Value *UndefRsrc = UndefValue::get(RsrcTy);2040    Value *UndefOff = UndefValue::get(OffTy);2041    SplitUsers.insert(&I);2042    return {UndefRsrc, UndefOff};2043  }2044 2045  if (I.getSrcAddressSpace() != AMDGPUAS::BUFFER_RESOURCE)2046    reportFatalUsageError(2047        "only buffer resources (addrspace 8) and null/poison pointers can be "2048        "cast to buffer fat pointers (addrspace 7)");2049  SplitUsers.insert(&I);2050  return {In, ZeroOff};2051}2052 2053PtrParts SplitPtrStructs::visitICmpInst(ICmpInst &Cmp) {2054  Value *Lhs = Cmp.getOperand(0);2055  if (!isSplitFatPtr(Lhs->getType()))2056    return {nullptr, nullptr};2057  Value *Rhs = Cmp.getOperand(1);2058  IRB.SetInsertPoint(&Cmp);2059  ICmpInst::Predicate Pred = Cmp.getPredicate();2060 2061  assert((Pred == ICmpInst::ICMP_EQ || Pred == ICmpInst::ICMP_NE) &&2062         "Pointer comparison is only equal or unequal");2063  auto [LhsRsrc, LhsOff] = getPtrParts(Lhs);2064  auto [RhsRsrc, RhsOff] = getPtrParts(Rhs);2065  Value *RsrcCmp =2066      IRB.CreateICmp(Pred, LhsRsrc, RhsRsrc, Cmp.getName() + ".rsrc");2067  copyMetadata(RsrcCmp, &Cmp);2068  Value *OffCmp = IRB.CreateICmp(Pred, LhsOff, RhsOff, Cmp.getName() + ".off");2069  copyMetadata(OffCmp, &Cmp);2070 2071  Value *Res = nullptr;2072  if (Pred == ICmpInst::ICMP_EQ)2073    Res = IRB.CreateAnd(RsrcCmp, OffCmp);2074  else if (Pred == ICmpInst::ICMP_NE)2075    Res = IRB.CreateOr(RsrcCmp, OffCmp);2076  copyMetadata(Res, &Cmp);2077  Res->takeName(&Cmp);2078  SplitUsers.insert(&Cmp);2079  Cmp.replaceAllUsesWith(Res);2080  return {nullptr, nullptr};2081}2082 2083PtrParts SplitPtrStructs::visitFreezeInst(FreezeInst &I) {2084  if (!isSplitFatPtr(I.getType()))2085    return {nullptr, nullptr};2086  IRB.SetInsertPoint(&I);2087  auto [Rsrc, Off] = getPtrParts(I.getOperand(0));2088 2089  Value *RsrcRes = IRB.CreateFreeze(Rsrc, I.getName() + ".rsrc");2090  copyMetadata(RsrcRes, &I);2091  Value *OffRes = IRB.CreateFreeze(Off, I.getName() + ".off");2092  copyMetadata(OffRes, &I);2093  SplitUsers.insert(&I);2094  return {RsrcRes, OffRes};2095}2096 2097PtrParts SplitPtrStructs::visitExtractElementInst(ExtractElementInst &I) {2098  if (!isSplitFatPtr(I.getType()))2099    return {nullptr, nullptr};2100  IRB.SetInsertPoint(&I);2101  Value *Vec = I.getVectorOperand();2102  Value *Idx = I.getIndexOperand();2103  auto [Rsrc, Off] = getPtrParts(Vec);2104 2105  Value *RsrcRes = IRB.CreateExtractElement(Rsrc, Idx, I.getName() + ".rsrc");2106  copyMetadata(RsrcRes, &I);2107  Value *OffRes = IRB.CreateExtractElement(Off, Idx, I.getName() + ".off");2108  copyMetadata(OffRes, &I);2109  SplitUsers.insert(&I);2110  return {RsrcRes, OffRes};2111}2112 2113PtrParts SplitPtrStructs::visitInsertElementInst(InsertElementInst &I) {2114  // The mutated instructions temporarily don't return vectors, and so2115  // we need the generic getType() here to avoid crashes.2116  if (!isSplitFatPtr(cast<Instruction>(I).getType()))2117    return {nullptr, nullptr};2118  IRB.SetInsertPoint(&I);2119  Value *Vec = I.getOperand(0);2120  Value *Elem = I.getOperand(1);2121  Value *Idx = I.getOperand(2);2122  auto [VecRsrc, VecOff] = getPtrParts(Vec);2123  auto [ElemRsrc, ElemOff] = getPtrParts(Elem);2124 2125  Value *RsrcRes =2126      IRB.CreateInsertElement(VecRsrc, ElemRsrc, Idx, I.getName() + ".rsrc");2127  copyMetadata(RsrcRes, &I);2128  Value *OffRes =2129      IRB.CreateInsertElement(VecOff, ElemOff, Idx, I.getName() + ".off");2130  copyMetadata(OffRes, &I);2131  SplitUsers.insert(&I);2132  return {RsrcRes, OffRes};2133}2134 2135PtrParts SplitPtrStructs::visitShuffleVectorInst(ShuffleVectorInst &I) {2136  // Cast is needed for the same reason as insertelement's.2137  if (!isSplitFatPtr(cast<Instruction>(I).getType()))2138    return {nullptr, nullptr};2139  IRB.SetInsertPoint(&I);2140 2141  Value *V1 = I.getOperand(0);2142  Value *V2 = I.getOperand(1);2143  ArrayRef<int> Mask = I.getShuffleMask();2144  auto [V1Rsrc, V1Off] = getPtrParts(V1);2145  auto [V2Rsrc, V2Off] = getPtrParts(V2);2146 2147  Value *RsrcRes =2148      IRB.CreateShuffleVector(V1Rsrc, V2Rsrc, Mask, I.getName() + ".rsrc");2149  copyMetadata(RsrcRes, &I);2150  Value *OffRes =2151      IRB.CreateShuffleVector(V1Off, V2Off, Mask, I.getName() + ".off");2152  copyMetadata(OffRes, &I);2153  SplitUsers.insert(&I);2154  return {RsrcRes, OffRes};2155}2156 2157PtrParts SplitPtrStructs::visitPHINode(PHINode &PHI) {2158  if (!isSplitFatPtr(PHI.getType()))2159    return {nullptr, nullptr};2160  IRB.SetInsertPoint(*PHI.getInsertionPointAfterDef());2161  // Phi nodes will be handled in post-processing after we've visited every2162  // instruction. However, instead of just returning {nullptr, nullptr},2163  // we explicitly create the temporary extractvalue operations that are our2164  // temporary results so that they end up at the beginning of the block with2165  // the PHIs.2166  Value *TmpRsrc = IRB.CreateExtractValue(&PHI, 0, PHI.getName() + ".rsrc");2167  Value *TmpOff = IRB.CreateExtractValue(&PHI, 1, PHI.getName() + ".off");2168  Conditionals.push_back(&PHI);2169  SplitUsers.insert(&PHI);2170  return {TmpRsrc, TmpOff};2171}2172 2173PtrParts SplitPtrStructs::visitSelectInst(SelectInst &SI) {2174  if (!isSplitFatPtr(SI.getType()))2175    return {nullptr, nullptr};2176  IRB.SetInsertPoint(&SI);2177 2178  Value *Cond = SI.getCondition();2179  Value *True = SI.getTrueValue();2180  Value *False = SI.getFalseValue();2181  auto [TrueRsrc, TrueOff] = getPtrParts(True);2182  auto [FalseRsrc, FalseOff] = getPtrParts(False);2183 2184  Value *RsrcRes =2185      IRB.CreateSelect(Cond, TrueRsrc, FalseRsrc, SI.getName() + ".rsrc", &SI);2186  copyMetadata(RsrcRes, &SI);2187  Conditionals.push_back(&SI);2188  Value *OffRes =2189      IRB.CreateSelect(Cond, TrueOff, FalseOff, SI.getName() + ".off", &SI);2190  copyMetadata(OffRes, &SI);2191  SplitUsers.insert(&SI);2192  return {RsrcRes, OffRes};2193}2194 2195/// Returns true if this intrinsic needs to be removed when it is2196/// applied to `ptr addrspace(7)` values. Calls to these intrinsics are2197/// rewritten into calls to versions of that intrinsic on the resource2198/// descriptor.2199static bool isRemovablePointerIntrinsic(Intrinsic::ID IID) {2200  switch (IID) {2201  default:2202    return false;2203  case Intrinsic::amdgcn_make_buffer_rsrc:2204  case Intrinsic::ptrmask:2205  case Intrinsic::invariant_start:2206  case Intrinsic::invariant_end:2207  case Intrinsic::launder_invariant_group:2208  case Intrinsic::strip_invariant_group:2209  case Intrinsic::memcpy:2210  case Intrinsic::memcpy_inline:2211  case Intrinsic::memmove:2212  case Intrinsic::memset:2213  case Intrinsic::memset_inline:2214  case Intrinsic::experimental_memset_pattern:2215  case Intrinsic::amdgcn_load_to_lds:2216    return true;2217  }2218}2219 2220PtrParts SplitPtrStructs::visitIntrinsicInst(IntrinsicInst &I) {2221  Intrinsic::ID IID = I.getIntrinsicID();2222  switch (IID) {2223  default:2224    break;2225  case Intrinsic::amdgcn_make_buffer_rsrc: {2226    if (!isSplitFatPtr(I.getType()))2227      return {nullptr, nullptr};2228    Value *Base = I.getArgOperand(0);2229    Value *Stride = I.getArgOperand(1);2230    Value *NumRecords = I.getArgOperand(2);2231    Value *Flags = I.getArgOperand(3);2232    auto *SplitType = cast<StructType>(I.getType());2233    Type *RsrcType = SplitType->getElementType(0);2234    Type *OffType = SplitType->getElementType(1);2235    IRB.SetInsertPoint(&I);2236    Value *Rsrc = IRB.CreateIntrinsic(IID, {RsrcType, Base->getType()},2237                                      {Base, Stride, NumRecords, Flags});2238    copyMetadata(Rsrc, &I);2239    Rsrc->takeName(&I);2240    Value *Zero = Constant::getNullValue(OffType);2241    SplitUsers.insert(&I);2242    return {Rsrc, Zero};2243  }2244  case Intrinsic::ptrmask: {2245    Value *Ptr = I.getArgOperand(0);2246    if (!isSplitFatPtr(Ptr->getType()))2247      return {nullptr, nullptr};2248    Value *Mask = I.getArgOperand(1);2249    IRB.SetInsertPoint(&I);2250    auto [Rsrc, Off] = getPtrParts(Ptr);2251    if (Mask->getType() != Off->getType())2252      reportFatalUsageError("offset width is not equal to index width of fat "2253                            "pointer (data layout not set up correctly?)");2254    Value *OffRes = IRB.CreateAnd(Off, Mask, I.getName() + ".off");2255    copyMetadata(OffRes, &I);2256    SplitUsers.insert(&I);2257    return {Rsrc, OffRes};2258  }2259  // Pointer annotation intrinsics that, given their object-wide nature2260  // operate on the resource part.2261  case Intrinsic::invariant_start: {2262    Value *Ptr = I.getArgOperand(1);2263    if (!isSplitFatPtr(Ptr->getType()))2264      return {nullptr, nullptr};2265    IRB.SetInsertPoint(&I);2266    auto [Rsrc, Off] = getPtrParts(Ptr);2267    Type *NewTy = PointerType::get(I.getContext(), AMDGPUAS::BUFFER_RESOURCE);2268    auto *NewRsrc = IRB.CreateIntrinsic(IID, {NewTy}, {I.getOperand(0), Rsrc});2269    copyMetadata(NewRsrc, &I);2270    NewRsrc->takeName(&I);2271    SplitUsers.insert(&I);2272    I.replaceAllUsesWith(NewRsrc);2273    return {nullptr, nullptr};2274  }2275  case Intrinsic::invariant_end: {2276    Value *RealPtr = I.getArgOperand(2);2277    if (!isSplitFatPtr(RealPtr->getType()))2278      return {nullptr, nullptr};2279    IRB.SetInsertPoint(&I);2280    Value *RealRsrc = getPtrParts(RealPtr).first;2281    Value *InvPtr = I.getArgOperand(0);2282    Value *Size = I.getArgOperand(1);2283    Value *NewRsrc = IRB.CreateIntrinsic(IID, {RealRsrc->getType()},2284                                         {InvPtr, Size, RealRsrc});2285    copyMetadata(NewRsrc, &I);2286    NewRsrc->takeName(&I);2287    SplitUsers.insert(&I);2288    I.replaceAllUsesWith(NewRsrc);2289    return {nullptr, nullptr};2290  }2291  case Intrinsic::launder_invariant_group:2292  case Intrinsic::strip_invariant_group: {2293    Value *Ptr = I.getArgOperand(0);2294    if (!isSplitFatPtr(Ptr->getType()))2295      return {nullptr, nullptr};2296    IRB.SetInsertPoint(&I);2297    auto [Rsrc, Off] = getPtrParts(Ptr);2298    Value *NewRsrc = IRB.CreateIntrinsic(IID, {Rsrc->getType()}, {Rsrc});2299    copyMetadata(NewRsrc, &I);2300    NewRsrc->takeName(&I);2301    SplitUsers.insert(&I);2302    return {NewRsrc, Off};2303  }2304  case Intrinsic::amdgcn_load_to_lds: {2305    Value *Ptr = I.getArgOperand(0);2306    if (!isSplitFatPtr(Ptr->getType()))2307      return {nullptr, nullptr};2308    IRB.SetInsertPoint(&I);2309    auto [Rsrc, Off] = getPtrParts(Ptr);2310    Value *LDSPtr = I.getArgOperand(1);2311    Value *LoadSize = I.getArgOperand(2);2312    Value *ImmOff = I.getArgOperand(3);2313    Value *Aux = I.getArgOperand(4);2314    Value *SOffset = IRB.getInt32(0);2315    Instruction *NewLoad = IRB.CreateIntrinsic(2316        Intrinsic::amdgcn_raw_ptr_buffer_load_lds, {},2317        {Rsrc, LDSPtr, LoadSize, Off, SOffset, ImmOff, Aux});2318    copyMetadata(NewLoad, &I);2319    SplitUsers.insert(&I);2320    I.replaceAllUsesWith(NewLoad);2321    return {nullptr, nullptr};2322  }2323  }2324  return {nullptr, nullptr};2325}2326 2327void SplitPtrStructs::processFunction(Function &F) {2328  ST = &TM->getSubtarget<GCNSubtarget>(F);2329  SmallVector<Instruction *, 0> Originals(2330      llvm::make_pointer_range(instructions(F)));2331  LLVM_DEBUG(dbgs() << "Splitting pointer structs in function: " << F.getName()2332                    << "\n");2333  for (Instruction *I : Originals) {2334    // In some cases, instruction order doesn't reflect program order,2335    // so the visit() call will have already visited coertain instructions2336    // by the time this loop gets to them. Avoid re-visiting these so as to,2337    // for example, avoid processing the same conditional twice.2338    if (SplitUsers.contains(I))2339      continue;2340    auto [Rsrc, Off] = visit(I);2341    assert(((Rsrc && Off) || (!Rsrc && !Off)) &&2342           "Can't have a resource but no offset");2343    if (Rsrc)2344      RsrcParts[I] = Rsrc;2345    if (Off)2346      OffParts[I] = Off;2347  }2348  processConditionals();2349  killAndReplaceSplitInstructions(Originals);2350 2351  // Clean up after ourselves to save on memory.2352  RsrcParts.clear();2353  OffParts.clear();2354  SplitUsers.clear();2355  Conditionals.clear();2356  ConditionalTemps.clear();2357}2358 2359namespace {2360class AMDGPULowerBufferFatPointers : public ModulePass {2361public:2362  static char ID;2363 2364  AMDGPULowerBufferFatPointers() : ModulePass(ID) {}2365 2366  bool run(Module &M, const TargetMachine &TM);2367  bool runOnModule(Module &M) override;2368 2369  void getAnalysisUsage(AnalysisUsage &AU) const override;2370};2371} // namespace2372 2373/// Returns true if there are values that have a buffer fat pointer in them,2374/// which means we'll need to perform rewrites on this function. As a side2375/// effect, this will populate the type remapping cache.2376static bool containsBufferFatPointers(const Function &F,2377                                      BufferFatPtrToStructTypeMap *TypeMap) {2378  bool HasFatPointers = false;2379  for (const BasicBlock &BB : F)2380    for (const Instruction &I : BB) {2381      HasFatPointers |= (I.getType() != TypeMap->remapType(I.getType()));2382      // Catch null pointer constants in loads, stores, etc.2383      for (const Value *V : I.operand_values())2384        HasFatPointers |= (V->getType() != TypeMap->remapType(V->getType()));2385    }2386  return HasFatPointers;2387}2388 2389static bool hasFatPointerInterface(const Function &F,2390                                   BufferFatPtrToStructTypeMap *TypeMap) {2391  Type *Ty = F.getFunctionType();2392  return Ty != TypeMap->remapType(Ty);2393}2394 2395/// Move the body of `OldF` into a new function, returning it.2396static Function *moveFunctionAdaptingType(Function *OldF, FunctionType *NewTy,2397                                          ValueToValueMapTy &CloneMap) {2398  bool IsIntrinsic = OldF->isIntrinsic();2399  Function *NewF =2400      Function::Create(NewTy, OldF->getLinkage(), OldF->getAddressSpace());2401  NewF->copyAttributesFrom(OldF);2402  NewF->copyMetadata(OldF, 0);2403  NewF->takeName(OldF);2404  NewF->updateAfterNameChange();2405  NewF->setDLLStorageClass(OldF->getDLLStorageClass());2406  OldF->getParent()->getFunctionList().insertAfter(OldF->getIterator(), NewF);2407 2408  while (!OldF->empty()) {2409    BasicBlock *BB = &OldF->front();2410    BB->removeFromParent();2411    BB->insertInto(NewF);2412    CloneMap[BB] = BB;2413    for (Instruction &I : *BB) {2414      CloneMap[&I] = &I;2415    }2416  }2417 2418  SmallVector<AttributeSet> ArgAttrs;2419  AttributeList OldAttrs = OldF->getAttributes();2420 2421  for (auto [I, OldArg, NewArg] : enumerate(OldF->args(), NewF->args())) {2422    CloneMap[&NewArg] = &OldArg;2423    NewArg.takeName(&OldArg);2424    Type *OldArgTy = OldArg.getType(), *NewArgTy = NewArg.getType();2425    // Temporarily mutate type of `NewArg` to allow RAUW to work.2426    NewArg.mutateType(OldArgTy);2427    OldArg.replaceAllUsesWith(&NewArg);2428    NewArg.mutateType(NewArgTy);2429 2430    AttributeSet ArgAttr = OldAttrs.getParamAttrs(I);2431    // Intrinsics get their attributes fixed later.2432    if (OldArgTy != NewArgTy && !IsIntrinsic)2433      ArgAttr = ArgAttr.removeAttributes(2434          NewF->getContext(),2435          AttributeFuncs::typeIncompatible(NewArgTy, ArgAttr));2436    ArgAttrs.push_back(ArgAttr);2437  }2438  AttributeSet RetAttrs = OldAttrs.getRetAttrs();2439  if (OldF->getReturnType() != NewF->getReturnType() && !IsIntrinsic)2440    RetAttrs = RetAttrs.removeAttributes(2441        NewF->getContext(),2442        AttributeFuncs::typeIncompatible(NewF->getReturnType(), RetAttrs));2443  NewF->setAttributes(AttributeList::get(2444      NewF->getContext(), OldAttrs.getFnAttrs(), RetAttrs, ArgAttrs));2445  return NewF;2446}2447 2448static void makeCloneInPraceMap(Function *F, ValueToValueMapTy &CloneMap) {2449  for (Argument &A : F->args())2450    CloneMap[&A] = &A;2451  for (BasicBlock &BB : *F) {2452    CloneMap[&BB] = &BB;2453    for (Instruction &I : BB)2454      CloneMap[&I] = &I;2455  }2456}2457 2458bool AMDGPULowerBufferFatPointers::run(Module &M, const TargetMachine &TM) {2459  bool Changed = false;2460  const DataLayout &DL = M.getDataLayout();2461  // Record the functions which need to be remapped.2462  // The second element of the pair indicates whether the function has to have2463  // its arguments or return types adjusted.2464  SmallVector<std::pair<Function *, bool>> NeedsRemap;2465 2466  LLVMContext &Ctx = M.getContext();2467 2468  BufferFatPtrToStructTypeMap StructTM(DL);2469  BufferFatPtrToIntTypeMap IntTM(DL);2470  for (const GlobalVariable &GV : M.globals()) {2471    if (GV.getAddressSpace() == AMDGPUAS::BUFFER_FAT_POINTER) {2472      // FIXME: Use DiagnosticInfo unsupported but it requires a Function2473      Ctx.emitError("global variables with a buffer fat pointer address "2474                    "space (7) are not supported");2475      continue;2476    }2477 2478    Type *VT = GV.getValueType();2479    if (VT != StructTM.remapType(VT)) {2480      // FIXME: Use DiagnosticInfo unsupported but it requires a Function2481      Ctx.emitError("global variables that contain buffer fat pointers "2482                    "(address space 7 pointers) are unsupported. Use "2483                    "buffer resource pointers (address space 8) instead");2484      continue;2485    }2486  }2487 2488  {2489    // Collect all constant exprs and aggregates referenced by any function.2490    SmallVector<Constant *, 8> Worklist;2491    for (Function &F : M.functions())2492      for (Instruction &I : instructions(F))2493        for (Value *Op : I.operands())2494          if (isa<ConstantExpr, ConstantAggregate>(Op))2495            Worklist.push_back(cast<Constant>(Op));2496 2497    // Recursively look for any referenced buffer pointer constants.2498    SmallPtrSet<Constant *, 8> Visited;2499    SetVector<Constant *> BufferFatPtrConsts;2500    while (!Worklist.empty()) {2501      Constant *C = Worklist.pop_back_val();2502      if (!Visited.insert(C).second)2503        continue;2504      if (isBufferFatPtrOrVector(C->getType()))2505        BufferFatPtrConsts.insert(C);2506      for (Value *Op : C->operands())2507        if (isa<ConstantExpr, ConstantAggregate>(Op))2508          Worklist.push_back(cast<Constant>(Op));2509    }2510 2511    // Expand all constant expressions using fat buffer pointers to2512    // instructions.2513    Changed |= convertUsersOfConstantsToInstructions(2514        BufferFatPtrConsts.getArrayRef(), /*RestrictToFunc=*/nullptr,2515        /*RemoveDeadConstants=*/false, /*IncludeSelf=*/true);2516  }2517 2518  StoreFatPtrsAsIntsAndExpandMemcpyVisitor MemOpsRewrite(&IntTM, DL,2519                                                         M.getContext(), &TM);2520  LegalizeBufferContentTypesVisitor BufferContentsTypeRewrite(DL,2521                                                              M.getContext());2522  for (Function &F : M.functions()) {2523    bool InterfaceChange = hasFatPointerInterface(F, &StructTM);2524    bool BodyChanges = containsBufferFatPointers(F, &StructTM);2525    Changed |= MemOpsRewrite.processFunction(F);2526    if (InterfaceChange || BodyChanges) {2527      NeedsRemap.push_back(std::make_pair(&F, InterfaceChange));2528      Changed |= BufferContentsTypeRewrite.processFunction(F);2529    }2530  }2531  if (NeedsRemap.empty())2532    return Changed;2533 2534  SmallVector<Function *> NeedsPostProcess;2535  SmallVector<Function *> Intrinsics;2536  // Keep one big map so as to memoize constants across functions.2537  ValueToValueMapTy CloneMap;2538  FatPtrConstMaterializer Materializer(&StructTM, CloneMap);2539 2540  ValueMapper LowerInFuncs(CloneMap, RF_None, &StructTM, &Materializer);2541  for (auto [F, InterfaceChange] : NeedsRemap) {2542    Function *NewF = F;2543    if (InterfaceChange)2544      NewF = moveFunctionAdaptingType(2545          F, cast<FunctionType>(StructTM.remapType(F->getFunctionType())),2546          CloneMap);2547    else2548      makeCloneInPraceMap(F, CloneMap);2549    LowerInFuncs.remapFunction(*NewF);2550    if (NewF->isIntrinsic())2551      Intrinsics.push_back(NewF);2552    else2553      NeedsPostProcess.push_back(NewF);2554    if (InterfaceChange) {2555      F->replaceAllUsesWith(NewF);2556      F->eraseFromParent();2557    }2558    Changed = true;2559  }2560  StructTM.clear();2561  IntTM.clear();2562  CloneMap.clear();2563 2564  SplitPtrStructs Splitter(DL, M.getContext(), &TM);2565  for (Function *F : NeedsPostProcess)2566    Splitter.processFunction(*F);2567  for (Function *F : Intrinsics) {2568    // use_empty() can also occur with cases like masked load, which will2569    // have been rewritten out of the module by now but not erased.2570    if (F->use_empty() || isRemovablePointerIntrinsic(F->getIntrinsicID())) {2571      F->eraseFromParent();2572    } else {2573      std::optional<Function *> NewF = Intrinsic::remangleIntrinsicFunction(F);2574      if (NewF)2575        F->replaceAllUsesWith(*NewF);2576    }2577  }2578  return Changed;2579}2580 2581bool AMDGPULowerBufferFatPointers::runOnModule(Module &M) {2582  TargetPassConfig &TPC = getAnalysis<TargetPassConfig>();2583  const TargetMachine &TM = TPC.getTM<TargetMachine>();2584  return run(M, TM);2585}2586 2587char AMDGPULowerBufferFatPointers::ID = 0;2588 2589char &llvm::AMDGPULowerBufferFatPointersID = AMDGPULowerBufferFatPointers::ID;2590 2591void AMDGPULowerBufferFatPointers::getAnalysisUsage(AnalysisUsage &AU) const {2592  AU.addRequired<TargetPassConfig>();2593}2594 2595#define PASS_DESC "Lower buffer fat pointer operations to buffer resources"2596INITIALIZE_PASS_BEGIN(AMDGPULowerBufferFatPointers, DEBUG_TYPE, PASS_DESC,2597                      false, false)2598INITIALIZE_PASS_DEPENDENCY(TargetPassConfig)2599INITIALIZE_PASS_END(AMDGPULowerBufferFatPointers, DEBUG_TYPE, PASS_DESC, false,2600                    false)2601#undef PASS_DESC2602 2603ModulePass *llvm::createAMDGPULowerBufferFatPointersPass() {2604  return new AMDGPULowerBufferFatPointers();2605}2606 2607PreservedAnalyses2608AMDGPULowerBufferFatPointersPass::run(Module &M, ModuleAnalysisManager &MA) {2609  return AMDGPULowerBufferFatPointers().run(M, TM) ? PreservedAnalyses::none()2610                                                   : PreservedAnalyses::all();2611}2612