Previous commit message:
> Original commit message:
>
>>When users explicitly specify a PTX version via -mattr=+ptxNN that's
insufficient for their target SM, we now emit a fatal error. Previously,
we silently upgraded the PTX version to the minimum required for the
target SM.
>>
>>When no SM or PTX version is specified, we now use PTX 3.2 (the
minimum for the default SM 3.0) instead of PTX 6.0.
>
>The following commits should fix the failures that arose when I
previously tried to land this commit:
>
>9fc5fd0ad6
should address the llvm-nvptx*-nvidia-* build failures:
https://github.com/llvm/llvm-project/pull/174834#issuecomment-3742242651
>
>600514a637
should address the MLIR failures
---
The previous commit was reverted with
d23cb79ba497281de050ef609cb91b91058bf323 because the
[mlir-nvidia](https://lab.llvm.org/buildbot/#/builders/138/builds/24797)
and
[mlir-nvidia-gcc7](https://lab.llvm.org/buildbot/#/builders/116/builds/23929)
Buildbots were failing.
Those tests failed because MLIR's default SM was 5.0, which caused NVPTX
to target PTX ISA v4.0, which did not support the intrinsics used in the
failing tests.
243f011577193c99358ccc4142b296d4fa80ea11 should address this by bumping
MLIR's default SM to 7.5. Now, using MLIR's new default SM, NVPTX
targets the PTX ISA v6.3, which supports the intrinsics used in the
failing tests.
As noted on #174761 - we need to ensure custom lowering to logic/add/sub
ops might actually occur before assuming the node will have been on the
vector unit
`global_load_lds` and `buffer_load to lds` do only increment `vmcnt` and
not touch `lgkmcnt`. This causes invalid `waitcnts` for some Triton
kernels, similar to the added lit tests.
Note that the change for buffer ops is not necesssary, i.e. the lit test
passes even before this PR, because it seems like `SIInsertWaitcnts`
does not use `LGKM_CNT` for buffer ops. But this change might prevent a
bug in the future.
Extending the work from https://github.com/llvm/llvm-project/pull/162678
which skips `llvm.compiler.used`, we also need to skip processing of
`llvm.used`.
The OpenMP frontend puts an `addrspace(2)` variable in `llvm_used`, but
the `llvm.used` array type uses the generic AS `addrspace(4)` so there's
a `addrspacecast` from `2` to `4` inside the initalizer for `llvm.used`
which is illegal and errors in the backend.
There is
[discussion](https://github.com/llvm/llvm-project/pull/162678#issuecomment-3396735124)
in the above linked PR that the handing should be the same for
`llvm.used` and `llvm.compiler.used`, but `llvm.used` was not added to
minimize the scope of the PR.
This is required to use the OpenMP Device RTL with SPIR-V.
Signed-off-by: Nick Sarnie <nick.sarnie@intel.com>
On GFX9, BUFFER_WBL2 is used to write back dirty cache lines and
requires an s_waitcnt vmcnt(0) afterwards to ensure completion.
This patch fixes by incrementing vmcnt for buffer_wbl2 instruction
---------
Co-authored-by: Jay Foad <jay.foad@gmail.com>
Currently we only fold an op into vmerge if we know the smaller of the
two vls, because we can't increase the vl.
However if the op's passthru is undef, then we can just use vmerge's vl
because the lanes past op's vl were undef anyway. We need to make sure
that the op doesn't access memory though. Other instructions where the
result depends on the VL should already be handled by the
RISCVII::elementsDependOnVL check below.
This is probably always profitable because even though we increase vl,
we remove a vmerge which needs to process vl elements anyway.
This removes some regressions in #179622.
If the source of an fpto*i doesn't fit in the destination type, the
result is poison. For i1 destinations, this means the result needs to be
0 or 1/-1, so we can just compare the result to 0 directly instead of
truncating.
The VP lowering for fpto*i already does this.
GISel CallLowering currently does a Type -> EVT -> Type roundtrip early
on when populating ArgInfo in splitToValueType(). This is a bit odd as
this structure operates at the IR Type level. Keep the original type
there and only convert to EVT when performing assignments.
Convert "denormal-fp-math" and "denormal-fp-math-f32" into a first
class denormal_fpenv attribute. Previously the query for the effective
denormal mode involved two string attribute queries with parsing. I'm
introducing more uses of this, so it makes sense to convert this
to a more efficient encoding. The old representation was also awkward
since it was split across two separate attributes. The new encoding
just stores the default and float modes as bitfields, largely avoiding
the need to consider if the other mode is set.
The syntax in the common cases looks like this:
`denormal_fpenv(preservesign,preservesign)`
`denormal_fpenv(float: preservesign,preservesign)`
`denormal_fpenv(dynamic,dynamic float: preservesign,preservesign)`
I wasn't sure about reusing the float type name instead of adding a
new keyword. It's parsed as a type but only accepts float. I'm also
debating switching the name to subnormal to match the current
preferred IEEE terminology (also used by nofpclass and other
contexts).
This has a behavior change when using the command flag debug
options to set the denormal mode. The behavior of the flag
ignored functions with an explicit attribute set, per
the default and f32 version. Now that these are one attribute,
the flag logic can't distinguish which of the two components
were explicitly set on the function. Only one test appeared to
rely on this behavior, so I just avoided using the flags in it.
This also does not perform all the code cleanups this enables.
In particular the attributor handling could be cleaned up.
I also guessed at how to support this in MLIR. I followed
MemoryEffects as a reference; it appears bitfields are expanded
into arguments to attributes, so the representation there is
a bit uglier with the 2 2-element fields flattened into 4 arguments.
cttz.elts is usually lowered (for SVE) to a brkb followed by a cntp. If
we then want a mask based on that (say, for early exit masking) then we
would use a whilelo from 0 to the result of cntp. But that just gives us
the same mask as the initial brkb, so we can just remove the cntp and
the whilelo.
Brka matches the extra +1 in the pattern.
SrcRegOff >= 0 is not needed at all for unsigned.
This fixes the warning:
```
llvm/lib/Target/Mips/MipsSEInstrInfo.cpp: In member function ‘virtual void llvm::MipsSEInstrInfo::copyPhysReg(llvm::MachineBasicBlock&, llvm::MachineBasicBlock::iterator, const llvm::DebugLoc&, llvm::Register, llvm::Register, bool, bool, bool) const’:
llvm/lib/Target/Mips/MipsSEInstrInfo.cpp:245:48: warning: comparison of unsigned expression in ‘>= 0’ is always true [-Wtype-limits]
245 | if (SrcRegOff == DestRegOff && SrcRegOff >= 0 && SrcRegOff <= 31)
| ~~~~~~~~~~^~~~
llvm/lib/Target/Mips/MipsSEInstrInfo.cpp:256:48: warning: comparison of unsigned expression in ‘>= 0’ is always true [-Wtype-limits]
256 | if (SrcRegOff == DestRegOff && SrcRegOff >= 0 && SrcRegOff <= 31)
```
XOR should be OR to match the comment.
Found while reviewing #179622 which deletes this function. I would like
to commit this first so we have a correct baseline for reviewing that
patch.
This PR adds full support for atomicrmw in NVPTX. This includes:
- Memory order and syncscope support (changes in AtomicExpandPass.cpp,
NVPTXIntrinsics.td)
- Script-generated tests for integer and atomic operations
(atomicrmw.py, atomicrmw-sm*.ll in tests/CodeGen/NVPTX). Existing
atomics tests which are subsumed by these have been removed
(atomics-sm*.ll, atomics.ll, atomicrmw-expand.ll).
- ~~Changes shouldExpandAtomicRMWInIR to take a constant argument: This
is to allow some other TargetLowering constant-argument functions to
call it. This change touches several backends. An alternative solution
exists, but to me, this seems the "right" way.~~ Has been split out into
https://github.com/llvm/llvm-project/pull/176073. Rebased.
- NOTE: The initial load issued for atomicrmw emulation loops (and
cmpxchg emulation loops) must be a strong load. Currently,
AtomicExpandPass issues a weak load. Fixing this breaks several
backends. I'm planning to follow up with a separate PR.
Initially failed due to error: ptxas fatal : Value 'sm_60' is not
defined for option 'gpu-name'. Updated RUN lines in atomicrmw-sm*.py to
skip the ptxas-verify check if ptxas does not support that SM version.
The only caller of this function (`PeepholeOptimizer::optimizeSelect`)
did not use most of the parameters, was broadly equivalent to
`MI->isSelect()`, and the `optimizeSelect` hook can return `nullptr`
anyway.
Update `optimizeSelect` to return `nullptr` by default rather than
asserting when not implemented.
Replace with hasStdExtZcd() and hastStdExtZcf().
Creation of RISCVSubtarget/MCSubtargetInfo handles implication of Zcf
and Zcd now. The exception is .option arch handling which will require
+zcf and +zcd to be listed explicitly. I'll try to fix this in a follow
up. #155035 had the same issue.
I've left the error messages mentioning both Zcf and C+F/D. We can
consider changing that in a follow up.
Accurately represent both the load and the store part of those intrinsics.
The test changes seem to be mostly fairly insignificant changes caused
by subtly different scheduler behavior.
The balanceSubTree() function in HexagonISelDAGToDAG was accessing node
operands after recursive calls that could invalidate those nodes through
RAUW.
Use pre-captured Op0/Op1 values instead of re-accessing N's operands
after BST calls, add bounds checking before accessing node operands in
several places, check if nodes were RAUW'd before calling getHeight() on
them.
The eventCounter() function searches through the array of events. This
array is owned by the WaitcntGenerator class.
This patch moves the function into the WaitcntGenerator class which
helps hide the event array from the user.
It also renames it to getCounterFromEvent().
This should be NFC.
This patch fixes https://github.com/llvm/llvm-project/issues/150737.
The original computed CSRCost is too small, so the optimization of
spilling instead of using CSR is rarely triggered.
Also the original cost model is too difficult to be understood and too
hard to be tuned by backend developers and users.
So this patch changes the CSRCost to be
CSRCost = TRI->getCSRFirstUseCost() * EntryFreq * Scale
TRI->getCSRFirstUseCost() is the raw cost of save/restore a CSR. Usually
we don't need to tune this number.
EntryFreq is the BlockFrequency of the entry block.
Scale is used to scale down the CSRCost, because we usually prefer a CSR
register instead of spilling if we have similar CSRCost and spill cost,
so it should be less than 100%. We usually tune this number.
Another problem is the original function RAGreedy::calcSpillCost()
actually computes a cost for block split, so this patch also implements
a correct RAGreedy::calcSpillCost() function.
This new behavior is not enabled by default. This optimization is used
by 3 targets (AArch64 / AMDGPU / RISCV), I will change them one by one
in following patches.
This ensures the feature bits and RISCVSubtarget flags match what
RISCVISAInfo would do.
I'm not excited about the code duplication, but I need to set the
RISCVSubtarget flags along with calling ToggleFeature. I'll think about
how to improve this.
We were folding undef inputs to qnan which is incorrect. The instruction
never returns nan. Out of bounds segment select will return 0, so fold
undef segment to 0.
create t16 pseudos for mubuffer d16 load/store with vgpr16 in vdst/vdata
and use these t16 pseudo for isel pattern. Lower them back to d16
machine inst in mc level.
Implement lowerConstants for SystemZ and handle special cases where
entries need to be created in the ADA for static functions or VCon for
externals.
---------
Co-authored-by: Tony Tao <tonytao@ca.ibm.com>
When the triple is spirv-unknown-unknown, SPIRVSubtarget::Env starts as
Unknown and was set via const_cast in SPIRVCallLowering when the first
entry point was lowered. This is too late: SPIRVLegalizerInfo has
already been constructed with, for example, the wrong vector size
limits, causing a crash (at best) or invalid SPIR-V generation.
Resolve the environment early in SPIRVPrepareFunctions::runOnModule() by
scanning the module for "hlsl.shader" attributes. Reinitialize the
legalizer and extended instruction sets after resolution. Remove the
const_cast lazy setting from SPIRVCallLowering.
Fixes: https://github.com/llvm/llvm-project/issues/171898
