SIInstrInfo::resultDependsOnExec assumes that operand 0 of a comparison
is always the destination of the instruction. This is not true for
instructions in VOPC form where it is "src0". This led to a crash in
machine-cse.
---------
Co-authored-by: Matt Arsenault <arsenm2@gmail.com>
The PR will fix the issue
https://github.com/llvm/llvm-project/issues/122728
This patch addresses the signed/zero extension of poison by using a
poison value of the extended type instead of a constant zero of the
extended type.
This avoids the need to have special handling at every use site.
Unfortunately this means we unnecessarily emit AssertZext in the DAG
(where we already directly understand the range of the intrinsic), andt
we regress in undefined cases as we don't fold out asserts on undef.
On targets that support v_cvt_pk_f16_f32 instruction, if we make v2f64
-> v2f16 Legal, we will generate the following sequence of instructions:
v_cvt_f32_f64_e32 v1, s[6:7]
v_cvt_f32_f64_e32 v2, s[4:5]
v_cvt_pk_f16_f32 v1, v2, v1
It possibly returns imprecise results due to double rounding. This patch
fixes the issue by not setting the conversion Legal. While we may still
expect the above sequence of code when unsafe fpmath is set, I hope
https://github.com/llvm/llvm-project/pull/134738 can address that
performance concern.
Fixes: SWDEV-523856
Re-apply https://github.com/llvm/llvm-project/pull/130577
Which is reverted in https://github.com/llvm/llvm-project/pull/133880
The old application failed in address sanitizer due to
`tryNarrowMathIfNoOverflow` was called after `I.eraseFromParent();` in
`AMDGPUCodeGenPrepareImpl::visitBinaryOperator`, it create a use after
free failure.
To fix this, `tryNarrowMathIfNoOverflow` will be called before and
directly return if `tryNarrowMathIfNoOverflow` result in true.
Previously the AnnotateKernelFeatures pass infers two attributes:
amdgpu-calls and amdgpu-stack-objects, which are used to help determine
if flat scratch init is allowed. PR #118907 created the
amdgpu-no-flat-scratch-init attribute. Continuing with that work, this
patch makes use of this attribute to determine flat scratch init,
replacing amdgpu-calls and amdgpu-stack-objects. This also leads to the
removal of the AnnotateKernelFeatures pass.
This patch addresses three problems when promoting allocas to vectors:
- Element types with size < 1 byte in allocas with a vector type caused
divisions by zero.
- Element types whose size doesn't match their AllocSize hit an assertion.
- Access types whose size doesn't match their AllocSize hit an assertion.
With this patch, we do not attempt to promote affected allocas to vectors. In
principle, we could handle these cases in PromoteAlloca, e.g., by truncating
and extending elements from/to their allocation size. It's however unclear if
we ever encounter such cases in practice, so that doesn't seem worth the added
complexity.
For SWDEV-511252
This adds IGLP mutation to the iterative schedulers
(`gcn-iterative-max-occupancy-experimental`, `gcn-iterative-minreg`, and
`gcn-iterative-ilp`).
The `gcn-iterative-minreg` and `gcn-iterative-ilp` schedulers never
actually applied the mutations added, so this also has the effect of
teaching them about mutations in general. The
`gcn-iterative-max-occupancy-experimental` scheduler has calls to
`ScheduleDAGMILive::schedule()`, so, before this, mutations were applied
at this point. Now this is done during calls to `BuildDAG`, with IGLP
superseding other mutations (similar to the other schedulers). We may
end up scheduling regions multiple times, with mutations being applied
each time, so we need to track for
`AMDGPU::SchedulingPhase::PreRAReentry`
Currently, the only way for users to try these schedulers is via
`-misched=` . However, this overrides the default scheduler for all
targets. This causes problems for various toolchains / drivers which
spawn jobs for both x86 and AMDGPU -- e.g. hipcc. On the other hand,
`amdgpu-sched-strategy` only changes the scheduler for AMDGPU target.
PHI operands and results must belong to the same register class.
If a PHI node produces an SGPR, but one of its operands is a VGPR, we
insert a VGPR-to-SGPR copy in the operand’s source block. The PHI
operand is then updated to use the destination register of the inserted
copy.
These inserted copies are processed immediately when they are created.
Therefore, we should avoid reprocessing them when handling their parent
block later.
---------
Co-authored-by: Matt Arsenault <arsenm2@gmail.com>
Add target feature for point sample acceleration and enable it for
relevant
targets.
Also add support to insert waitcnts where required when point sample
accel may
have occurred. This has implications for out of order returns, which is
why
extra waitcnts are required.
Add a VMEM_NOSAMPLER bit in the register masks to determine when
waitcnt is required.
FP_ROUND and FP_EXTEND the input value before FABSing it. This avoids
some bit twiddling to copy the sign bit from the input to the result. It
does introduce one extra FABS, but that is folded into another
instruction for free on AMDGPU, which is the only target currently
affected by this change.
This patch dismantles G_SHUFFLE_VECTOR before lowering. The original
lowering would emit extract vector element ops. We found that by using
unmerged values the build vector op combine could find ways to fold.
Only enabled on AMDGPU.
This resolves#123631
Background:
"amdgpu-sw-lower-lds" pass lowers LDS accesses based on
"sanitize_address" attribute being tagged to kernel or non-kernels.
"amdgpu-sw-lower-lds" pass ideally should either lower all LDS accesses
or should not lower any based on if asan is enabled.
Issue:
But there has been cases when instrumented and non instrumented bitcodes
are linked and this is leading to few LDS being lowered correctly while
others are not. This typically leads to below error in the subsequent
pass.
"Module cannot mix absolute and non-absolute LDS GVs"
Fix:
This patch fixes this issue, by checking if any kernels in module are
tagged with "sanitize_address" attribute and then lowers all the LDS
accesses in all other kernels and non-kernels even though they do not
have "sanitize_address" attribute.
This reverts commit d1a05721172272f7aab685b56d99e86814a15bff.
There was further discussion on the PR about whether the intinsics
should exist in this form.
function `bitcast_v64i16_to_v128i8` in newly added test file
`llvm-project/llvm/test/CodeGen/AMDGPU/amdgcn.bitcast.1024bit.ll` from
PR:https://github.com/llvm/llvm-project/pull/133052 failed in expansive
check. (passes normal lit check)
remove it for now
This PR add test cases for all types of bit conversation, it prepares
for PR: https://github.com/llvm/llvm-project/pull/132899
All tests passed due to:
1. For DAG, pattern will not separate SReg and VReg. One of the sample
is:
```
define <2 x double> @v_bitcast_v4f32_to_v2f64(<4 x float> inreg %a, i32
%b) {
%cmp = icmp eq i32 %b, 0
br i1 %cmp, label %cmp.true, label %cmp.false
cmp.true:
%a1 = fadd <4 x float> %a, splat (float 1.000000e+00)
%a2 = bitcast <4 x float> %a1 to <2 x double>
br label %end
cmp.false:
%a3 = bitcast <4 x float> %a to <2 x double>
br label %end
end:
%phi = phi <2 x double> [ %a2, %cmp.true ], [ %a3, %cmp.false ]
ret <2 x double> %phi
}
```
It suppose to select from scalar register patterns. But the Vreg pattern
is matched is as follow:
```
Debug log:
ISEL: Starting selection on root node: t3: v2f64 = bitcast t2
ISEL: Starting pattern match
Initial Opcode index to 440336
Skipped scope entry (due to false predicate) at index 440339, continuing
at 440367
Skipped scope entry (due to false predicate) at index 440368, continuing
at 440396
Skipped scope entry (due to false predicate) at index 440397, continuing
at 440435
Skipped scope entry (due to false predicate) at index 440436, continuing
at 440467
Skipped scope entry (due to false predicate) at index 440468, continuing
at 440499
Skipped scope entry (due to false predicate) at index 440500, continuing
at 440552
Skipped scope entry (due to false predicate) at index 440553, continuing
at 440587
Skipped scope entry (due to false predicate) at index 440588, continuing
at 440622
Skipped scope entry (due to false predicate) at index 440623, continuing
at 440657
Skipped scope entry (due to false predicate) at index 440658, continuing
at 440692
Skipped scope entry (due to false predicate) at index 440693, continuing
at 440727
Skipped scope entry (due to false predicate) at index 440728, continuing
at 440769
Skipped scope entry (due to false predicate) at index 440770, continuing
at 440798
Skipped scope entry (due to false predicate) at index 440799, continuing
at 440836
Skipped scope entry (due to false predicate) at index 440837, continuing
at 440870
TypeSwitch[v2f64] from 440873 to 440892
Patterns:
/*440892*/ OPC_CompleteMatch, 1, 0,
// Src: (bitconvert:{ *:[v2f64] } VReg_128:{ *:[v4f32] }:$src0) -
Complexity = 3
// Dst: VReg_128:{ *:[v2f64] }:$src0
```
2. Global isel will use `Select_COPY` to select bitcast
Add a buffer_fat_ptr_load_lds intrinsic, by analogy with
global_load_lds, which enables using `ptr addrspace(7)` to set the rsrc
and offset arguments to raw_ptr_buffer_load_lds.
f64 -> bf16 conversion can be lowered to f64 -> f32 followed by f32 ->
bf16:
v_cvt_f32_f64_e32 v0, v[0:1]
v_cvt_pk_bf16_f32 v0, v0, s0
Both conversion instructions will do round-to-even rounding, and thus we
will have double rounding issue which may generate incorrect result in
some data range. We need to add round-to-odd rounding during f64 -> f32
to avoid double rounding,.
NOTE: we are having the same issue with f64 -> f16 conversion. Will add
round-to-odd rounding for it in a separate patch, which fixes
SWDEV-523856
---------
Co-authored-by: Matt Arsenault <arsenm2@gmail.com>
Almost all of the rotate idioms that are valid for an 'or' are also
valid when the halves are combined with an 'add'. Further, many of these
cases are not handled by common bits tracking meaning that the 'add' is
not converted to a 'disjoint or'.
