Currently, each variant in the variant part of a structure type can only
contain a single member. This was sufficient for Rust, where each
variant is represented as its own type.
However, this isn't really enough for Ada, where a variant can have
multiple members.
This patch adds support for this scenario. This is done by allowing the
use of DW_TAG_variant by DICompositeType, and then changing the DWARF
generator to recognize when a DIDerivedType representing a variant holds
one of these. In this case, the fields from the DW_TAG_variant are
inlined into the variant, like so:
```
<4><7d>: Abbrev Number: 9 (DW_TAG_variant)
<7e> DW_AT_discr_value : 74
<5><7f>: Abbrev Number: 7 (DW_TAG_member)
<80> DW_AT_name : (indirect string, offset: 0x43): field0
<84> DW_AT_type : <0xa7>
<88> DW_AT_alignment : 8
<89> DW_AT_data_member_location: 0
<5><8a>: Abbrev Number: 7 (DW_TAG_member)
<8b> DW_AT_name : (indirect string, offset: 0x4a): field1
<8f> DW_AT_type : <0xa7>
<93> DW_AT_alignment : 8
<94> DW_AT_data_member_location: 8
```
Note that the intermediate DIDerivedType is still needed in this
situation, because that is where the discriminants are stored.
On some platforms (particularly macOS), a `\01` prefix gets added to the
name in an `asm` label. This gets stripped when we emit the
[`DW_AT_linkage_name`](2f877c2722/llvm/lib/CodeGen/AsmPrinter/DwarfUnit.cpp (L531)).
But we weren't stripping this prefix when inserting the linkage name
into accelerator tables.
This manifested in an issue where LLDB tried to look up a name in the
index by linkage name, but wasn't able to find it because we indexed it
with the `\01` unstripped.
This patch strips the prefix before indexing.
Proposed by
[2ed1598](2ed15984b4):
`fshl X, (or X, Y), C ==/!= 0 --> or (srl Y, BW-C), X ==/!= 0`
This transformation is valid when (C%Bitwidth) != 0 , as verified by
[Alive2](https://alive2.llvm.org/ce/z/TQYM-m).
Fixes#136746
And related "[AMDGPU] Regenerate mfma-loop.ll test"
Introduce memory error detected by Asan #125885.
This reverts commit 382a085a95b0abeac77b150b7b644b372bd08e78.
This reverts commit 067caaafb58a156d0d77229422607782a639f5b5.
Adds support for MSVC's undocumented `/funcoverride` flag, which marks
functions as being replaceable by the Windows kernel loader. This is
used to allow functions to be upgraded depending on the capabilities of
the current processor (e.g., the kernel can be built with the naive
implementation of a function, but that function can be replaced at boot
with one that uses SIMD instructions if the processor supports them).
For each marked function we need to generate:
* An undefined symbol named `<name>_$fo$`.
* A defined symbol `<name>_$fo_default$` that points to the `.data`
section (anywhere in the data section, it is assumed to be zero sized).
* An `/ALTERNATENAME` linker directive that points from `<name>_$fo$` to
`<name>_$fo_default$`.
This is used by the MSVC linker to generate the appropriate metadata in
the Dynamic Value Relocation Table.
Marked function must never be inlined (otherwise those inline sites
can't be replaced).
Note that I've chosen to implement this in AsmPrinter as there was no
way to create a `GlobalVariable` for `<name>_$fo$` that would result in
a symbol being emitted (as nothing consumes it and it has no
initializer). I tried to have `llvm.used` and `llvm.compiler.used` point
to it, but this didn't help.
Within LLVM I referred to this feature as "loader replaceable" as
"function override" already has a different meaning to C++ developers...
I also took the opportunity to extract the feature symbol generation
code used by both AArch64 and X86 into a common function in AsmPrinter.
Use all_uses and all_defs instead of separate Defs vector. Use
SmallSetVector instead of separate SmallSet and SmallVector. Remove
unneeded `Added` set. Fold FrameSetup/FrameDestroy into the main loop
instead of doing a separate loop over the bundled instructions.
As with the recently added subvector variants, provide the unsigned
index operand to simplify a bunch of code.
---------
Co-authored-by: Luke Lau <luke_lau@icloud.com>
When diagnosing scheduling issues it can be useful to know which
heuristics are driving the scheduler. This adds pre-RA and post-RA
statistics for all heuristics.
If there's an implicit-def of a super register, the propagation
must preserve this implicit-def. Knowing how and when to do this
may require target specific knowledge so just disable it for now.
Prior to 2def1c4, we checked that the copy had explicit 2 operands
when that was removed we started allowing implicit operands through.
This patch adds a check for implicit operands, but still allows
extra explicit operands which was the goal of 2def1c4.
Fixes#131478.
This patch adds a couple of improvements to the LLVM emission of DWARF
variant parts. One of these is desirable for Ada, and the other is
required.
Currently, when emitting a discriminant, LLVM follows the precise letter
of the DWARF standard, which says:
If the variant part has a discriminant, the discriminant is
represented by a separate debugging information entry which is a
child of the variant part entry.
However, for Ada this does not really make sense. In Ada, the
discriminant field exists outside of any variant part, and it makes more
sense to emit it separately rather than redundantly emit the field once
for each variant part.
This extension was arrived at when this was implemented in GCC, and was
accepted for DWARF 6, see:
https://dwarfstd.org/issues/180123.1.html
Here the patch simply lifts this restriction: if the discriminant field
was already emitted, it isn't re-emitted. This approach allows the Ada
compiler to do what it needs without affecting the Rust output.
Second, this patch extends the discriminant to allow multiple values.
This is needed by Ada. Here, I chose to use a ConstantDataArray of pairs
of integers, with each pair representing a range, as Ada also allows
ranges here. This seemed like a reasonably convenient representation.
Follow up to 6e654caab, use the new routines in more places. Note that
I've excluded from this patch any case which uses a getConstant index
instead of a getVectorIdxConstant index just to minimize room for
error. I'll get those in a separate follow up.
Mechanical change to introduce the new wrappers, and add enough users to
make the usage pattern clear. Once this lands, I'm going to do a further
pass to adjust more callsites as separate changes.
---------
Co-authored-by: Luke Lau <luke_lau@icloud.com>
PR #136875 was posted as a draft PR that handled a subset of these
cases, using the CompressPat mechanism. The consensus from that
discussion (and a conclusion I agree with) is that it would be
beneficial doing this optimisation earlier on, and in a way that isn't
limited just to cases that can be handled by instruction compression.
The most common source for instructions that can be
optimized/canonicalized in this way is through tail duplication in
MachineBlockPlacement followed by machine copy propagation. For RISC-V,
choosing a more canonical instruction allows it to be compressed when it
couldn't be before. There is the potential that it would make other
MI-level optimisations easier.
This modifies ~910 instructions across an llvm-test-suite build
including SPEC2017, targeting rva22u64. Looking at the diff, it seems
there's room for eliminating instructions or further propagating after
this.
Coverage of instructions is based on observations from a script written
to find redundant or improperly canonicalized instructions (though I aim
to support all instructions in a 'group' at once, e.g. MUL* even if I
only saw some variants of MUL in practice).
The initial patch (#135782 caused issues because it emits an error, and llc is sensitive to it.
It also caused compiler-rt/lib/scudo/standalone/tests/wrappers_cpp_test.cpp to fail.
Use warnings instead + reject lowering. That way, the fallback path is used without llc/clang returning a failure code.
If fallback isn't enabled then the warnings provide context as to why lowering failed.
Original commit description for #135782:
Instead of printing something to dbgs (which is not visible to all users),
emit a diagnostic like the DAG does. We still crash later because we fail to
select the inline assembly, but at least now users will know why it's crashing.
In a future patch we could also recover from the error like the DAG does, so the
lowering can keep going until it either crashes or gives a different error later.
AMDGPU scheduler's `PreRARematStage` attempts to increase function
occupancy w.r.t. ArchVGPR usage by rematerializing trivial
ArchVGPR-defining instruction next to their single use. It first
collects all eligible trivially rematerializable instructions in the
function, then sinks them one-by-one while recomputing occupancy in all
affected regions each time to determine if and when it has managed to
increase overall occupancy. If it does, changes are committed to the
scheduler's state; otherwise modifications to the IR are reverted and
the scheduling stage gives up.
In both cases, this scheduling stage currently involves repeated queries
for up-to-date occupancy estimates and some state copying to enable
reversal of sinking decisions when occupancy is revealed not to
increase. The current implementation also does not accurately track
register pressure changes in all regions affected by sinking decisions.
This commit refactors this scheduling stage, improving RP tracking and
splitting the stage into two distinct steps to avoid repeated occupancy
queries and IR/state rollbacks.
- Analysis and collection (`canIncreaseOccupancyOrReduceSpill`). The
number of ArchVGPRs to save to reduce spilling or increase function
occupancy by 1 (when there is no spilling) is computed. Then,
instructions eligible for rematerialization are collected, stopping as
soon as enough have been identified to be able to achieve our goal
(according to slightly optimistic heuristics). If there aren't enough of
such instructions, the scheduling stage stops here.
- Rematerialization (`rematerialize`). Instructions collected in the
first step are rematerialized one-by-one. Now we are able to directly
update the scheduler's state since we have already done the occupancy
analysis and know we won't have to rollback any state. Register
pressures for impacted regions are recomputed only once, as opposed to
at every sinking decision.
In the case where the stage attempted to increase occupancy, and if both
rematerializations alone and rescheduling after were unable to improve
occupancy, then all rematerializations are rollbacked.
Reapply "IR: Remove uselist for constantdata (#137313)"
This reverts commit 5936c02c8b9c6d1476f7830517781ce8b6e26e75.
Fix checking uselists of constants in assume bundle queries
Teach InterleavedAccessPass to recognize vp.load + shufflevector and
shufflevector + vp.store. Though this patch only adds RISC-V support to
actually lower this pattern. The vp.load/vp.store in this pattern
require constant mask.
Instead of printing something to dbgs (which is not visible to all users),
emit a diagnostic like the DAG does. We still crash later because we fail to
select the inline assembly, but at least now users will know why it's crashing.
In a future patch we could also recover from the error like the DAG does, so the
lowering can keep going until it either crashes or gives a different error later.
When diagnosing scheduler issues it can be useful to know how scheduling
changes the order of instructions, particularly for large functions when
it's not trivial to figure out from the debug output by looking at the
scheduling unit (SU) IDs.
This adds pre-RA and post-RA statistics to track 1) the number of
instructions that remain in source order after scheduling and 2) the
total number of instructions scheduled, to compare 1) against.
Register assembly printer passes in the pass registry.
This makes it possible to use `llc -start-before=<target>-asm-printer ...` in tests.
Adds a `char &ID` parameter to the AssemblyPrinter constructor to allow
targets to use the `INITIALIZE_PASS` macros and register the pass in the
pass registry. This currently has a default parameter so it won't break
any targets that have not been updated.
Migrate their usage to the `AnyMem*Inst` family, and add a isAtomic()
query on the base class for that hierarchy. This matches the idioms we
use for e.g. isAtomic on load, store, etc.. instructions, the existing
isVolatile idioms on mem* routines, and allows us to more easily share
code between atomic and non-atomic variants.
As with #138568, the goal here is to simplify the class hierarchy and
make it easier to reason about. I'm moving from easiest to hardest, and
will stop at some point when I hit "good enough". Longer term, I'd sorta
like to merge or reverse the naming on the plain Mem*Inst and the
AnyMem*Inst, but that's a much larger and more risky change. Not sure
I'm going to actually do that.
Generic DAG combine for ISD::PARTIAL_REDUCE_U/SMLA to convert:
PARTIAL_REDUCE_*MLA(Acc, ZEXT(UnextOp1), Splat(1)) into
PARTIAL_REDUCE_UMLA(Acc, UnextOp1, TRUNC(Splat(1))) and
PARTIAL_REDUCE_*MLA(Acc, SEXT(UnextOp1), Splat(1)) into
PARTIAL_REDUCE_SMLA(Acc, UnextOp1, TRUNC(Splat(1))).
---------
Co-authored-by: James Chesterman <james.chesterman@arm.com>
This is a follow up change to eliminating uselists for ConstantData.
In the previous revision, ConstantData had a replacement reference count
instead of a uselist. This reference count was misleading, and not useful
in the same way as it would be for another value. The references may not
have even been in the current module, since these are shared throughout
the LLVMContext.
This doesn't space leak any more than we previously did; nothing was
attempting to garbage collect unused constants.
Previously the use_empty, and hasNUses type of APIs were supported through
the reference count. These now behave as if the uses are always empty.
Ideally it would be illegal to inspect these, but this forces API complexity
into quite a few places. It may be doable to make it illegal to check these
counts, but I would like there to be a targeted fuzzing effort to make sure
every transform properly deals with a constant in every operand position.
All tests pass if I turn the hasNUses* and getNumUses queries into assertions,
only hasOneUse in particular appears to hit in some set of contexts. I've
added unit tests to ensure logical consistency between these cases
This is a resurrected version of the patch attached to this RFC:
https://discourse.llvm.org/t/rfc-constantdata-should-not-have-use-lists/42606
In this adaptation, there are a few differences. In the original patch, the Use's
use list was replaced with an unsigned* to the reference count in the value. This
version leaves them as null and leaves the ref counting only in Value.
Remove use-lists from instances of ConstantData (which are shared
across modules and have no operands).
To continue supporting most of the use-list API, store a ref-count in
place of the use-list; this is for API like Value::use_empty and
Value::hasNUses. Operations that actually need the use-list -- like
Value::use_begin -- will assert.
This change has three benefits:
1. The compiler output cannot in any way depend on the use-list order
of instances of ConstantData.
2. There's no use-list traffic when adding and removing simple
constants from operand lists (although there is ref-count traffic;
YMMV).
3. It's cheaper to serialize use-lists (since we're no longer
serializing the use-list order of things like i32 0).
The downside is that you can't look at all the users of ConstantData,
but traversals of users of i32 0 are already ill-advised.
Possible follow-ups:
- Track if an instance of a ConstantVector/ConstantArray/etc. is known
to have all ConstantData arguments, and drop the use-lists to
ref-counts in those cases. Callers need to check Value::hasUseList
before iterating through the use-list.
- Remove even the ref-counts. I'm not sure they have any benefit
besides minimizing the scope of this commit, and maintaining the
counts is not free.
Fixes#58629
Co-authored-by: Duncan P. N. Exon Smith <dexonsmith@apple.com>
This is a follow up to c0a264e, but note that there is a functional
difference here: the root changes for the memcpy.inline case. This
difference appears to have been accidental, but I kept this back to
facility separate review in case there's something I'm missing here.
This flag instructs the scheduler to stop scheduling after N
instructions, but
in the debug output it appears as if it's scheduling N+1 instructions,
e.g.
$ llc -misched-cutoff=10 -debug-only=machine-scheduler
example.ll 2>&1 | grep "^Scheduling SU" | wc -l
11
as it calls pickNode before calling checkSchedLimit.
Handles bitreverse for vector types which were previously falling back
onto Selection DAG. Includes 8-bit element vectors greater than 128 bits
and less than 64 bits: <32 x i8>, <4 x i8>, and odd vector types: <9 x
i8>.
This provides the `disable-schedmodel-in-sched-mi` flag. Using this, we
will disable the SchedModel / Itineraries during scheduling. This has
the effect of not using any latency / hardware resource information for
scheduling decisions.
We have the `schedmodel` flag, but this disables the `SchedModel` for
all passes. This allows disabling only for scheduling while preserving
the behavior of other passes (e.g. MachineLICM). This is conceptually
similar to other flags like `enable-aa-sched-mi`
This is a reland of #138434 except that:
- the bits for llvm/lib/CodeGen/RenameIndependentSubregs.cpp
have been dropped because they caused a test failure under asan, and
- the bits for llvm/lib/CodeGen/SelectionDAG/ScheduleDAGFast.cpp have
been improved with structured bindings.
The current implementation always creates a 1 bit constant for the
result of the `G_ICMP`, which will cause issues if the destination
register size is larger than that. With asserts enabled, it will cause a
crash in `buildConstant`:
```
llvm/lib/CodeGen/GlobalISel/MachineIRBuilder.cpp:322: virtual MachineInstrBuilder llvm::MachineIRBuilder::buildConstant(const DstOp &, const ConstantInt &): Assertion `EltTy.getScalarSizeInBits() == Val.getBitWidth() && "creating constant with the wrong size"' failed.
```
This reverts commit a9699a334bc9666570418a3bed9520bcdc21518b.
Breaks CodeGen/AMDGPU/collapse-endcf.ll in several configs
(sanitizer builds; macOS; possibly more), see comments on
https://github.com/llvm/llvm-project/pull/138434
