33 Commits

Author SHA1 Message Date
jacquesguan
e609417cdc [mlir][Math] Add more constant folder for Math ops.
This revision add constant folder for abs, copysign, ctlz, cttz and
ctpop.

Differential Revision: https://reviews.llvm.org/D122115
2022-03-22 10:23:15 +08:00
jacquesguan
26c95ae389 [mlir][Math] Add constant folder for sqrt.
Differential Revision: https://reviews.llvm.org/D121980
2022-03-18 16:01:44 +08:00
William S. Moses
164c7afaf5 [MLIR][Math] Add constant folder for powf
Constant fold powf, given two constant operands and a compatible type

Reviewed By: ftynse

Differential Revision: https://reviews.llvm.org/D121845
2022-03-17 14:19:47 -04:00
River Riddle
2f5715dc78 [mlir][NFC] Rename the old Standard dialect test directory to Func
The remanants of Standard was renamed to Func, but the test directory
remained named as Standard. In adidition to fixing the name, this commit
also moves the tests for operations not in the Func dialect to the proper
parent dialect test directory.
2022-03-01 13:48:34 -08:00
Tres Popp
f5efe28070 [mlir] Propagate NaNs in PolynomialApproximation
Previously, NaNs would be dropped in favor of bounded values which was
strictly incorrect. Now the min/max operation propagate this
information. Not all uses of min/max need this, but the given change
will help protect future additions, and this prevents the need for an
additional cmpf and select operation to handle NaNs.

Differential Revision: https://reviews.llvm.org/D120020
2022-02-18 09:25:36 +01:00
Adrian Kuegel
b122cbebec [mlir][Math] Fix NaN handling in Exp approximation
Differential Revision: https://reviews.llvm.org/D119832
2022-02-15 15:17:56 +01:00
Adrian Kuegel
87de451bc5 [mlir][Math] Fix NaN handling in ExpM1 approximation.
Differential Revision: https://reviews.llvm.org/D119822
2022-02-15 12:10:12 +01:00
Jacques Pienaar
bbddd19ec7 [mlir][math] Expand coverage of atan2 expansion
Reuse the higher precision F32 approximation for the F16 one (by expanding and
truncating). This is partly RFC as I'm not sure what the expectations are here
(e.g., these are only for F32 and should not be expanded, that reusing
higher-precision ones for lower precision is undesirable due to increased
compute cost and only approximations per exact type is preferred, or this is
appropriate [at least as fallback] but we need to see how to make it more
generic across all the patterns here).

Differential Revision: https://reviews.llvm.org/D118968
2022-02-08 15:00:39 -08:00
River Riddle
dec8af701f [mlir] Move SelectOp from Standard to Arithmetic
This is part of splitting up the standard dialect. See https://llvm.discourse.group/t/standard-dialect-the-final-chapter/ for discussion.

Differential Revision: https://reviews.llvm.org/D118648
2022-02-02 14:45:12 -08:00
Rob Suderman
2f9f9afa4e [mlir] Add polynomial approximation for atan and atan2
Implement a taylor series approximation for atan and add an atan2 lowering
that uses atan's appromation. This includes tests for edge cases and tests
for each quadrant.

Reviewed By: NatashaKnk

Differential Revision: https://reviews.llvm.org/D115682
2022-01-21 12:22:58 -08:00
William S. Moses
1773dddadf [MLIR][Math] Enable constant folding of ops
Enable constant folding of ops within the math dialect, and introduce constant folders for ceil and log2

Reviewed By: mehdi_amini

Differential Revision: https://reviews.llvm.org/D117085
2022-01-12 12:19:29 -05:00
William S. Moses
d2c547342c Revert "[MLIR][Math] Enable constant folding of ops"
This reverts commit 2f8b956ab6e63e57fbaa60749c9ba7752b071993.

There is a linker error for mlir-nvidia as seen on
https://lab.llvm.org/buildbot/#/builders/61/builds/19939.

As it's late for me here, I'm oing to rever this for now to be
investigated later.
2022-01-12 02:00:53 -05:00
William S. Moses
2f8b956ab6 [MLIR][Math] Enable constant folding of ops
Enable constant folding of ops within the math dialect, and introduce constant folders for ceil and log2

Reviewed By: mehdi_amini

Differential Revision: https://reviews.llvm.org/D117085
2022-01-12 01:55:48 -05:00
Eugene Zhulenev
627fa0b9a8 [mlir] MathApproximations: unroll virtual vectors into hardware vectors for ISA specific operation
Reviewed By: cota

Differential Revision: https://reviews.llvm.org/D112736
2021-10-28 12:52:04 -07:00
Alexander Belyaev
96cee29762 [mlir] Allow polynomial approximations for N-d vectors.
Polynomial approximation can be extented to support N-d vectors.
N-dimensional vectors are useful when vectorizing operations on N-dimensional
tiles. Before lowering to LLVM these vectors are usually unrolled or flattened
to 1-dimensional vectors.

Differential Revision: https://reviews.llvm.org/D112566
2021-10-26 20:50:00 +02:00
Boian Petkantchin
f1b922188e [MLIR][Math] Add erf to math dialect
Add math.erf lowering to libm call.
Add math.erf polynomial approximation.

Reviewed By: silvas, ezhulenev

Differential Revision: https://reviews.llvm.org/D112200
2021-10-25 18:30:17 +00:00
Emilio Cota
35553d452b [mlir] Add polynomial approximation for vectorized math::Rsqrt
This patch adds a polynomial approximation that matches the
approximation in Eigen.

Note that the approximation only applies to vectorized inputs;
the scalar rsqrt is left unmodified.

The approximation is protected with a flag since it emits an AVX2
intrinsic (generated via the X86Vector). This is the only reasonably
clean way that I could find to generate the exact approximation that
I wanted (i.e. an identical one to Eigen's).

I considered two alternatives:

1. Introduce a Rsqrt intrinsic in LLVM, which doesn't exist yet.
   I believe this is because there is no definition of Rsqrt that
   all backends could agree on, since hardware instructions that
   implement it have widely varying degrees of precision.
   This is something that the standard could mandate, but Rsqrt is
   not part of IEEE754, so I don't think this option is feasible.

2. Emit fdiv(1.0, sqrt) with fast math flags to allow reciprocal
   transformations. Although portable, this doesn't allow us
   to generate exactly the code we want; it is the LLVM backend,
   and not MLIR, who controls what code is generated based on the
   target CPU.

Reviewed By: ezhulenev

Differential Revision: https://reviews.llvm.org/D112192
2021-10-23 04:56:12 -07:00
Ahmed Taei
21f9e4a1ed Avoid infinity arithmetics when computing exp approximations
Otherwise this can result a poison value on some platforms see https://bugs.llvm.org/show_bug.cgi?id=51204

Reviewed By: ezhulenev

Differential Revision: https://reviews.llvm.org/D112115
2021-10-21 10:09:18 -07:00
Eugene Zhulenev
bf32bb7e05 [mlir] Update approximation range for Tanh operation
Use wider range for approximating Tanh to match results computed in Eigen with AVX.

Reviewed By: cota

Differential Revision: https://reviews.llvm.org/D112011
2021-10-18 10:57:31 -07:00
Mogball
a54f4eae0e [MLIR] Replace std ops with arith dialect ops
Precursor: https://reviews.llvm.org/D110200

Removed redundant ops from the standard dialect that were moved to the
`arith` or `math` dialects.

Renamed all instances of operations in the codebase and in tests.

Reviewed By: rriddle, jpienaar

Differential Revision: https://reviews.llvm.org/D110797
2021-10-13 03:07:03 +00:00
Chris Lattner
42431b8207 [tests] Make testsuite more resilient to "order of constant" changes. NFC. 2021-09-08 10:10:10 -07:00
Benjamin Kramer
c1ebefdf77 [mlir] Make polynomial approximation emit std instead of LLVM ops
This is a bit cleaner and removes issues with 2d vectors. It also has a
big impact on constant folding, hence the test changes.

Differential Revision: https://reviews.llvm.org/D107896
2021-08-11 16:37:21 +02:00
bakhtiyar
391456f33c Fix a bug in algebraic simplification, and enable the tests.
Reviewed By: ezhulenev

Differential Revision: https://reviews.llvm.org/D107788
2021-08-10 04:15:56 -07:00
Eugene Zhulenev
d94426d22a [mlir] Math: add algebraic simplification patterns to math transforms
Reviewed By: bkramer

Differential Revision: https://reviews.llvm.org/D106822
2021-07-27 09:22:33 -07:00
Emilio Cota
3c952ab25f [mlir] Check generated IR of math_polynomial_approx.mlir
Instead of just checking that we emit something.

Differential Revision: https://reviews.llvm.org/D101940
2021-05-05 16:42:48 -07:00
Emilio Cota
0edc4bc84a [mlir] Add polynomial approximation for math::ExpM1
This approximation matches the one in Eigen.

```
name                      old cpu/op  new cpu/op  delta
BM_mlir_Expm1_f32/10      90.9ns ± 4%  52.2ns ± 4%  -42.60%    (p=0.000 n=74+87)
BM_mlir_Expm1_f32/100      837ns ± 3%   231ns ± 4%  -72.43%    (p=0.000 n=79+69)
BM_mlir_Expm1_f32/1k      8.43µs ± 3%  1.58µs ± 5%  -81.30%    (p=0.000 n=77+83)
BM_mlir_Expm1_f32/10k     83.8µs ± 3%  15.4µs ± 5%  -81.65%    (p=0.000 n=83+69)
BM_eigen_s_Expm1_f32/10   68.8ns ±17%  72.5ns ±14%   +5.40%  (p=0.000 n=118+115)
BM_eigen_s_Expm1_f32/100   694ns ±11%   717ns ± 2%   +3.34%   (p=0.000 n=120+75)
BM_eigen_s_Expm1_f32/1k   7.69µs ± 2%  7.97µs ±11%   +3.56%   (p=0.000 n=95+117)
BM_eigen_s_Expm1_f32/10k  88.0µs ± 1%  89.3µs ± 6%   +1.45%   (p=0.000 n=74+106)
BM_eigen_v_Expm1_f32/10   44.3ns ± 6%  45.0ns ± 8%   +1.45%   (p=0.018 n=81+111)
BM_eigen_v_Expm1_f32/100   351ns ± 1%   360ns ± 9%   +2.58%    (p=0.000 n=73+99)
BM_eigen_v_Expm1_f32/1k   3.31µs ± 1%  3.42µs ± 9%   +3.37%   (p=0.000 n=71+100)
BM_eigen_v_Expm1_f32/10k  33.7µs ± 8%  34.1µs ± 9%   +1.04%    (p=0.007 n=99+98)
```

Reviewed By: ezhulenev

Differential Revision: https://reviews.llvm.org/D101852
2021-05-05 14:31:34 -07:00
Emilio Cota
1c0374e770 [mlir] Add polynomial approximation for math::Log1p
This approximation matches the one in Eigen.

```
name                      old cpu/op  new cpu/op  delta
BM_mlir_Log1p_f32/10      83.2ns ± 7%  34.8ns ± 5%  -58.19%    (p=0.000 n=84+71)
BM_mlir_Log1p_f32/100      664ns ± 4%   129ns ± 4%  -80.57%    (p=0.000 n=82+82)
BM_mlir_Log1p_f32/1k      6.75µs ± 4%  0.81µs ± 3%  -88.07%    (p=0.000 n=88+79)
BM_mlir_Log1p_f32/10k     76.5µs ± 3%   7.8µs ± 4%  -89.84%    (p=0.000 n=80+80)
BM_eigen_s_Log1p_f32/10   70.1ns ±14%  72.6ns ±14%   +3.49%  (p=0.000 n=116+112)
BM_eigen_s_Log1p_f32/100   706ns ± 9%   717ns ± 3%   +1.60%   (p=0.018 n=117+80)
BM_eigen_s_Log1p_f32/1k   8.26µs ± 1%  8.26µs ± 1%     ~       (p=0.567 n=84+86)
BM_eigen_s_Log1p_f32/10k  92.1µs ± 5%  92.6µs ± 6%   +0.60%  (p=0.047 n=115+115)
BM_eigen_v_Log1p_f32/10   31.8ns ±24%  34.9ns ±17%   +9.72%    (p=0.000 n=98+96)
BM_eigen_v_Log1p_f32/100   169ns ±10%   177ns ± 5%   +4.66%   (p=0.000 n=119+81)
BM_eigen_v_Log1p_f32/1k   1.42µs ± 4%  1.46µs ± 8%   +2.70%   (p=0.000 n=93+113)
BM_eigen_v_Log1p_f32/10k  14.4µs ± 5%  14.9µs ± 8%   +3.61%  (p=0.000 n=115+110)
```

Reviewed By: ezhulenev, ftynse

Differential Revision: https://reviews.llvm.org/D101765
2021-05-03 15:11:37 -07:00
Emilio Cota
c0891706bc [mlir] Add polynomial approximation for math::Log2
```
name                     old cpu/op  new cpu/op  delta
BM_mlir_Log2_f32/10       134ns ±15%    45ns ± 4%  -66.39%  (p=0.000 n=20+17)
BM_mlir_Log2_f32/100     1.03µs ±16%  0.12µs ±10%  -88.78%  (p=0.000 n=20+18)
BM_mlir_Log2_f32/1k      10.3µs ±16%   0.7µs ± 5%  -93.24%  (p=0.000 n=20+17)
BM_mlir_Log2_f32/10k      104µs ±15%     7µs ±14%  -93.25%  (p=0.000 n=20+20)
BM_eigen_s_Log2_f32/10   95.3ns ±17%  90.9ns ± 6%     ~     (p=0.228 n=20+18)
BM_eigen_s_Log2_f32/100   907ns ± 3%   911ns ± 6%     ~     (p=0.539 n=16+20)
BM_eigen_s_Log2_f32/1k   9.88µs ± 4%  9.85µs ± 3%     ~     (p=0.790 n=16+17)
BM_eigen_s_Log2_f32/10k   105µs ±10%   110µs ±16%     ~     (p=0.459 n=16+20)
BM_eigen_v_Log2_f32/10   32.5ns ±31%  33.9ns ±14%   +4.31%  (p=0.028 n=17+20)
BM_eigen_v_Log2_f32/100   176ns ± 8%   180ns ± 7%   +2.19%  (p=0.045 n=16+17)
BM_eigen_v_Log2_f32/1k   1.44µs ± 4%  1.50µs ± 9%   +3.91%  (p=0.001 n=16+17)
BM_eigen_v_Log2_f32/10k  14.5µs ±10%  15.0µs ± 8%   +3.92%  (p=0.002 n=16+19)
```

Reviewed By: ezhulenev

Differential Revision: https://reviews.llvm.org/D98282
2021-03-10 14:49:22 -08:00
Ahmed Taei
ea7f211b2e [mlir] Add polynomial approximation for math::ExpOp
Similar to fast_exp in https://github.com/boulos/syrah

Differential Revision: https://reviews.llvm.org/D97599
2021-03-01 11:02:07 -08:00
Eugene Zhulenev
ce976d2db3 [mlir] Add polynomial approximation for math::LogOp (using builders API)
Replace math::LogOp with an approximations from the the Julien Pommier's SSE math library

Link: http://gruntthepeon.free.fr/ssemath

Reviewed By: asaadaldien

Differential Revision: https://reviews.llvm.org/D97304
2021-02-24 07:50:25 -08:00
Eugene Zhulenev
f99ccf6516 [mlir] Add math polynomial approximation pass
This gives ~30x speedup compared to expanding Tanh into exp operations:

```
name                  old cpu/op  new cpu/op  delta
BM_mlir_Tanh_f32/10    253ns ± 3%    55ns ± 7%  -78.35%  (p=0.000 n=44+41)
BM_mlir_Tanh_f32/100  2.21µs ± 4%  0.14µs ± 8%  -93.85%  (p=0.000 n=48+49)
BM_mlir_Tanh_f32/1k   22.6µs ± 4%   0.7µs ± 5%  -96.68%  (p=0.000 n=32+42)
BM_mlir_Tanh_f32/10k   225µs ± 5%     7µs ± 6%  -96.88%  (p=0.000 n=49+55)

name                  old time/op             new time/op             delta
BM_mlir_Tanh_f32/10    259ns ± 1%               56ns ± 2%  -78.31%        (p=0.000 n=41+39)
BM_mlir_Tanh_f32/100  2.27µs ± 1%             0.14µs ± 5%  -93.89%        (p=0.000 n=46+49)
BM_mlir_Tanh_f32/1k   22.9µs ± 1%              0.8µs ± 4%  -96.67%        (p=0.000 n=30+42)
BM_mlir_Tanh_f32/10k   230µs ± 0%                7µs ± 3%  -96.88%        (p=0.000 n=37+55)
```

This approximations is based on Eigen::generic_fast_tanh function

Reviewed By: mehdi_amini

Differential Revision: https://reviews.llvm.org/D96739
2021-02-19 12:43:36 -08:00
Adrian Kuegel
9f581815ae Add Expm1 op to the math dialect.
Differential Revision: https://reviews.llvm.org/D96704
2021-02-16 08:33:37 +01:00
Stephan Herhut
4348d8ab7f [mlir][math] Split off the math dialect.
This does not split transformations, yet. Those will be done as future clean ups.

Differential Revision: https://reviews.llvm.org/D96272
2021-02-12 10:55:12 +01:00