6918314918
If a test took more than one attempt to complete, show the number of attempts and the maximum allowed attempts as `2 of 4 attempts` inside the `<progress info>` (see [TEST RUN OUTPUT FORMAT](https://llvm.org/docs/CommandGuide/lit.html#test-run-output-format)). NOTE: Additionally this is a fixup for #141851 where the tests were not quite right. `max-retries-per-test/allow-retries-test_retry_attempts/test.py` was added but never used there. Now we're calling it. To correlate better between the test output and the test script I've used higher numbers of max allowed retries.
125 lines
5.9 KiB
Python
125 lines
5.9 KiB
Python
# Check the behavior of the ALLOW_RETRIES keyword.
|
|
|
|
# This test uses a file that's stable across retries of the test to fail and
|
|
# only succeed the fourth time it is retried.
|
|
#
|
|
# RUN: rm -f %t.counter
|
|
# RUN: %{lit} %{inputs}/allow-retries/succeeds-within-limit.py -Dcounter=%t.counter -Dpython=%{python} | FileCheck --check-prefix=CHECK-TEST1 %s
|
|
# CHECK-TEST1: Passed With Retry: 1
|
|
|
|
# Test that a per-file ALLOW_RETRIES overwrites the config-wide test_retry_attempts property, if any.
|
|
#
|
|
# RUN: rm -f %t.counter
|
|
# RUN: %{lit} %{inputs}/allow-retries/succeeds-within-limit.py -Dtest_retry_attempts=2 -Dcounter=%t.counter -Dpython=%{python} | FileCheck --check-prefix=CHECK-TEST2 %s
|
|
# CHECK-TEST2: Passed With Retry: 1
|
|
|
|
# This test does not succeed within the allowed retry limit
|
|
#
|
|
# Check that the execution trace isn't corrupt due to reprocessing the script
|
|
# multiple times (e.g., '%dbg(...)' processing used to accumulate across
|
|
# retries).
|
|
#
|
|
# RUN: not %{lit} %{inputs}/allow-retries/does-not-succeed-within-limit.py -v |\
|
|
# RUN: FileCheck --check-prefix=CHECK-TEST3 -match-full-lines %s
|
|
#
|
|
# CHECK-TEST3: FAIL: allow-retries :: does-not-succeed-within-limit.py (1 of 1, 4 of 4 attempts)
|
|
# CHECK-TEST3-NEXT: {{\**}} TEST 'allow-retries :: does-not-succeed-within-limit.py' FAILED {{\**}}
|
|
# CHECK-TEST3-NEXT: Exit Code: 1
|
|
# CHECK-TEST3-EMPTY:
|
|
# CHECK-TEST3-NEXT: Command Output (stdout):
|
|
# CHECK-TEST3-NEXT: --
|
|
# CHECK-TEST3-NEXT: # {{RUN}}: at line 3
|
|
# CHECK-TEST3-NEXT: false
|
|
# CHECK-TEST3-NEXT: # executed command: false
|
|
# CHECK-TEST3-NEXT: # note: command had no output on stdout or stderr
|
|
# CHECK-TEST3-NEXT: # error: command failed with exit status: 1
|
|
# CHECK-TEST3-EMPTY:
|
|
# CHECK-TEST3-NEXT: --
|
|
# CHECK-TEST3: Failed Tests (1):
|
|
# CHECK-TEST3: allow-retries :: does-not-succeed-within-limit.py
|
|
|
|
# This test should be UNRESOLVED since it has more than one ALLOW_RETRIES
|
|
# lines, and that is not allowed.
|
|
#
|
|
# RUN: not %{lit} %{inputs}/allow-retries/more-than-one-allow-retries-lines.py | FileCheck --check-prefix=CHECK-TEST4 %s
|
|
# CHECK-TEST4: Unresolved Tests (1):
|
|
# CHECK-TEST4: allow-retries :: more-than-one-allow-retries-lines.py
|
|
|
|
# This test does not provide a valid integer to the ALLOW_RETRIES keyword.
|
|
# It should be unresolved.
|
|
#
|
|
# RUN: not %{lit} %{inputs}/allow-retries/not-a-valid-integer.py | FileCheck --check-prefix=CHECK-TEST5 %s
|
|
# CHECK-TEST5: Unresolved Tests (1):
|
|
# CHECK-TEST5: allow-retries :: not-a-valid-integer.py
|
|
|
|
# This test checks that the config-wide test_retry_attempts property is used
|
|
# when no ALLOW_RETRIES keyword is present.
|
|
#
|
|
# RUN: rm -f %t.counter
|
|
# RUN: %{lit} %{inputs}/test_retry_attempts/test.py -Dcounter=%t.counter -Dpython=%{python} | FileCheck --check-prefix=CHECK-TEST6 %s
|
|
# CHECK-TEST6: Passed With Retry: 1
|
|
|
|
# This test checks that --per-test-coverage doesn't accumulate inserted
|
|
# LLVM_PROFILE_FILE= commands across retries.
|
|
#
|
|
# RUN: rm -f %t.counter
|
|
# RUN: %{lit} -a %{inputs}/test_retry_attempts/test.py --per-test-coverage\
|
|
# RUN: -Dcounter=%t.counter -Dpython=%{python} | \
|
|
# RUN: FileCheck --check-prefix=CHECK-TEST7 %s
|
|
# CHECK-TEST7: Command Output (stdout):
|
|
# CHECK-TEST7: # executed command: export LLVM_PROFILE_FILE=
|
|
# CHECK-TEST7-NOT: # executed command: export LLVM_PROFILE_FILE=
|
|
# CHECK-TEST7: Passed With Retry: 1
|
|
|
|
# This test only passes on the 4th try. Here we check that a test can be re-run when:
|
|
# * The "--max-retries-per-test" is specified high enough (7).
|
|
# * No ALLOW_RETRIES keyword is used in the test script.
|
|
# * No config.test_retry_attempts is adjusted in the test suite config file.
|
|
# RUN: rm -f %t.counter
|
|
# RUN: %{lit} %{inputs}/max-retries-per-test/no-allow-retries-no-test_retry_attempts/test.py \
|
|
# RUN: --max-retries-per-test=7 \
|
|
# RUN: -Dcounter=%t.counter \
|
|
# RUN: -Dpython=%{python} \
|
|
# RUN: | FileCheck --check-prefix=CHECK-TEST8 %s
|
|
# CHECK-TEST8: FLAKYPASS: no-allow-retries-no-test_retry_attempts :: test.py (1 of 1, 4 of 8 attempts)
|
|
# CHECK-TEST8: Passed With Retry: 1
|
|
|
|
# This test only passes on the 4th try. Here we check that a test can be re-run when:
|
|
# * The "--max-retries-per-test" is specified too low (2).
|
|
# * ALLOW_RETRIES is specified high enough (8)
|
|
# * No config.test_retry_attempts is adjusted in the test suite config file.
|
|
# RUN: rm -f %t.counter
|
|
# RUN: %{lit} %{inputs}/max-retries-per-test/allow-retries-no-test_retry_attempts/test.py \
|
|
# RUN: --max-retries-per-test=2 \
|
|
# RUN: -Dcounter=%t.counter \
|
|
# RUN: -Dpython=%{python} \
|
|
# RUN: | FileCheck --check-prefix=CHECK-TEST9 %s
|
|
# CHECK-TEST9: FLAKYPASS: allow-retries-no-test_retry_attempts :: test.py (1 of 1, 4 of 9 attempts)
|
|
# CHECK-TEST9: Passed With Retry: 1
|
|
|
|
# This test only passes on the 4th try. Here we check that a test can be re-run when:
|
|
# * The "--max-retries-per-test" is specified too low (2).
|
|
# * No ALLOW_RETRIES keyword is used in the test script.
|
|
# * config.test_retry_attempts is set high enough (9).
|
|
# RUN: rm -f %t.counter
|
|
# RUN: %{lit} %{inputs}/max-retries-per-test/no-allow-retries-test_retry_attempts/test.py \
|
|
# RUN: --max-retries-per-test=2 \
|
|
# RUN: -Dcounter=%t.counter \
|
|
# RUN: -Dpython=%{python} \
|
|
# RUN: | FileCheck --check-prefix=CHECK-TEST10 %s
|
|
# CHECK-TEST10: FLAKYPASS: no-allow-retries-test_retry_attempts :: test.py (1 of 1, 4 of 10 attempts)
|
|
# CHECK-TEST10: Passed With Retry: 1
|
|
|
|
# This test only passes on the 4th try. Here we check that a test can be re-run when:
|
|
# * The "--max-retries-per-test" is specified too low (1).
|
|
# * ALLOW_RETRIES keyword is set high enough (10)
|
|
# * config.test_retry_attempts is set too low (2).
|
|
# RUN: rm -f %t.counter
|
|
# RUN: %{lit} %{inputs}/max-retries-per-test/allow-retries-test_retry_attempts/test.py \
|
|
# RUN: --max-retries-per-test=1 \
|
|
# RUN: -Dcounter=%t.counter \
|
|
# RUN: -Dpython=%{python} \
|
|
# RUN: | FileCheck --check-prefix=CHECK-TEST11 %s
|
|
# CHECK-TEST11: FLAKYPASS: allow-retries-test_retry_attempts :: test.py (1 of 1, 4 of 11 attempts)
|
|
# CHECK-TEST11: Passed With Retry: 1
|