shylie/llvm-project
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
llvm-project/openmp/tools/omptest/test/unittests/asserter-seq-test.cpp
Michael Halkenhäuser 7c1d2467f1
Reland: [OpenMP] Add ompTest library to OpenMP (#154786) 
Reland of https://github.com/llvm/llvm-project/pull/147381

Added changes to fix observed BuildBot failures:
 * CMake version (reduced minimum to `3.20`, was: `3.22`)
 * GoogleTest linking (missing `./build/lib/libllvm_gtest.a`)
* Related header issue (missing `#include
"llvm/Support/raw_os_ostream.h"`)

Original message

Description
===========
OpenMP Tooling Interface Testing Library (ompTest) ompTest is a unit testing framework for testing OpenMP implementations. It offers a simple-to-use framework that allows a tester to check for OMPT events in addition to regular unit testing code, supported by linking against GoogleTest by default. It also facilitates writing concise tests while bridging the semantic gap between the unit under test and the OMPT-event testing.

Background
==========
This library has been developed to provide the means of testing OMPT implementations with reasonable effort. Especially, asynchronous or unordered events are supported and can be verified with ease, which may prove to be challenging with LIT-based tests. Additionally, since the assertions are part of the code being tested, ompTest can reference all corresponding variables during assertion.

Basic Usage
===========
OMPT event assertions are placed before the code, which shall be tested. These assertion can either be provided as one block or interleaved with the test code. There are two types of asserters: (1) sequenced "order-sensitive" and (2) set "unordered" assserters. Once the test is being run, the corresponding events are triggered by the OpenMP runtime and can be observed. Each of these observed events notifies asserters, which then determine if the test should pass or fail.

Example (partial, interleaved)
==============================
```c++
  int N = 100000;
  int a[N];
  int b[N];

  OMPT_ASSERT_SEQUENCE(Target, TARGET, BEGIN, 0);
  OMPT_ASSERT_SEQUENCE(TargetDataOp, ALLOC, N * sizeof(int)); // a ?
  OMPT_ASSERT_SEQUENCE(TargetDataOp, H2D, N * sizeof(int), &a);
  OMPT_ASSERT_SEQUENCE(TargetDataOp, ALLOC, N * sizeof(int)); // b ?
  OMPT_ASSERT_SEQUENCE(TargetDataOp, H2D, N * sizeof(int), &b);
  OMPT_ASSERT_SEQUENCE(TargetSubmit, 1);
  OMPT_ASSERT_SEQUENCE(TargetDataOp, D2H, N * sizeof(int), nullptr, &b);
  OMPT_ASSERT_SEQUENCE(TargetDataOp, D2H, N * sizeof(int), nullptr, &a);
  OMPT_ASSERT_SEQUENCE(TargetDataOp, DELETE);
  OMPT_ASSERT_SEQUENCE(TargetDataOp, DELETE);
  OMPT_ASSERT_SEQUENCE(Target, TARGET, END, 0);

#pragma omp target parallel for
  {
    for (int j = 0; j < N; j++)
      a[j] = b[j];
  }
```

References
==========
This work has been presented at SC'24 workshops, see: https://ieeexplore.ieee.org/document/10820689

Current State and Future Work
=============================
ompTest's development was mostly device-centric and aimed at OMPT device callbacks and device-side tracing. Consequentially, a substantial part of host-related events or features may not be supported in its current state. However, we are confident that the related functionality can be added and ompTest provides a general foundation for future OpenMP and especially OMPT testing. This PR will allow us to upstream the corresponding features, like OMPT device-side tracing in the future with significantly reduced risk of introducing regressions in the process.

Build
=====
ompTest is linked against LLVM's GoogleTest by default, but can also be built 'standalone'. Additionally, it comes with a set of unit tests, which in turn require GoogleTest (overriding a standalone build). The unit tests are added to the `check-openmp` target.

Use the following parameters to perform the corresponding build: 
`LIBOMPTEST_BUILD_STANDALONE` (Default: ${OPENMP_STANDALONE_BUILD})
`LIBOMPTEST_BUILD_UNITTESTS` (Default: OFF)

---------

Co-authored-by: Jan-Patrick Lehr <JanPatrick.Lehr@amd.com>
Co-authored-by: Joachim <protze@rz.rwth-aachen.de>
Co-authored-by: Joachim Jenke <jenke@itc.rwth-aachen.de>
2025-08-22 13:56:12 +02:00

359 lines
14 KiB
C++
Raw Blame History

#include "OmptAliases.h"
#include "OmptAsserter.h"
#include <omp-tools.h>
#include <sstream>
#include "gtest/gtest.h"
using namespace omptest;
using OAE = omptest::OmptAssertEvent;
using OS = omptest::ObserveState;
/// SequencedAsserter test-fixture class to avoid code duplication among tests.
class OmptSequencedAsserterTest : public testing::Test {
protected:
OmptSequencedAsserterTest() {
// Construct default sequenced asserter
SeqAsserter = std::make_unique<omptest::OmptSequencedAsserter>();
// Silence all potential log prints
SeqAsserter->getLog()->setLoggingLevel(logging::Level::Critical);
}
std::unique_ptr<omptest::OmptSequencedAsserter> SeqAsserter;
};
TEST_F(OmptSequencedAsserterTest, DefaultState) {
// Assertion should neither start as 'deactivated' nor 'suspended'
ASSERT_EQ(SeqAsserter->isActive(), true);
ASSERT_EQ(SeqAsserter->AssertionSuspended, false);
// Assertion should begin with event ID zero
ASSERT_EQ(SeqAsserter->NextEvent, 0);
// Assertion should begin without previous notifications or assertions
ASSERT_EQ(SeqAsserter->getNotificationCount(), 0);
ASSERT_EQ(SeqAsserter->getSuccessfulAssertionCount(), 0);
// There should be no expected events
ASSERT_EQ(SeqAsserter->Events.empty(), true);
ASSERT_EQ(SeqAsserter->getRemainingEventCount(), 0);
// Default mode should be Strict
ASSERT_NE(SeqAsserter->getOperationMode(), AssertMode::Relaxed);
ASSERT_EQ(SeqAsserter->getOperationMode(), AssertMode::Strict);
// Default state should be passing
ASSERT_NE(SeqAsserter->getState(), AssertState::Fail);
ASSERT_EQ(SeqAsserter->getState(), AssertState::Pass);
ASSERT_NE(SeqAsserter->checkState(), AssertState::Fail);
ASSERT_EQ(SeqAsserter->checkState(), AssertState::Pass);
}
TEST_F(OmptSequencedAsserterTest, IgnoreNotificationsWhenEmpty) {
// ParallelBegin events are suppressed by default
auto SuppressedEvent = OAE::ParallelBegin(
/*Name=*/"ParBegin", /*Group=*/"", /*Expected=*/OS::Always,
/*NumThreads=*/3);
// DeviceFinalize events are not ignored by default
auto IgnoredEvent = OAE::DeviceFinalize(
/*Name=*/"DevFini", /*Group=*/"", /*Expected=*/OS::Always,
/*DeviceNum=*/7);
// Situation: There is nothing to assert.
// Result: All notifications are ignored.
// Hence, check that the perceived count of notifications remains unchanged
ASSERT_EQ(SeqAsserter->getNotificationCount(), 0);
SeqAsserter->notify(std::move(SuppressedEvent));
ASSERT_EQ(SeqAsserter->getNotificationCount(), 0);
SeqAsserter->notify(std::move(IgnoredEvent));
ASSERT_EQ(SeqAsserter->getNotificationCount(), 0);
ASSERT_EQ(SeqAsserter->getSuccessfulAssertionCount(), 0);
ASSERT_EQ(SeqAsserter->checkState(), AssertState::Pass);
}
TEST_F(OmptSequencedAsserterTest, IgnoreNotificationsWhileDeactivated) {
auto ExpectedEvent = OAE::DeviceUnload(
/*Name=*/"DevUnload", /*Group=*/"", /*Expected=*/OS::Always);
SeqAsserter->insert(std::move(ExpectedEvent));
ASSERT_EQ(SeqAsserter->Events.empty(), false);
// Deactivate asserter, effectively ignoring notifications
SeqAsserter->setActive(false);
ASSERT_EQ(SeqAsserter->isActive(), false);
ASSERT_EQ(SeqAsserter->getNotificationCount(), 0);
// DeviceFinalize events are not ignored by default
auto IgnoredEvent = OAE::DeviceFinalize(
/*Name=*/"DevFini", /*Group=*/"", /*Expected=*/OS::Always,
/*DeviceNum=*/7);
SeqAsserter->notify(std::move(IgnoredEvent));
// Assertion was deactivated: No change
ASSERT_EQ(SeqAsserter->getNotificationCount(), 0);
ASSERT_EQ(SeqAsserter->getSuccessfulAssertionCount(), 0);
SeqAsserter->setActive(true);
ASSERT_EQ(SeqAsserter->isActive(), true);
auto ObservedEvent = OAE::DeviceUnload(
/*Name=*/"DevFini", /*Group=*/"", /*Expected=*/OS::Always);
SeqAsserter->notify(std::move(ObservedEvent));
// Assertion was activated, one notification expected
ASSERT_EQ(SeqAsserter->getNotificationCount(), 1);
ASSERT_EQ(SeqAsserter->getSuccessfulAssertionCount(), 1);
ASSERT_EQ(SeqAsserter->checkState(), AssertState::Pass);
}
TEST_F(OmptSequencedAsserterTest, AddEvent) {
ASSERT_EQ(SeqAsserter->getRemainingEventCount(), 0);
auto ExpectedEvent = OAE::DeviceFinalize(
/*Name=*/"DevFini", /*Group=*/"", /*Expected=*/OS::Always,
/*DeviceNum=*/7);
SeqAsserter->insert(std::move(ExpectedEvent));
// Sanity check: Notifications should not be triggered
ASSERT_EQ(SeqAsserter->getNotificationCount(), 0);
// Adding an expected event must change the event count but not the state
ASSERT_EQ(SeqAsserter->getRemainingEventCount(), 1);
ASSERT_EQ(SeqAsserter->getSuccessfulAssertionCount(), 0);
ASSERT_EQ(SeqAsserter->getState(), AssertState::Pass);
}
TEST_F(OmptSequencedAsserterTest, AddEventIgnoreSuppressed) {
auto ExpectedEvent = OAE::DeviceFinalize(
/*Name=*/"DevFini", /*Group=*/"", /*Expected=*/OS::Always,
/*DeviceNum=*/7);
SeqAsserter->insert(std::move(ExpectedEvent));
// ParallelBegin events are suppressed by default
auto SuppressedEvent = OAE::ParallelBegin(
/*Name=*/"ParBegin", /*Group=*/"", /*Expected=*/OS::Always,
/*NumThreads=*/3);
// Situation: There is one expected event and ParallelBegins are suppressed.
// Notification count remains unchanged for suppressed events
ASSERT_EQ(SeqAsserter->getNotificationCount(), 0);
SeqAsserter->notify(std::move(SuppressedEvent));
ASSERT_EQ(SeqAsserter->getNotificationCount(), 0);
ASSERT_EQ(SeqAsserter->getSuccessfulAssertionCount(), 0);
ASSERT_EQ(SeqAsserter->getState(), AssertState::Pass);
}
TEST_F(OmptSequencedAsserterTest, AddEventObservePass) {
auto ExpectedEvent = OAE::DeviceFinalize(
/*Name=*/"DevFini", /*Group=*/"", /*Expected=*/OS::Always,
/*DeviceNum=*/7);
SeqAsserter->insert(std::move(ExpectedEvent));
// DeviceFinalize events are not ignored by default
auto ObservedEvent = OAE::DeviceFinalize(
/*Name=*/"DevFini", /*Group=*/"", /*Expected=*/OS::Always,
/*DeviceNum=*/7);
SeqAsserter->notify(std::move(ObservedEvent));
ASSERT_EQ(SeqAsserter->getNotificationCount(), 1);
ASSERT_EQ(SeqAsserter->getSuccessfulAssertionCount(), 1);
ASSERT_EQ(SeqAsserter->checkState(), AssertState::Pass);
}
TEST_F(OmptSequencedAsserterTest, AddEventObserveFail) {
auto ExpectedEvent = OAE::DeviceFinalize(
/*Name=*/"DevFini", /*Group=*/"", /*Expected=*/OS::Always,
/*DeviceNum=*/7);
SeqAsserter->insert(std::move(ExpectedEvent));
// DeviceFinalize events are not ignored by default
// Provide wrong DeviceNum
auto ObservedEvent = OAE::DeviceFinalize(
/*Name=*/"DevFini", /*Group=*/"", /*Expected=*/OS::Always,
/*DeviceNum=*/23);
SeqAsserter->notify(std::move(ObservedEvent));
ASSERT_EQ(SeqAsserter->getNotificationCount(), 1);
// Observed and expected event do not match: Fail
ASSERT_EQ(SeqAsserter->getSuccessfulAssertionCount(), 0);
ASSERT_EQ(SeqAsserter->checkState(), AssertState::Fail);
}
TEST_F(OmptSequencedAsserterTest, AddEventObserveDifferentType) {
auto ExpectedEvent = OAE::DeviceUnload(
/*Name=*/"DevUnload", /*Group=*/"", /*Expected=*/OS::Always);
SeqAsserter->insert(std::move(ExpectedEvent));
// DeviceFinalize events are not ignored by default
auto ObservedEvent = OAE::DeviceFinalize(
/*Name=*/"DevFini", /*Group=*/"", /*Expected=*/OS::Always,
/*DeviceNum=*/7);
SeqAsserter->notify(std::move(ObservedEvent));
ASSERT_EQ(SeqAsserter->getNotificationCount(), 1);
// Observed and expected event do not match: Fail
ASSERT_EQ(SeqAsserter->getSuccessfulAssertionCount(), 0);
ASSERT_EQ(SeqAsserter->checkState(), AssertState::Fail);
}
TEST_F(OmptSequencedAsserterTest, CheckTargetGroupNoEffect) {
// Situation: Groups are designed to be used as an indicator -WITHIN- target
// regions. Hence, comparing two target regions w.r.t. their groups has no
// effect on pass or fail.
auto ExpectedEvent = OAE::Target(
/*Name=*/"Target", /*Group=*/"MyTargetGroup", /*Expected=*/OS::Always,
/*Kind=*/TARGET, /*Endpoint=*/BEGIN,
/*DeviceNum=*/7, /*TaskData=*/nullptr, /*TargetId=*/23,
/*CodeptrRA=*/nullptr);
SeqAsserter->insert(std::move(ExpectedEvent));
ASSERT_EQ(SeqAsserter->Events.empty(), false);
// Deactivate asserter, effectively ignoring notifications
SeqAsserter->setActive(false);
ASSERT_EQ(SeqAsserter->isActive(), false);
ASSERT_EQ(SeqAsserter->getNotificationCount(), 0);
// Target events are not ignored by default
auto ObservedEvent = OAE::Target(
/*Name=*/"Target", /*Group=*/"MyTargetGroup", /*Expected=*/OS::Always,
/*Kind=*/TARGET, /*Endpoint=*/BEGIN, /*DeviceNum=*/7,
/*TaskData=*/nullptr, /*TargetId=*/23, /*CodeptrRA=*/nullptr);
SeqAsserter->notify(std::move(ObservedEvent));
// Assertion was deactivated: No change
ASSERT_EQ(SeqAsserter->getNotificationCount(), 0);
ASSERT_EQ(SeqAsserter->getSuccessfulAssertionCount(), 0);
ASSERT_EQ(SeqAsserter->getRemainingEventCount(), 1);
// Re-activate asserter
SeqAsserter->setActive(true);
ASSERT_EQ(SeqAsserter->isActive(), true);
ASSERT_EQ(SeqAsserter->getNotificationCount(), 0);
// Actually observe a target event from "AnotherGroup"
auto AnotherObservedEvent = OAE::Target(
/*Name=*/"Target", /*Group=*/"AnotherGroup", /*Expected=*/OS::Always,
/*Kind=*/TARGET, /*Endpoint=*/BEGIN, /*DeviceNum=*/7,
/*TaskData=*/nullptr, /*TargetId=*/23, /*CodeptrRA=*/nullptr);
SeqAsserter->notify(std::move(AnotherObservedEvent));
// Observed all expected events; groups of target regions do not affect pass
ASSERT_EQ(SeqAsserter->getNotificationCount(), 1);
ASSERT_EQ(SeqAsserter->getSuccessfulAssertionCount(), 1);
ASSERT_EQ(SeqAsserter->getRemainingEventCount(), 0);
ASSERT_EQ(SeqAsserter->checkState(), AssertState::Pass);
}
TEST_F(OmptSequencedAsserterTest, CheckSyncPoint) {
auto ExpectedEvent = OAE::Target(
/*Name=*/"Target", /*Group=*/"MyTargetGroup", /*Expected=*/OS::Always,
/*Kind=*/TARGET, /*Endpoint=*/BEGIN,
/*DeviceNum=*/7, /*TaskData=*/nullptr, /*TargetId=*/23,
/*CodeptrRA=*/nullptr);
SeqAsserter->insert(std::move(ExpectedEvent));
ASSERT_EQ(SeqAsserter->Events.empty(), false);
ASSERT_EQ(SeqAsserter->getRemainingEventCount(), 1);
// Target events are not ignored by default
auto ObservedEvent = OAE::Target(
/*Name=*/"Target", /*Group=*/"MyTargetGroup", /*Expected=*/OS::Always,
/*Kind=*/TARGET, /*Endpoint=*/BEGIN, /*DeviceNum=*/7,
/*TaskData=*/nullptr, /*TargetId=*/23, /*CodeptrRA=*/nullptr);
SeqAsserter->notify(std::move(ObservedEvent));
ASSERT_EQ(SeqAsserter->getNotificationCount(), 1);
SeqAsserter->notify(OAE::AssertionSyncPoint(
/*Name=*/"", /*Group=*/"", /*Expected=*/OS::Always,
/*SyncPointName=*/"SyncPoint 1"));
ASSERT_EQ(SeqAsserter->getNotificationCount(), 2);
ASSERT_EQ(SeqAsserter->getSuccessfulAssertionCount(), 1);
// All events processed: SyncPoint "passes"
ASSERT_EQ(SeqAsserter->checkState(), AssertState::Pass);
auto AnotherExpectedEvent = OAE::Target(
/*Name=*/"Target", /*Group=*/"MyTargetGroup", /*Expected=*/OS::Always,
/*Kind=*/TARGET, /*Endpoint=*/BEGIN,
/*DeviceNum=*/7, /*TaskData=*/nullptr, /*TargetId=*/23,
/*CodeptrRA=*/nullptr);
ASSERT_EQ(SeqAsserter->getRemainingEventCount(), 0);
SeqAsserter->insert(std::move(AnotherExpectedEvent));
ASSERT_EQ(SeqAsserter->getRemainingEventCount(), 1);
// Remaining events present: SyncPoint "fails"
SeqAsserter->notify(OAE::AssertionSyncPoint(
/*Name=*/"", /*Group=*/"", /*Expected=*/OS::Always,
/*SyncPointName=*/"SyncPoint 2"));
ASSERT_EQ(SeqAsserter->checkState(), AssertState::Fail);
}
TEST_F(OmptSequencedAsserterTest, CheckExcessNotify) {
auto ExpectedEvent = OAE::Target(
/*Name=*/"Target", /*Group=*/"MyTargetGroup", /*Expected=*/OS::Always,
/*Kind=*/TARGET, /*Endpoint=*/BEGIN,
/*DeviceNum=*/7, /*TaskData=*/nullptr, /*TargetId=*/23,
/*CodeptrRA=*/nullptr);
SeqAsserter->insert(std::move(ExpectedEvent));
ASSERT_EQ(SeqAsserter->Events.empty(), false);
ASSERT_EQ(SeqAsserter->getRemainingEventCount(), 1);
// Target events are not ignored by default
auto ObservedEvent = OAE::Target(
/*Name=*/"Target", /*Group=*/"MyTargetGroup", /*Expected=*/OS::Always,
/*Kind=*/TARGET, /*Endpoint=*/BEGIN, /*DeviceNum=*/7,
/*TaskData=*/nullptr, /*TargetId=*/23, /*CodeptrRA=*/nullptr);
SeqAsserter->notify(std::move(ObservedEvent));
ASSERT_EQ(SeqAsserter->getNotificationCount(), 1);
// All events processed: pass
ASSERT_EQ(SeqAsserter->checkState(), AssertState::Pass);
// Target events are not ignored by default
auto AnotherObservedEvent = OAE::Target(
/*Name=*/"Target", /*Group=*/"MyTargetGroup", /*Expected=*/OS::Always,
/*Kind=*/TARGET, /*Endpoint=*/BEGIN, /*DeviceNum=*/7,
/*TaskData=*/nullptr, /*TargetId=*/23, /*CodeptrRA=*/nullptr);
// No more events expected: notify "fails"
SeqAsserter->notify(std::move(AnotherObservedEvent));
ASSERT_EQ(SeqAsserter->getNotificationCount(), 2);
ASSERT_EQ(SeqAsserter->checkState(), AssertState::Fail);
}
TEST_F(OmptSequencedAsserterTest, CheckSuspend) {
SeqAsserter->insert(OAE::AssertionSuspend(
/*Name=*/"", /*Group=*/"", /*Expected=*/OS::Never));
ASSERT_EQ(SeqAsserter->Events.empty(), false);
// Being notified while the next expected event is a "suspend" should change
// the asserter's state
ASSERT_EQ(SeqAsserter->getNotificationCount(), 0);
ASSERT_EQ(SeqAsserter->AssertionSuspended, false);
SeqAsserter->notify(OAE::DeviceFinalize(
/*Name=*/"DevFini", /*Group=*/"", /*Expected=*/OS::Always,
/*DeviceNum=*/7));
ASSERT_EQ(SeqAsserter->AssertionSuspended, true);
ASSERT_EQ(SeqAsserter->getNotificationCount(), 1);
ASSERT_EQ(SeqAsserter->getSuccessfulAssertionCount(), 0);
ASSERT_EQ(SeqAsserter->checkState(), AssertState::Pass);
auto ExpectedEvent = OAE::Target(
/*Name=*/"Target", /*Group=*/"MyTargetGroup", /*Expected=*/OS::Always,
/*Kind=*/TARGET, /*Endpoint=*/BEGIN,
/*DeviceNum=*/7, /*TaskData=*/nullptr, /*TargetId=*/23,
/*CodeptrRA=*/nullptr);
SeqAsserter->insert(std::move(ExpectedEvent));
// Being notified with an observed event, which matches the next expected
// event, resumes assertion (suspended = false)
ASSERT_EQ(SeqAsserter->AssertionSuspended, true);
SeqAsserter->notify(OAE::Target(
/*Name=*/"Target", /*Group=*/"MyTargetGroup", /*Expected=*/OS::Always,
/*Kind=*/TARGET, /*Endpoint=*/BEGIN,
/*DeviceNum=*/7, /*TaskData=*/nullptr, /*TargetId=*/23,
/*CodeptrRA=*/nullptr));
ASSERT_EQ(SeqAsserter->AssertionSuspended, false);
ASSERT_EQ(SeqAsserter->getNotificationCount(), 2);
ASSERT_EQ(SeqAsserter->getSuccessfulAssertionCount(), 1);
ASSERT_EQ(SeqAsserter->checkState(), AssertState::Pass);
}
Reference in New Issue View Git Blame Copy Permalink