Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 13 additions & 5 deletions tests/ctst/HOW_TO_WRITE_TESTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,13 +36,16 @@ restart of multiple services, must be created before the test starts, or in a
dedicated set of tests, that is, a set of test executed before all tests having
a specific tag used for identifying the feature(s).

Note: Testing the reconfiguration of the environment is recommended, but it
should be done carefully to not affect other tests.
When a scenario *must* reconfigure the environment (e.g., create or modify
locations, add overlay endpoints), tag it with `@Exclusive`. This ensures no
other scenario runs in parallel while the exclusive scenario executes, and the
exclusive scenario only starts once all running scenarios have finished. See
the implementation in `common/hooks.ts`.

## 4. Do not use `atMostOnePicklePerTag`.
## 4. Avoid unnecessary parallel restrictions.

If a set of scenario requires the use of `atMostOnePicklePerTag`, they won't
be executed in parallel, which is:
Adding `atMostOnePicklePerTag` for a set of scenarios means they won't be
executed in parallel, which is:

- Not realistic for the production environment.
- Not efficient for the test execution: the duration of tests will suffer.
Expand All @@ -59,6 +62,11 @@ possible. Solutions exist:
relative checks.
- As a last resort, we might have a dedicated test suite.

Note: `@Exclusive` (see **Rule #3**) is an exception — it is reserved for
scenarios that mutate cluster-wide state (location creation, overlay changes)
and cannot be made idempotent because the operator reconciliation affects all
running pods. Use it sparingly.

## 5. Focus on validating features.

We only want to assert against externally visible state, as given in the
Expand Down
16 changes: 15 additions & 1 deletion tests/ctst/common/hooks.ts
Original file line number Diff line number Diff line change
Expand Up @@ -32,7 +32,21 @@ const noParallelRun = atMostOnePicklePerTag([
...replicationLockTags
]);

setParallelCanAssign(noParallelRun);
const EXCLUSIVE_TAG = '@Exclusive';

function hasTag(pickle: { tags: readonly { name: string }[] }, tagName: string): boolean {
return pickle.tags.some(t => t.name === tagName);
}

setParallelCanAssign((pickle, runningPickles) => {
Comment thread
francoisferrand marked this conversation as resolved.
if (runningPickles.some(p => hasTag(p, EXCLUSIVE_TAG))) {
return false;
}
if (hasTag(pickle, EXCLUSIVE_TAG)) {
return runningPickles.length === 0;
}
return noParallelRun(pickle, runningPickles);
});

Before(async function (this: Zenko, scenario: ITestCaseHookParameter) {
this.resetSaved();
Expand Down
1 change: 1 addition & 0 deletions tests/ctst/features/azureArchive.feature
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,7 @@ Feature: Azure Archive
@Flaky
@AzureArchive
@ColdStorage
@Exclusive
Scenario Outline: Create, read, update and delete azure archive location
Given an azure archive location "<locationName>"
And a "<versioningConfiguration>" bucket
Expand Down
1 change: 1 addition & 0 deletions tests/ctst/features/bucketWebsite.feature
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ Feature: Bucket Websites
@2.6.0
@PreMerge
@BucketWebsite
@Exclusive
Scenario Outline: Bucket Website CRUD
# The scenario should test that we can put a bucket website configuration on a bucket
# send an index.html
Expand Down
1 change: 1 addition & 0 deletions tests/ctst/features/pra.feature
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ Feature: PRA operations
@Dmf
@PRA
@ColdStorage
@Exclusive
Scenario Outline: PRA (nominal case)
# Prepare objects in the primary site
Given a "<versioningConfiguration>" bucket
Expand Down
Loading