Context
The test framework currently performs testing by making a comparison of all of the files in the --expected-output-dir and the /kratix/output filed generated by the pipeline/stage. Tests can fail if:
- Any
/kratix/outputs file generated by the workflow is absent in the expected directory
- There is any difference in the content of files between the generated files and the expected directory
This makes the setup of the test assets non-trivial, especially for verbose assets such as long terraform definitions (such as those that could be generated by the init tf-module-promise cli command (see example here).
To make testing more lightweight and specific to the content users care about, they should be able to assert against a subset of their generated assets and the content of those assets.
Proposals
- The introduction of a
--file-contains flag that takes the name of a file as a key and a string representing some expected output as the value e.g.
kratix test stage \
--image-tag ghcr.io/syntasso/check-for-issue:v0.0.1 \
--input-dir standard-demo/promises/app-as-a-service/test \
--file-contains "s3_default_example-s3.tf.json"="\"region\"\: \"us-east-1\""
The test only asserts that file in the key is present and that within it, the content in value can be found. No assertions are made against any other files generated by the container.
- The introduction of pairs of flags representing the expected filename and, optionally, the expected subset of file content.
kratix test stage \
--image-tag ghcr.io/syntasso/check-for-issue:v0.0.1 \
--input-dir standard-demo/promises/app-as-a-service/test \
--expected-file s3_default_example-s3.tf.json
--file-contains "\"region\"\: \"us-east-1\""
kratix test stage \
--image-tag ghcr.io/syntasso/check-for-issue:v0.0.1 \
--input-dir standard-demo/promises/app-as-a-service/test \
--expected-file s3_default_example-s3.tf.json
--file-contains=$(cat expected-outputs/tf-vars.tf)
When both new flags are specified, the test only asserts that the --expected-file is present and that within it, the content in --file-contains can be found. No assertions are made against any other files generated by the container.
kratix test stage \
--image-tag ghcr.io/syntasso/check-for-issue:v0.0.1 \
--input-dir standard-demo/promises/app-as-a-service/test
--expected-file s3_default_example-s3.tf.json
--expected-output-dir standard-demo/promises/app-as-a-service/test
When the --expected-file flag is used alongside the existing --expected-output-dir flag, the test asserts that the --expected-file is present and that its content matches the content of a file of the same name in the --expected-output directory.
Acceptance
Given I have a promise stage that generates `pod.yaml` and `configmap.yaml`
When run the kratix test command
Then I can specify that I want to test that the generated `pod.yaml` includes the string "replicas: 2"
Context
The test framework currently performs testing by making a comparison of all of the files in the
--expected-output-dirand the/kratix/outputfiled generated by the pipeline/stage. Tests can fail if:/kratix/outputsfile generated by the workflow is absent in the expected directoryThis makes the setup of the test assets non-trivial, especially for verbose assets such as long terraform definitions (such as those that could be generated by the
init tf-module-promisecli command (see example here).To make testing more lightweight and specific to the content users care about, they should be able to assert against a subset of their generated assets and the content of those assets.
Proposals
--file-containsflag that takes the name of a file as a key and a string representing some expected output as the value e.g.The test only asserts that file in the key is present and that within it, the content in value can be found. No assertions are made against any other files generated by the container.
When both new flags are specified, the test only asserts that the
--expected-fileis present and that within it, the content in--file-containscan be found. No assertions are made against any other files generated by the container.kratix test stage \ --image-tag ghcr.io/syntasso/check-for-issue:v0.0.1 \ --input-dir standard-demo/promises/app-as-a-service/test --expected-file s3_default_example-s3.tf.json --expected-output-dir standard-demo/promises/app-as-a-service/testWhen the
--expected-fileflag is used alongside the existing--expected-output-dirflag, the test asserts that the--expected-fileis present and that its content matches the content of a file of the same name in the--expected-outputdirectory.Acceptance