Skip to content

feat: users can test that a subset of their expected outputs are present in the generated outputs #200

Description

@SaphMB

Context
The test framework currently performs testing by making a comparison of all of the files in the --expected-output-dir and the /kratix/output filed generated by the pipeline/stage. Tests can fail if:

  • Any /kratix/outputs file generated by the workflow is absent in the expected directory
  • There is any difference in the content of files between the generated files and the expected directory

This makes the setup of the test assets non-trivial, especially for verbose assets such as long terraform definitions (such as those that could be generated by the init tf-module-promise cli command (see example here).

To make testing more lightweight and specific to the content users care about, they should be able to assert against a subset of their generated assets and the content of those assets.

Proposals

  1. The introduction of a --file-contains flag that takes the name of a file as a key and a string representing some expected output as the value e.g.
kratix test stage \
  --image-tag ghcr.io/syntasso/check-for-issue:v0.0.1 \
  --input-dir standard-demo/promises/app-as-a-service/test \
  --file-contains "s3_default_example-s3.tf.json"="\"region\"\: \"us-east-1\""

The test only asserts that file in the key is present and that within it, the content in value can be found. No assertions are made against any other files generated by the container.

  1. The introduction of pairs of flags representing the expected filename and, optionally, the expected subset of file content.
kratix test stage \
  --image-tag ghcr.io/syntasso/check-for-issue:v0.0.1 \
  --input-dir standard-demo/promises/app-as-a-service/test \
  --expected-file s3_default_example-s3.tf.json 
  --file-contains "\"region\"\: \"us-east-1\""
kratix test stage \
  --image-tag ghcr.io/syntasso/check-for-issue:v0.0.1 \
  --input-dir standard-demo/promises/app-as-a-service/test \
  --expected-file s3_default_example-s3.tf.json 
  --file-contains=$(cat expected-outputs/tf-vars.tf)

When both new flags are specified, the test only asserts that the --expected-file is present and that within it, the content in --file-contains can be found. No assertions are made against any other files generated by the container.

kratix test stage \
  --image-tag ghcr.io/syntasso/check-for-issue:v0.0.1 \
  --input-dir standard-demo/promises/app-as-a-service/test 
  --expected-file s3_default_example-s3.tf.json 
  --expected-output-dir standard-demo/promises/app-as-a-service/test

When the --expected-file flag is used alongside the existing --expected-output-dir flag, the test asserts that the --expected-file is present and that its content matches the content of a file of the same name in the --expected-output directory.

Acceptance

Given I have a promise stage that generates `pod.yaml` and `configmap.yaml`
When run the kratix test command
Then I can specify that I want to test that the generated `pod.yaml` includes the string "replicas: 2"

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions