[Feature] Add interactive mode to demo.sh with deployment type selection

[osullivandonal]

Changes

This PR add's an interactive mode when running demo.sh, this interactive mode prompts the user to enter the following:

• What elasticsearch deployment type to use, either serverless or cloud-hosted
• The environment to deploy the demo in, either docker or k8s

This PR also add's integration tests that run in the PR's CI.

Merge Requirements

For new features contributions, please make sure you have completed the following
essential items:

• CHANGELOG.md updated to document new feature additions
• Appropriate documentation updates in the docs
• Appropriate Helm chart updates in the helm-charts

Maintainers will not merge until the above have been completed. If you're unsure
which docs need to be changed ping the
@open-telemetry/demo-approvers.

[dmathieu]

There's a failure in the k8s tests, but the CI output isn't very helpful. Could we try to display appropriate logs when that happens?

[dmathieu]

If I run ./demo.sh locally once, it asks me for creds, as expected. But then, if I stop the app and run it a second time (the creds are stored on disk), it asks for creds again.
How about detecting the presence of creds and reusing that config on second runs?

[osullivandonal]

There's a failure in the k8s tests, but the CI output isn't very helpful. Could we try to display appropriate logs when that happens?

We can indeed, I had a look at the bashunit docs and have updated the tests with some custom assertions, see bashunit docs here.

[osullivandonal]

If I run ./demo.sh locally once, it asks me for creds, as expected. But then, if I stop the app and run it a second time (the creds are stored on disk), it asks for creds again. How about detecting the presence of creds and reusing that config on second runs?

Good spot, I have updated demo.sh to only prompt for creds if they do not exist on disc, see 7d43da2

[osullivandonal]

k8s integration tests have been skipped as there is an issue with the helm charts in elastic-agent atm:

Running test/elastic/basic_k8s_test.sh
↷ Skipped: Launch demo k8s    K8s test disabled: helm chart null mapping issue
↷ Skipped: Destroy demo k8s    K8s test disabled: helm chart null mapping issue

There is an open PR to fix this -> elastic/elastic-agent#11481

[rogercoll]

Overall lgtm, just added some comments to avoid upstream conflicts

[rogercoll]

Why is this needed? Did we reach any resource limit? Maybe run the test on a smaller distro?

[osullivandonal]

Yup this free's up space as the k8s test run fails due to insuffienent space. I tried a few different options, including running ubuntu-slim and using the maximize-build-disk-space. They failed due to ubuntu-slim not having docker installed and the maximize-build-disk-space fails due to kind erroring out on install.

I also have updated the elastic stack install to only install elasticsearch and not kibana, see here. That helped a bit but I still needed 1GB of space.

I followed this blog and updated the disc clean up. I think for now this is our only option.