Use Podman volume instead of host path for Caddy config storage

[adarshagrawal38]

Replaced the host path bind mount with a Podman-managed volume for storing Caddy configuration, avoiding host filesystem dependencies and solving permission issue for non root caddy user.

[adarshagrawal38]

Configuring catalog with custom certs.

[root@onprem133726058 ai-services]# ./bin/ai-services-linux-ppc64le catalog configure --ssl-cert /root/adarsh/manali-4096/manali-4096.pem --ssl-key /root/adarsh/manali-4096/manali-4096.key --runtime podman
Existing secret found: b8d0011c531509fd8f8acc45b
Existing secret found: fb29ce4f275de51917f181a10
Existing secret found: f7b88d64a2f632b65f2f3064e

 Executing Layer 1/3: [catalog-secret.yaml.tmpl catalog-db-secret.yaml.tmpl auth-secret.yaml.tmpl catalog-caddy-ssl-secret.yaml.tmpl]

Layer 1 completed

 Executing Layer 2/3: [catalog-db.yaml.tmpl caddy.yaml.tmpl]
..
..
-------
'caddy.yaml.tmpl', 'ai-services--caddy': Pod has been successfully deployed and ready!
-------
-------
-------
⠴ Configuring catalog service...'catalog-db.yaml.tmpl', 'ai-services--db', 'ai-services--db-postgresql': Readiness Check for the container is completed!
-------
'catalog-db.yaml.tmpl', 'ai-services--db', '6bd6b020ea31-infra': Performing Container Readiness check...
'catalog-db.yaml.tmpl', 'ai-services--db', '6bd6b020ea31-infra': Waiting for Container Readiness... Timeout set: 2m0s
'catalog-db.yaml.tmpl', 'ai-services--db', '6bd6b020ea31-infra': Readiness Check for the container is completed!
-------
'catalog-db.yaml.tmpl', 'ai-services--db': Pod has been successfully deployed and ready!
-------
..
...
Layer 2 completed

 Executing Layer 3/3: [catalog.yaml.tmpl]
-------
Processing template: catalog
⠹ Configuring catalog service...'catalog.yaml.tmpl', 'ai-services--catalog': Starting Pod Readiness check...
⠹ Configuring catalog service...'catalog.yaml.tmpl', 'ai-services--catalog', 'ai-services--catalog-backend': Readiness Check for the container is completed!
-------
Layer 3 completed
✔ Catalog service deployed successfully
-------
SSL certificates loaded successfully into Caddy
Successfully registered routes for 1 pod(s)
Next Steps:
-------
- Access the Catalog UI at https://catalog-ui.powervm-spyre-pok.cis.ibm.net

- Access the Catalog Backend at https://catalog-api.powervm-spyre-pok.cis.ibm.net

CATALOG login

[root@onprem133726058 ai-services]# ./bin/ai-services-linux-ppc64le catalog login --server https://catalog-api.powervm-spyre-pok.cis.ibm.net --username admin --runtime podman --insecure
Password:
WARNING: WARNING: TLS certificate verification is disabled. This should NOT be used in production environments.
Logging in to https://catalog-api.powervm-spyre-pok.cis.ibm.net as "admin"...
Login successful.
[root@onprem133726058 ai-services]#

[adarshagrawal38]

Existing caddy files structure

[root@onprem133726058 ai-services]# ls -lrt /var/lib/ai-services/common/caddy-config/caddy/
total 4
-rw-------. 1 root root 1908 Aug 18 21:45 autosave.json

[root@onprem133726058 ai-services]# ls -lrt /var/lib/ai-services/common/caddy
total 12
drwx------. 3 root root  25 Aug 18 10:58 pki
-rw-------. 1 root root 107 Aug 18 10:58 last_clean.json
-rw-------. 1 root root  36 Aug 18 10:58 instance.uuid
drwx------. 3 root root  19 Aug 18 11:07 certificates
drwx------. 2 root root   6 Aug 19 03:25 locks
-rw-r--r--. 1 root root 458 Aug 19 04:22 Caddyfile. 

Directory file from inside caddy container.
files are replicated, as we are using one volume for two mount points.

[root@ai-services--caddy srv]# ls -lrt /data/caddy/
total 16
-rw-r--r--. 1 root root 459 Aug 19 06:51 Caddyfile
drwx------. 3 root root  25 Aug 19 06:51 pki
drwx------. 2 root root   6 Aug 19 06:51 locks
-rw-------. 1 root root 107 Aug 19 06:51 last_clean.json
-rw-------. 1 root root  36 Aug 19 06:51 instance.uuid
-rw-------. 1 root root 911 Aug 19 06:53 autosave.json


[root@ai-services--caddy srv]# ls -lrt /config/caddy/
total 16
-rw-r--r--. 1 root root 459 Aug 19 06:51 Caddyfile
drwx------. 3 root root  25 Aug 19 06:51 pki
drwx------. 2 root root   6 Aug 19 06:51 locks
-rw-------. 1 root root 107 Aug 19 06:51 last_clean.json
-rw-------. 1 root root  36 Aug 19 06:51 instance.uuid
-rw-------. 1 root root 911 Aug 19 06:53 autosave.json
[root@ai-services--caddy srv]#

Custom certs are mounted in path

[root@ai-services--caddy srv]# ls -lrt /etc/secret/ssl/
total 8
-rw-r--r--. 1 root root 3242 Aug 19 06:51 tls.key
-rw-r--r--. 1 root root 2170 Aug 19 06:51 tls.crt

[shwetha-s-poojary]

We used this staged directory to check if the previous failed deployment was using custom cert-key or caddy generated cert-key.We needed it to ensure its idempotent.Just make sure we are not breaking this behaviour.