HyperShell

HyperShell provisions and manages OpenShell gateways at scale and across clouds.

Kubernetes Prerequisites

The control plane requires the following resources to be present on the target cluster before it can fully reconcile gateways.

Agent Sandbox controller

The Agent Sandbox controller (agents.x-k8s.io) must be installed on the cluster. Gateway pods manage sandboxes via the Sandbox custom resource, and the provisioned RBAC grants permissions on agents.x-k8s.io/sandboxes. NetworkPolicies also reference sandbox labels (agents.x-k8s.io/sandbox-name-hash) for pod-level traffic control.

cert-manager

cert-manager must be installed for automatic TLS certificate provisioning. The control plane auto-detects cert-manager at startup. If absent, TLS certificates must be provisioned manually via the certgen job.

kubectl apply -f https://github.com/cert-manager/cert-manager/releases/latest/download/cert-manager.yaml

Gateway API CRDs

The Gateway API CRDs must be installed if external routing via GRPCRoute is desired. The control plane auto-detects Gateway API availability at startup and skips route provisioning if the CRDs are not present.

kubectl apply -f https://github.com/kubernetes-sigs/gateway-api/releases/latest/download/standard-install.yaml

Shared Gateway

A pre-existing Gateway resource must be provisioned by an administrator before the control plane can create tenant GRPCRoutes. The Gateway should use a wildcard hostname and a cert-manager-issued wildcard TLS certificate. All tenant GRPCRoutes attach to this shared Gateway.

See deploy/openshift/infrastructure/GATEWAY-SETUP.md for step-by-step setup instructions including cert-manager configuration.

The Gateway name is configured via the GATEWAY_API_GATEWAY_NAME environment variable (required).

Trusted CA bundle (optional)

If the gateway needs to interact with an OIDC issuer (e.g., Keycloak) that uses a self-signed or private CA certificate, create a ConfigMap named gateway-trusted-ca in the control plane namespace (default: hypershell-system). The control plane copies this ConfigMap into each tenant namespace and mounts it into gateway pods so they can validate the issuer's TLS certificate when fetching JWKS keys or verifying tokens.

kubectl -n hypershell-system create configmap gateway-trusted-ca --from-file=ca-bundle.crt=/path/to/ca.crt

Keycloak OIDC client provisioning (hypershell-keycloak-admin)

The control plane provisions an OIDC client in Keycloak for each gateway it reconciles. It authenticates to Keycloak using a confidential client whose credentials are read from a Secret named hypershell-keycloak-admin in the control plane namespace (default: hypershell-system). If this Secret is absent at startup, Keycloak integration is silently disabled for the lifetime of that pod.

1. Create a realm

Log in to the Keycloak Admin Console and create a realm for HyperShell (e.g., hypershell), or use an existing realm. You can also do this via the REST API:

KEYCLOAK_URL="https://keycloak.example.com"
ADMIN_TOKEN=$(curl -s -X POST "$KEYCLOAK_URL/realms/master/protocol/openid-connect/token" \
  -H "Content-Type: application/x-www-form-urlencoded" \
  -d "client_id=admin-cli&username=admin&password=<admin-password>&grant_type=password" \
  | jq -r '.access_token')

curl -s -X POST "$KEYCLOAK_URL/admin/realms" \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"realm": "hypershell", "enabled": true, "displayName": "HyperShell"}'

2. Create a confidential client

Create a client named hypershell-control-plane in the realm. It must use service-account authentication (no standard or direct-grant flows):

curl -s -X POST "$KEYCLOAK_URL/admin/realms/hypershell/clients" \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "clientId": "hypershell-control-plane",
    "name": "HyperShell Control Plane",
    "enabled": true,
    "clientAuthenticatorType": "client-secret",
    "serviceAccountsEnabled": true,
    "standardFlowEnabled": false,
    "directAccessGrantsEnabled": false,
    "publicClient": false
  }'

3. Grant realm-management roles to the service account

The control plane needs manage-clients, manage-users, and view-users from the built-in realm-management client so it can create and delete gateway OIDC clients:

# Retrieve IDs
CLIENT_UUID=$(curl -s "$KEYCLOAK_URL/admin/realms/hypershell/clients?clientId=hypershell-control-plane" \
  -H "Authorization: Bearer $ADMIN_TOKEN" | jq -r '.[0].id')
SA_USER_ID=$(curl -s "$KEYCLOAK_URL/admin/realms/hypershell/clients/$CLIENT_UUID/service-account-user" \
  -H "Authorization: Bearer $ADMIN_TOKEN" | jq -r '.id')
RM_UUID=$(curl -s "$KEYCLOAK_URL/admin/realms/hypershell/clients?clientId=realm-management" \
  -H "Authorization: Bearer $ADMIN_TOKEN" | jq -r '.[0].id')

# Fetch the three role objects and assign them
ROLES=$(curl -s "$KEYCLOAK_URL/admin/realms/hypershell/clients/$RM_UUID/roles" \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  | jq '[.[] | select(.name | IN("manage-clients","manage-users","view-users"))]')

curl -s -X POST "$KEYCLOAK_URL/admin/realms/hypershell/users/$SA_USER_ID/role-mappings/clients/$RM_UUID" \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d "$ROLES"

4. Create the Secret

Retrieve the generated client secret and create the Kubernetes Secret in the control plane namespace:

CLIENT_SECRET=$(curl -s "$KEYCLOAK_URL/admin/realms/hypershell/clients/$CLIENT_UUID/client-secret" \
  -H "Authorization: Bearer $ADMIN_TOKEN" | jq -r '.value')

kubectl -n hypershell-system create secret generic hypershell-keycloak-admin \
  --from-literal=server-url="$KEYCLOAK_URL/" \
  --from-literal=realm="hypershell" \
  --from-literal=client-id="hypershell-control-plane" \
  --from-literal=client-secret="$CLIENT_SECRET"

If you need to rotate the client secret or update any value, delete and recreate the Secret then restart the control plane pod -- the Secret is read once at startup.

kubectl -n hypershell-system delete secret hypershell-keycloak-admin
# recreate with updated values, then:
kubectl -n hypershell-system rollout restart deployment/hypershell-controller

Confirm the control plane picked up the configuration:

kubectl -n hypershell-system logs deployment/hypershell-controller | grep -i keycloak
# Expected: INFO keycloak integration enabled: server=... realm=hypershell

Base domain (GATEWAY_API_BASE_DOMAIN)

The control plane requires GATEWAY_API_BASE_DOMAIN to derive GRPCRoute hostnames for tenant gateways. Without it, GRPCRoute creation is skipped and gateways will not be externally reachable.

On OpenShift, look up the cluster's default base domain:

oc get ingresses.config.openshift.io cluster -o jsonpath='{.spec.domain}'

This typically returns a value like apps.<cluster-name>.<base-domain>. Set this value as GATEWAY_API_BASE_DOMAIN on the controller deployment using one of:

# Option 1: Patch the deployment directly
oc set env deployment/hypershell-controller -n hypershell-system \
  GATEWAY_API_BASE_DOMAIN="$(oc get ingresses.config.openshift.io cluster -o jsonpath='{.spec.domain}')"

Or apply a kustomize patch via deploy/openshift/kustomization.yaml with your specific base domain value (see that file for the patch structure).

Control plane environment variables

Variable	Default	Required	Description
HYPERSHELL_GRPC_SERVER_ADDR	localhost:9000	✓	gRPC address of the API server
HYPERSHELL_API_SERVER_URL	http://localhost:8000	✓	HTTP address of the API server
HYPERSHELL_NAMESPACE	pod namespace	✓	Namespace this controller runs in. In cluster this is the downward API (metadata.namespace), so the value is unique to that controller. It is also the hypershell.redhat.io/instance identity stamped on gateway namespaces so GC never reaps another HyperShell's workloads.
GATEWAY_IMAGE	(none)	✓ required	Container image for tenant gateways (pinned by digest; no fallback). Set in deploy/base/controller.yaml
GATEWAY_SUPERVISOR_IMAGE	(none)	✓ required	Container image for gateway supervisors (pinned by digest; no fallback). Set in deploy/base/controller.yaml
GATEWAY_API_GATEWAY_NAME	(required)	✓	Name of the pre-existing Gateway resource that tenant GRPCRoutes attach to
GATEWAY_API_GATEWAY_NAMESPACE	openshift-ingress	✓	Namespace where the pre-existing Gateway resource lives
GATEWAY_API_BASE_DOMAIN	(none)		Base domain for tenant hostname generation (e.g., openshell.example.com → gw-<ns>.openshell.example.com)
GATEWAY_MANIFESTS_DIR	/manifests/gateway	✓	Path to gateway manifest templates

Observability

Prometheus metrics

The API server exposes Prometheus metrics on its metrics port (default :8080/metrics).

Metric	Type	Description
`hypershell_gateways_total{phase="Running"	"Provisioning"	"Degraded"

The control plane also exports these metrics through OTLP:

Metric	Type	Description
gateway.provision.duration	Histogram	Time from Gateway creation to its first successful Running phase
reconcile.queue.depth	Observable gauge	Ready resource keys that wait for a reconcile worker
reconcile.queue.wait.duration	Histogram	Time from ready queue admission to the start of reconciliation

The queue metrics use the bounded resource.kind attribute. They do not use a resource identifier. A standard Prometheus conversion exposes duration values in seconds.

Grafana dashboard

A pre-built Grafana dashboard is provided at dashboards/hypershell-dashboard.yml. It is packaged as a Kubernetes ConfigMap with the grafana_dashboard: "true" label so it is picked up automatically by the Grafana sidecar.

The dashboard includes:

• Running Gateways - stat panel showing the current gateway count
• Running Gateways Over Time - timeseries graph of gateway count over the selected time range

To deploy the dashboard, apply the ConfigMap to the namespace where Grafana is running:

kubectl apply -f dashboards/hypershell-dashboard.yml