Skip to content

Add a one-command local devbox #11

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dejanzele wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Add a one-command local devbox #11

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 23 additions & 15 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,8 +4,9 @@

[Flyte 2](https://github.com/flyteorg/flyte) lets you write batch workflows as plain async Python.
[Armada](https://github.com/armadaproject/armada) schedules millions of jobs a day across many
Kubernetes clusters, with fair-share, gang scheduling, and preemption. `armada-flyte` connects the
two: your Flyte task runs as an Armada job, with one line of config and no new API to learn.
Kubernetes clusters, with fair-share, gang scheduling (running a group of pods all-or-nothing), and
preemption. `armada-flyte` connects the two: your Flyte task runs as an Armada job, with one line of
config and no new API to learn.

## The whole integration

Expand All @@ -25,10 +26,11 @@ async def greet(name: str) -> str:
return f"hello {name}, from an Armada pod" # runs in an Armada-scheduled pod
```

A stock `@env.task` and one `plugin_config` line. Fan out with `asyncio.gather`, pass dataclasses
between tasks, gang-schedule a group: it is all just Flyte, running on Armada.
The `queue` names the Armada queue the job goes into, its fair-share bucket. A stock `@env.task` and
one `plugin_config` line. Fan out with `asyncio.gather`, pass dataclasses between tasks, or gang-schedule
a group all-or-nothing with `armada_flyte.Gang`: it is all just Flyte, running on Armada.

The connector submits to the Armada at `ARMADA_URL` (default `localhost:50051`). Point it at a
The connector submits to Armada at `ARMADA_URL` (default `localhost:50051`). Point it at a
remote cluster by setting that env var, or in code:

```python
Expand All @@ -41,18 +43,22 @@ never lands in the control plane. See [docs/getting-started.md](docs/getting-sta

## See it run

With a local Armada cluster up, the [demo](demo/) stands up a Flyte backend and the connector in one
command, then you submit the task:
`./hack/up.sh` brings up Armada and Flyte in one local kind cluster (see the
[quickstart](demo/README.md)). Submitting a task then returns its typed result to your terminal, and the
run appears in the Flyte UI:

```console
$ ./demo/setup.sh
$ ./.venv/bin/python examples/hello.py
submitted run rxc4nspfkjqr5px6q9nj
UI: http://localhost:30080/v2/.../runs/rxc4nspfkjqr5px6q9nj
Flyte console: http://localhost:5001/v2/.../runs/rxc4nspfkjqr5px6q9nj
Armada Lookout: http://localhost:30000

hello armada, from an Armada pod
```

The run shows up in the Flyte UI, scheduled and executed by Armada. See
[getting started](docs/getting-started.md) for the walkthrough.
This is an integration between two systems, so there is real setup, but `./hack/up.sh` does it in one
command: the [quickstart](demo/README.md) covers what it stands up and how to run this. From there,
[examples/](examples/) ladder from `hello` up to a gang inside a DAG.

## Why both

Expand All @@ -62,7 +68,7 @@ The run shows up in the Flyte UI, scheduled and executed by Armada. See
| The Flyte console: runs, lineage, logs | Fair-share between queues, gang scheduling, preemption |
| Local execution for fast iteration | Battle-tested at millions of jobs a day |

You keep Flyte's authoring and console; Armada does the scheduling. No rewrite, no second SDK.
You keep Flyte's authoring and console, and Armada does the scheduling. No rewrite, no second SDK.

## How it works

Expand All @@ -83,12 +89,14 @@ Flyte UI.

## Where to go next

- **Run it locally.** [demo/](demo/) stands up the backend and connector in one command (the
[See it run](#see-it-run) commands above).
- **Run it locally.** `./hack/up.sh` brings up Armada, Flyte, and the connector in one kind cluster;
the [quickstart](demo/README.md) covers it.
- **Write tasks.** Start from [examples/hello.py](examples/hello.py), then [examples/](examples/) for
fan-out, gang scheduling, and a gang inside a DAG.
- **Run against your own backend.** [Getting started](docs/getting-started.md) covers installing the
connector, running it as a service, and building the task image.
connector, running it as a service, and building the task image. The examples read `FLYTE_ENDPOINT`
(the Flyte API) and `FLYTE_UI_BASE` (the console link they print) from the environment, both
defaulting to the local devbox, so point them at your backend there.
- **Understand the internals.** [How it works](docs/architecture.md): the connector, state mapping,
and gang scheduling.
- **Deploy the connector as a service.** [deploy/](deploy/).
Expand Down
189 changes: 114 additions & 75 deletions demo/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,106 +1,145 @@
# Local quickstart
# Quickstart

Run a Flyte 2 backend in the same Kind cluster Armada already schedules onto, then submit a task.
This is the one-command way to see the connector work end to end.
`armada-flyte` runs Flyte tasks as Armada jobs, so trying it means bringing up both. `./hack/up.sh`
does that in one command: a local kind cluster with Armada (operator, dependencies, and the component
CRs), Traefik, the Flyte 2 backend, the connector, and the `flyte` queue, all wired together.

```
host Kind cluster "armada-test"
┌────────────┐ localhost:30080 ┌─────────────────────────────────────────┐
│ Flyte CLI │ ─────────────────▶│ flyte-binary (API + console + TaskAction │
│ (examples) │ localhost:30900 │ reconciler) │
├────────────┤ ─────────────────▶│ postgres (metadata + runs) │
│ connector │◀──── :8000 ───────│ minio (blob store) │
│ c0 │ ──┐ │ armada-<jobid> pods ──in-cluster──▶ minio │
└────────────┘ │ :50051 └─────────────────────────────────────────┘
└──▶ Armada control plane (docker, from `dev:full`)
host kind cluster "armada"
┌──────────┐ localhost:30080 ┌───────────────────────────────────────────┐
│ examples │ ─────────────────▶│ flyte-binary (API + console) │
│ (client) │ │ armada-flyte-connector ──▶ armada-server │
│ browser │ localhost:5001 │ minio (blob store), postgres (metadata) │
│ │ ─────────────────▶│ armada (operator): server, scheduler, │
└──────────┘ │ executor ──▶ job pods │
└───────────────────────────────────────────┘
```

`flyte-binary`, minio, and postgres run in the cluster. The connector `c0` runs on the host and is
the only process bridging Flyte and Armada. The Armada pods reach minio in-cluster at
`minio.flyte:9000`, and `setup.sh` port-forwards the Flyte API and minio to `localhost` so the client
reaches them without any kind port mapping.
Everything runs in the cluster. `hack/kind-config.yaml` maps every host-facing port to a NodePort, so
the client (the examples) and your browser reach the API and console straight from the host with no
port-forwards. Every hop between components is cluster DNS.

## Prerequisites

1. **Armada**, with a real executor against the Kind cluster and the `flyte` queue created. Clone
[armada](https://github.com/armadaproject/armada), then:
```
go run github.com/magefile/mage@v1.17.2 dev:full # kind "armada-test" + full stack
go run cmd/armadactl/main.go create queue flyte --armadaUrl localhost:50051
```
Wait for the executor to log `Reporting current free resource` before submitting.
2. **This repo's venv** (an arm64 Python on Apple Silicon, since Flyte's `obstore` wheel has no x86 build):
```
python3.11 -m venv .venv && ./.venv/bin/pip install -e .
```
3. `docker`, `helm`, `kubectl`, `kind` on PATH.

`setup.sh` pulls the published flyte-binary chart from the flyteorg helm repo and a stock upstream
`cr.flyte.org/flyteorg/flyte-binary-v2` image, so no Flyte checkout is needed. The pinned image is the
first commit that registers the connector-service plugin in the v2 executor
([flyteorg/flyte#7565](https://github.com/flyteorg/flyte/pull/7565)). Earlier stock builds do not have
it.

## Run
- `docker` (running), `helm`, `kubectl`, `kind`, and `python3` (3.10+) on your PATH. `up.sh` checks
these before it does any work.
- `armadactl` is fetched to `~/bin` if it is not already on your PATH; add `~/bin` to your PATH if the
command is not found afterwards.

## Bring it up

```
export KUBECONFIG=<armada-checkout>/.kube/external/config # the kubeconfig `dev:full` writes
./demo/setup.sh
./hack/up.sh
./.venv/bin/python examples/hello.py
```

`setup.sh` deploys minio + postgres, installs the flyte-binary chart, builds and loads the task
image, starts the connector, and port-forwards the Flyte API and minio to `localhost`. Then submit an
example. It points at `localhost:30080` and targets queue `flyte`:
`up.sh` creates the kind cluster; installs Armada and its dependencies (cert-manager, Pulsar, Postgres,
Redis) via the operator and creates the `flyte` queue; builds the repo's virtualenv; then deploys
Traefik and the Flyte 2 backend (`flyte-binary`), builds and loads the task image, and deploys the
connector. The
first run pulls a lot of images before it settles. It waits for Armada to register executor capacity, so
when it prints `Devbox up` the integration is ready to submit:

```
./.venv/bin/python examples/hello.py
Devbox up. Run an example:
./.venv/bin/python examples/hello.py
./.venv/bin/python examples/dag.py
```

Open the printed `UI:` link to watch Armada schedule the pod and record the typed result. Other
examples work the same way (`examples/fanout.py`, `examples/gang.py`, `examples/dag.py`).
`hello.py` prints its result to your terminal (`hello armada, from an Armada pod`), and the run appears
in the Flyte console at the link it prints. That is the whole loop: a Flyte task, scheduled and run by
Armada.

## Files
## The UIs

| File | What it is |
|------|-----------|
| `setup.sh` | One-command stand-up of the backend + connector. Idempotent. |
| `minio.yaml` | Blob store. Pods reach it in-cluster. The host reaches it via a port-forward `setup.sh` starts. |
| `postgres.yaml` | Flyte's `flyte` (metadata) and `runs` (run-graph) databases. |
| `flyte-binary-values.yaml` | Chart overrides. `__HOST_IP__` (the connector endpoint) is filled in by `setup.sh`. |
Each example prints two links:

## Overrides
- **Flyte console** (`http://localhost:5001/v2`) - the run graph, status, and pod logs. Served by
Traefik through the flyte-binary chart's own ingress.
- **Armada Lookout** (`http://localhost:30000`) - the job and pod as Armada sees them.

Per-task input/output values do not render in the Flyte console yet. The Flyte 2 console is pre-release
and its I/O panel is incomplete upstream. The values are recorded correctly: each example prints its
result, and the data is in the blob store.

## Work through the examples

Once `hello.py` runs, the [examples](../examples/) build up in order:

`setup.sh` reads these env vars: `KIND_CLUSTER` (default `armada-test`), `FLYTE_CHART` (default
`flyteorg/flyte-binary`, or a local directory for an unreleased chart), `FLYTE_CHART_VERSION`
(default `v2.0.27`), `FLYTE_IMAGE` (default the pinned stock `flyte-binary-v2` build), `TASK_IMAGE`
(default `armada-flyte-task:v1`), `ARMADA_URL` (default `localhost:50051`), `HOST_IP` (auto-detected).
1. [`function.py`](../examples/function.py) - one task doing real work (a Black-Scholes price).
2. [`fanout.py`](../examples/fanout.py) - a typed dataclass through a parallel fan-out / fan-in.
3. [`gang.py`](../examples/gang.py) - N co-scheduled workers as one Armada gang (all-or-nothing).
4. [`dag.py`](../examples/dag.py) - the full shape: generate a dataset, run a gang over it, aggregate.

Each runs the same way: `./.venv/bin/python examples/<name>.py`.

## Teardown

```
helm -n flyte uninstall flyte-binary
kubectl delete namespace flyte
pkill -f "bin/c0 --port 8000"
pkill -f "port-forward svc/flyte-binary-http"
pkill -f "port-forward svc/minio"
./hack/down.sh
```

The kind cluster and Armada come down with `go run github.com/magefile/mage@v1.17.2 dev:fullDown` in
the armada repo.
`down.sh` deletes the kind cluster and clears the Flyte client's upload cache. The cache is keyed by the
endpoint (`localhost:30080`, the same for every devbox), so clearing it here keeps the next `up.sh` from
reusing entries that point at the deleted cluster's minio and 404-ing on their code bundle.

## Overrides

`up.sh` reads `KIND_CLUSTER` (default `armada`). `demo/setup.sh`, which it calls for the Flyte side,
reads `FLYTE_CHART` (default `flyteorg/flyte-binary`, or a local directory for an unreleased chart),
`FLYTE_CHART_VERSION` (default `v2.0.27`), `FLYTE_IMAGE` (default the pinned stock `flyte-binary-v2`
build; this only pre-loads the image into kind — the tag the pod runs is pinned in
`flyte-binary-values.yaml`, so change both together), `TASK_IMAGE` (default `armada-flyte-task:v1`),
and `CONNECTOR_IMAGE` (default
`gresearch/armada-flyte-connector:0.2.0`). The pinned Flyte image is the first that registers the
connector plugin in the executor
([flyteorg/flyte#7565](https://github.com/flyteorg/flyte/pull/7565)).

## Manual setup without the devbox script

`up.sh` is the supported path. If you want the steps by hand, or to submit to an Armada cluster you
already run, the pieces are:

1. **Armada.** `hack/setup-armada.sh` installs Armada into the current kind context (operator,
dependencies, and the CRs in `hack/armada/`). The stock
[armada-operator](https://github.com/armadaproject/armada-operator) `make kind-all` also works, but
its kind config maps only ports 30000-30002, so the Flyte console on 5001 will not be reachable from
the host - create the cluster from `hack/kind-config.yaml` for that.
2. **Flyte.** `./demo/setup.sh` deploys Traefik, the `flyte-binary` backend (with its minio blob store
and postgres), builds the task image, and deploys the connector.
3. **Queue.** `armadactl create queue flyte`.

## Files

| File | What it is |
|------|-----------|
| `setup.sh` | Stands up the Flyte side (Traefik + backend + connector) in the cluster. Idempotent. |
| `minio.yaml` | Blob store. Pods reach it in-cluster; the host reaches it via the kind NodePort mapping. |
| `postgres.yaml` | Flyte's `flyte` (metadata) and `runs` (run-graph) databases. |
| `flyte-binary-values.yaml` | Chart overrides: storage, the chart ingress, and the connector routing. |

## Troubleshooting

- **`no connector found for task type [armada]`**: the connector config did not reach the running
- **`no connector found for task type [armada]`**: the connector routing did not reach the running
config. It must live under `configuration.inline.plugins.connector-service` with
`supportedTaskTypes: [armada]`. The chart's top-level `configuration.connectorService` is not wired
in. Confirm with `kubectl -n flyte get cm flyte-binary-config -o yaml | grep -A4 connector-service`.
- **`TaskAction` stuck `Queued`**: Armada has no executor yet, or the `flyte` queue is missing.
Check the connector log (`/tmp/armada-flyte-c0.log`) and `armadactl get queue flyte`.
- **`localhost:30080` connection refused**: the port-forward is not running. Re-run `setup.sh`, or
start it by hand: `kubectl -n flyte port-forward svc/flyte-binary-http 30080:8090`.
`supportedTaskTypes: [armada]`; the chart's top-level `configuration.connectorService` is not wired in.
Confirm with `kubectl -n flyte get cm flyte-binary-config -o yaml | grep -A4 connector-service`.
- **Task stuck `Queued`**: Armada has no executor yet, or the `flyte` queue is missing. Check
`kubectl -n armada get pods` and `armadactl get queue flyte`.
- **A gang (`gang.py` / `dag.py`) is `REJECTED` or stuck `Queued`**: on your own Armada, the gang's
node-uniformity label must be tracked by the executor and indexed by the scheduler, or the scheduler
cannot place the gang. Confirm `kubernetes.io/hostname` is in the executor's
`kubernetes.trackedNodeLabels` and the scheduler's `scheduling.indexedNodeLabels`. The devbox CRs
(`hack/armada/armada-crs.yaml`) set both already. Also make sure the whole gang fits: with
`kubernetes.io/hostname` uniformity all members land on one node, so the sum of their requests must
fit that node's free capacity.
- **`localhost:30080` connection refused**: the cluster is not up, or was created without
`hack/kind-config.yaml`, so the NodePort is not mapped to the host. Bring the devbox up with
`./hack/up.sh`.
- **Flyte console shows "No actions found"**: open the exact `http://localhost:5001/v2/...` link the
example prints. The console reaches the API through Traefik on the same origin.
- **Pod `Completed` but the run fails resolving outputs**: the pod could not reach minio in-cluster.
Confirm the connector was started with `FLYTE_BLOB_ENDPOINT=http://minio.flyte.svc.cluster.local:9000`
and that the minio credentials match `minio.yaml`.
- **Pod fails with a 404 on `fast<hash>.tar.gz` after a re-run**: you recreated the blob store (a
fresh minio) but the client cached the previous upload and skipped re-uploading the code bundle to
the new one. Clear the cache and rerun: `rm -f ~/.flyte/local-cache/cache.db`.
Confirm the connector's `FLYTE_BLOB_ENDPOINT` is `http://minio.flyte.svc.cluster.local:9000` and the
minio credentials match `minio.yaml`.
- **Pod fails with a 404 on `fast<hash>.tar.gz` after a re-run**: you recreated the blob store but the
client cached the previous upload. Tear down with `./hack/down.sh`, which clears the cache.
Loading