Skip to content

Add Communication to ImportAsset - #48

Draft
hdm wants to merge 6 commits into
mainfrom
custom-integration-communications
Draft

hdm wants to merge 6 commits into
mainfrom
custom-integration-communications

Conversation

@hdm

@hdm hdm commented Oct 7, 2026 •

Copy link
Copy Markdown
Contributor

Note: Moving to DRAFT until the platform side is updated.

Summary

The runZero console now accepts aggregated traffic summaries from custom integrations. This PR adds the matching runzero.types.Communication type and and a ImportAsset.communications field so SDK users can report them.

A record carries role (client for outbound, server for inbound), protocol, optional ports, a start_ts/end_ts window, and optional bytes_tx/bytes_rx. The console sums records that share a role and protocol, and an asset uploaded without communications keeps whatever was stored before.

It also fixes a long-standing timestamp problem that would have bitten the new window fields first.

Changes

  • Communication REST model with the API's field names and aliases (startTS, endTS, bytesTx, bytesRx), capped at 1000 records per asset.
  • Wrapped Communication lower-cases the role and upper-cases the protocol to match the console, and rejects invalid ports.
  • Exported from runzero.types; example and changelog updated.
  • Naive datetimes are treated as local time. The API requires RFC 3339 timestamps with an offset, and Go's parser rejects the offset-less string a naive datetime serialization. A validator on every upload-bound datetime field (first_seen_ts, installed_at, published_ts, first_detected_ts, last_detected_ts, start_ts, end_ts) now attaches the local offset with astimezone(), following the stdlib's own rule for naive values. Aware values are untouched. The example uses plain datetime.now() to show this is safe.

Testing

pytest tests/runzero --without-integration, black --check, isort --check-only, mypy ./runzero, and pylint ./runzero all pass. pylint reports the same five pre-existing too-many-positional-arguments messages as main.

Poetry's interpreter on this machine is broken, so checks ran in a uv virtualenv with pydantic 1.10 and the dev dependencies from pyproject.toml.

Rate limiting

Live tests share one account rate limit, and the console refuses a burst with 429 Too Many Requests plus a Retry-After header. The client now retries such a request, waiting for Retry-After or an exponential backoff (capped at two minutes), before raising. Client(rate_limit_retries=..., rate_limit_backoff_seconds=...) tunes it; rate_limit_retries=0 restores the raise-immediately behavior. Every 429 is a RateLimitError, which now subclasses ClientError so existing handlers keep catching it, and carries retry_after. Unit tests drive the retry loop through a fake session.

Communications on the test console

The console side (runZeroInc/platform#28315) has not shipped to the integration test console yet. The test config gains an optional communications flag (test_configs.toml or the communications environment variable, wired as "false" in check_pr.yml). The new upload test skips until it is set to true. A console without the feature ignores the field, so an upgraded SDK does not break against it.

Python version floor

The first CI run failed before any test ran: the hosted runner's virtualenv no longer seeds pip for Python 3.8, so every job pinned to 3.8 died in poetry install, and fail-fast cancelled the other unit jobs after their tests had passed.

This PR raises the minimum to Python 3.11 and tests 3.11 through 3.14. The package API is unchanged, so on a supported interpreter it is a direct upgrade.

  • pyproject.toml: python = "^3.11", classifiers 3.11 to 3.14, mypy and codegen targets 3.11, tox envs py311 to py314.
  • pydantic >= 1.10.25: the first 1.x release that runs on Python 3.14. The lock is regenerated for the new interpreter range with every other pin unchanged.
  • check_pr.yml matrix is 3.11 to 3.14; the docs, lints, and integration jobs and both release workflows run on 3.12.

Verified locally with poetry install --with dev from the new lock on 3.11, 3.12, and 3.14, then the unit tests, black, isort, mypy, pylint, deptry, and a Sphinx build.

Integration job status

The remaining integration failures are environmental: the staging console answers the organization token with API key not found and rejects the configured organization ID, so every organization-scoped test fails at setup and the retries trip an authentication lockout. The STAGING_PYTHON_SDK_ORG_KEY and STAGING_PYTHON_SDK_ORG_ID secrets need to point at a live organization on that console. The one non-auth failure, an icon-validation message assertion, is fixed here.

hdm added 6 commits October 7, 2026 13:37
The console now accepts aggregated traffic summaries from custom
integrations through ImportAsset.communications. Each record carries a
role (client or server), a protocol, optional ports, a time window, and
optional transmitted and received byte counts.

The wrapped type lower-cases the role and upper-cases the protocol to
match the console, and rejects ports outside 1..65535 and a window that
ends before it starts, so bad records fail before upload.
The API requires RFC 3339 timestamps with an offset and rejects a
naive datetime at import. Python treats a naive datetime as local
time, so every upload-bound datetime field now attaches the local
offset through a validator, leaving aware values untouched.
The hosted runner's virtualenv no longer seeds pip for Python 3.8, so
every job pinned to 3.8 fails before installing dependencies, and the
matrix's fail-fast then cancels the other unit jobs. The docs, lints
and integration jobs move to 3.9 and 3.8 leaves the unit matrix.
Python 3.8 is past end of life and the hosted runners no longer build
environments for it. The package now requires Python 3.11 or newer,
the PR matrix covers 3.11 through 3.14, and the single-version jobs
and release workflows run on 3.12.

pydantic moves to 1.10.25 or newer, the first 1.x release that runs on
Python 3.14; the locked 1.10.15 fails there. The lock is regenerated
for the new interpreter range with every other pin unchanged.
The integration test sends bytes that are not a PNG, and the console
now rejects them for that reason before checking dimensions.
Every 429 response now raises RateLimitError, which subclasses
ClientError so existing handlers keep catching it and carries the
server's Retry-After seconds. Request.execute retries a rate-limited
request, waiting for Retry-After or an exponential backoff, before
raising. Client exposes rate_limit_retries and
rate_limit_backoff_seconds; zero retries restores the old behavior.

Integration tests gain a communications flag in the test config. The
new upload test skips until the console accepts asset communications.

The changelog describes the Python floor as a direct upgrade rather
than a breaking change.
@hdm
hdm marked this pull request as draft October 7, 2026 19:28

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant