Is your feature request related to a problem? Please describe.
Currently, UrbanOS has two potential destinations for data after ingestion: Websocket Streaming and Trino (via Hive). These are specialized technologies that are not universally applicable. Trino is very heavyweight for small data volumes, and Websockets are often only used for sub-minute data feeds. The platform needs a broader selection of data storage solutions.
Describe the solution you'd like
Similar to how the platform supports multiple ingestions, it should be able to support multiple destinations. Currently it supports Trino (a big-data solution), and Discovery Streams (a websocket streaming api). We should structure UrbanOS so that it is not only easy to add additional destinations, but to customize them from the Curator UI.
Some initial prospects for extra destinations:
Kafka
This is easy, but needs to be able to be any arbitrary Kafka cluster to allow a user to route messages onto other Kafkas.
Postgres
One of the most common databases, we already use one for the metadata for the system. This would give us a lightweight alternative (or addition!) to Trino.
API Push
This could allow us to route transformed and filtered messages to a different system, allowing for webhook support and others.
Vendor-Specific Destinations
- Microsoft Event Hub
- S3 Buckets
- etc.
Is your feature request related to a problem? Please describe.
Currently, UrbanOS has two potential destinations for data after ingestion: Websocket Streaming and Trino (via Hive). These are specialized technologies that are not universally applicable. Trino is very heavyweight for small data volumes, and Websockets are often only used for sub-minute data feeds. The platform needs a broader selection of data storage solutions.
Describe the solution you'd like
Similar to how the platform supports multiple ingestions, it should be able to support multiple destinations. Currently it supports Trino (a big-data solution), and Discovery Streams (a websocket streaming api). We should structure UrbanOS so that it is not only easy to add additional destinations, but to customize them from the Curator UI.
Some initial prospects for extra destinations:
Kafka
This is easy, but needs to be able to be any arbitrary Kafka cluster to allow a user to route messages onto other Kafkas.
Postgres
One of the most common databases, we already use one for the metadata for the system. This would give us a lightweight alternative (or addition!) to Trino.
API Push
This could allow us to route transformed and filtered messages to a different system, allowing for webhook support and others.
Vendor-Specific Destinations