What
Since Typestream is all about leveraging the Kafka ecosystem without anyone knowing much about Kafka, we want to make it easy for engineers to start using it.
One way would be having an rest endpoint source that will write to Kafka topics.
The tricky thing would be that we need to register the schema so that we can a) validate the data and b) we always maintain a fully schemaed inside the Typestream ecosystem.
Example would be a webhook coming from Zapier. Say it's every row change/update/delete from a Google Sheet.
Open questions
- we don't secure the endpoints by design. We assume the person implementing will put an API gateway in front of it
- we want as our of the box as possible. Would we use the Kafka connect rest interface? Is there a kotlin/java library?
- how will we input the schema? Probably the user will have to know the expected schema and register it when it does the Typestream plan/deploy. I'm not sure how the user can test it.
- what should happen if the message doesn't conform? Likely an HTTP error right? Do we dead letter queue it?
@claude can you think this all through? Review the current node design
What
Since Typestream is all about leveraging the Kafka ecosystem without anyone knowing much about Kafka, we want to make it easy for engineers to start using it.
One way would be having an rest endpoint source that will write to Kafka topics.
The tricky thing would be that we need to register the schema so that we can a) validate the data and b) we always maintain a fully schemaed inside the Typestream ecosystem.
Example would be a webhook coming from Zapier. Say it's every row change/update/delete from a Google Sheet.
Open questions
@claude can you think this all through? Review the current node design