Serialization changes

[tom-kuchler]

The requests and responses currently have intermixed metadata and context data, should separate them so we can update the metadata layout etc. more easily without having impact on serialization and deserialization.
Move to all metadata in the beginning and then all data, should make it easier to serialize and deserialize as well as improve performance.

[rabbull]

Hi @tom-kuchler! I came across this project through the paper and found it really interesting. This issue looks like a good place for me to get started. Are external contributions welcome here? If so, I'd be happy to work on this.

[tom-kuchler]

Hi Rabbull,

Sorry for the slow response, I was on vacation.
I think generally we are not closed to external contributions, but it is likely somewhat hard to get the full context.
This could indeed be a good starter task, since it is technically not very complicated, but it has a few other things that make it a bit complicated.
The main issue is synchronizing this change with also replacing the serialization on the other things we are building that interact with dandelion.
I think at the moment that would mainly concern @tostocker would it be fine with your client to switch the serialization in the near future or how complicated would that be?

Also @romameie, I think the ideal setup would probably be based on something like protobuf, would that cause any issues for you, since that adds another meta compiler? I think the proto compiler just generates rust, so if your systems supports the rust cross compilation this should be fine, but it does potentially add complexity.

[tostocker]

Yes, should not be too much work on my side.

[tom-kuchler]

Sounds good.
I think then we can move forward with this.
The plan was to do something that is similar to what we are doing in the multi node communication.
In my head, the body would become a version number for the protocol, the metadata size, the metadata structure (the protobuf serialized version of the current metadata) and then the data.
The important change here is, that we want to move the data to after the metadata and not interleave it with the metadata, since that should simplify the deserialization and remove size restrictions from the serialization on the payload size.