High speed parser using SIMD and Aho-corasick ? ( I think i made some silly mistake in my repo )

[Shabhareash]

I had an idea for a feature to be added to vector especially for high volume log sources (Tb of logs per day). Essentially we use a set of static rules and remove the flexibility of vector. I have the repo here : https://github.com/Shabhareash/RustLogger .
I know the figures in the repo does not make sense (But that is what I got ) and to the best of my knowledge it seems right . Kindly let me know where I made the mistake. Thanks

[pront]

Thanks for sharing!

A few things that would help us give useful feedback:

• This looks more like a Vector-level ingestion/pipeline concern than a VRL language feature. Could you clarify what change to VRL specifically you're proposing?
• For the benchmark discrepancy, we'd need to see how the Vector side was configured (topology, transforms used, traffic generator) to compare fairly against a single-purpose Rust binary. I would suggest creating a folder in your repo with all the experiment scripts and details.

[Shabhareash]

Thanks for the reply!

First of all, I'd like to apologize for opening this discussion in the VRL repository. I know it's primarily intended for VRL-related discussions. I considered posting it in the Vector repository, but since it doesn't have an Ideas category, I wasn't sure where this kind of benchmarking and design discussion would best fit.

As requested, I've updated my repository with a dedicated folder containing the benchmark scripts, the benchmarking methodology, the Vector configuration used for the comparison, and the flame graphs for both RustLogger and Vector. I hope this provides enough detail to review the benchmark. If there's anything else you'd like me to include, I'd be happy to add it.