Minimal News_

Hello, I am the founder of this project and I am very happy that a friend has shared it.

ArkFlow is positioned as a lightweight distributed stream processing engine that integrates streaming batches. With the help of datafusion's huge ecosystem and ArkFlow's scalable capabilities, we hope to build a huge data processing ecosystem to help the community simplify the threshold for data processing, because we always believe that flowing data can generate greater value.

Finally, thanks to everyone for their attention.

How do you educate people on stream processing? For pipeline like systems stream processing is essential IMO - backpressure/circuit breakers/etc are critical for resilient systems. Yet I have a hard time building an engineering team that can utilize stream processing; Instead of just falling back on synchronous procedures that are easier to understand (But nearly always slower and more error prone)

seems like a simplified equivalent of https://vector.dev/

a major difference seems to be converting things to arrow and using SQL instead of using a DSL (vrl)

Previous discussion (46 days ago): https://news.ycombinator.com/item?id=43358682

Looks very similar to redpanda-connect/benthos

Very similar in appearance to Redpanda Connect (Benthos) which isn’t a bad thing at all. Would be good to elaborate on how error handling is done and what message delivery guarantees it comes with.

"High-performance" is just a meaningless buzzword if you don't have any benchmarks or performance comparisons to comparable software

Does this include broker capabilities? If not, what's a recommended broker these days (for hosting in the cloud, i.e., an EC2 instance; I know AWS has its own Mqtt Broker but it's quite pricy for high volumes).

Looks interesting, how does this compare to arroyo and vector.dev?

Very interesting - is WARC support on the roadmap?

How do the creators of this plan to make money?

So Kafka Connect and Kafka Streams but with rust?