ArkFlow：高性能 Rust 流处理引擎
Navigation Menu
切换导航 Sign in
- Product
- GitHub Copilot Write better code with AI
- GitHub Advanced Security Find and fix vulnerabilities
- Actions Automate any workflow
- Codespaces Instant dev environments
- Issues Plan and track work
- Code Review Manage code changes
- Discussions Collaborate outside of code
- Code Search Find more, search less Explore
- Why GitHub
- All features
- Documentation
- GitHub Skills
- Blog
- Solutions By company size
- Resources Topics
- Open Source
- Enterprise
- Pricing
Search or jump to...
Search code, repositories, users, issues, pull requests...
Search Clear Search syntax tips
Provide feedback
We read every piece of feedback, and take your input very seriously. Include my email address so I can be contacted Cancel Submit feedback
Saved searches
Use saved searches to filter your results more quickly
Name Query To see all available qualifiers, see our documentation. Cancel Create saved search Sign in Sign up Reseting focus You signed in with another tab or window. Reload to refresh your session. You signed out in another tab or window. Reload to refresh your session. You switched accounts on another tab or window. Reload to refresh your session. Dismiss alert {{ message }} arkflow-rs / **arkflow ** Public
- Notifications You must be signed in to change notification settings
- Fork 20
- Star 654
高性能 Rust 流处理引擎，提供强大的数据流处理能力，支持多种输入/输出源和处理器。 arkflow-rs.com
License
Apache-2.0 license 654 stars 20 forks Branches Tags Activity Star Notifications You must be signed in to change notification settings
Additional navigation options
arkflow-rs/arkflow
main BranchesTags Go to file Code
Folders and files
Name| Name| Last commit message| Last commit date
---|---|---|---
Latest commit
History
195 Commits
.github| .github
crates| crates
docker| docker
docs| docs
examples| examples
.gitignore| .gitignore
Cargo.lock| Cargo.lock
Cargo.toml| Cargo.toml
LICENSE| LICENSE
README.md| README.md
README_zh.md| README_zh.md
View all files
Repository files navigation
ArkFlow
English | 中文
高性能 Rust 流处理引擎，提供强大的数据流处理能力，支持多种输入/输出源和处理器。
功能特性
- 高性能：构建于 Rust 和 Tokio 异步运行时之上，提供卓越的性能和低延迟。
- 多种数据源：支持 Kafka、MQTT、HTTP、文件以及其他输入/输出源。
- 强大的处理能力：内置 SQL 查询、JSON 处理、Protobuf 编码/解码、批量处理等处理器。
- 可扩展：模块化设计，易于扩展新的输入、输出和处理器组件。
安装
从源码构建
# 克隆仓库
git clone https://github.com/arkflow-rs/arkflow.git
cd arkflow
# 构建项目
cargo build --release
# 运行测试
cargo test
快速开始
- 创建一个配置文件
config.yaml：
logging:
level: info
streams:
- input:
type: "generate"
context: '{ "timestamp": 1625000000000, "value": 10, "sensor": "temp_1" }'
interval: 1s
batch_size: 10
pipeline:
thread_num: 4
processors:
- type: "json_to_arrow"
- type: "sql"
query: "SELECT * FROM flow WHERE value >= 10"
output:
type: "stdout"
error_output:
type: "stdout"
- 运行 ArkFlow：
./target/release/arkflow --config config.yaml
配置指南
ArkFlow 使用 YAML 格式的配置文件，支持以下主要配置项：
顶层配置
logging:
level: info # 日志级别: debug, info, warn, error
streams: # 流定义列表
- input: # 输入配置
# ...
pipeline: # 处理流水线配置
# ...
output: # 输出配置
# ...
error_output: # 错误输出配置
# ...
buffer: # 缓冲区配置
# ...
输入组件
ArkFlow 支持多种输入源：
- Kafka：从 Kafka 主题读取数据
- MQTT：订阅来自 MQTT 主题的消息
- HTTP：通过 HTTP 接收数据
- File：使用 SQL 从文件（CSV、JSON、Parquet、Avro、Arrow）读取数据
- Generator：生成测试数据
- Database：从数据库（MySQL、PostgreSQL、SQLite、Duckdb）查询数据
示例：
input:
type: kafka
brokers:
- localhost:9092
topics:
- test-topic
consumer_group: test-group
client_id: arkflow
start_from_latest: true
处理器
ArkFlow 提供多种数据处理器：
- JSON：JSON 数据处理和转换
- SQL：使用 SQL 查询处理数据
- Protobuf：Protobuf 编码/解码
- Batch Processing：批量处理消息
示例：
pipeline:
thread_num: 4
processors:
- type: json_to_arrow
- type: sql
query: "SELECT * FROM flow WHERE value >= 10"
输出组件
ArkFlow 支持多种输出目标：
- Kafka：将数据写入 Kafka 主题
- MQTT：将消息发布到 MQTT 主题
- HTTP：通过 HTTP 发送数据
- Standard Output：将数据输出到控制台
- Drop：丢弃数据
示例：
output:
type: kafka
brokers:
- localhost:9092
topic:
type: value
value: test-topic
client_id: arkflow-producer
错误输出组件
ArkFlow 支持多种错误输出目标：
- Kafka：将错误数据写入 Kafka 主题
- MQTT：将错误消息发布到 MQTT 主题
- HTTP：通过 HTTP 发送错误数据
- Standard Output：将错误数据输出到控制台
- Drop：丢弃错误数据
示例：
error_output:
type: kafka
brokers:
- localhost:9092
topic:
type: value
value: error-topic
client_id: error-arkflow-producer
缓冲组件
ArkFlow 提供缓冲功能来处理背压和临时存储消息：
- Memory Buffer：内存缓冲区，适用于高吞吐量场景和窗口聚合
示例：
buffer:
type: memory
capacity: 10000 # 要缓冲的最大消息数
timeout: 10s # 缓冲消息的最长时间
示例
Kafka 到 Kafka 数据处理
streams:
- input:
type: kafka
brokers:
- localhost:9092
topics:
- test-topic
consumer_group: test-group
pipeline:
thread_num: 4
processors:
- type: json_to_arrow
- type: sql
query: "SELECT * FROM flow WHERE value > 100"
output:
type: kafka
brokers:
- localhost:9092
topic: processed-topic
生成测试数据并处理
streams:
- input:
type: "generate"
context: '{ "timestamp": 1625000000000, "value": 10, "sensor": "temp_1" }'
interval: 1ms
batch_size: 10000
pipeline:
thread_num: 4
processors:
- type: "json_to_arrow"
- type: "sql"
query: "SELECT count(*) FROM flow WHERE value >= 10 group by sensor"
output:
type: "stdout"
ArkFlow 插件
许可证
ArkFlow 使用 Apache License 2.0 许可证。
社区
Discord: https://discord.gg/CwKhzb8pux
如果你喜欢或正在使用此项目来学习或启动你的解决方案，请给它一个 star⭐。谢谢！
关于
高性能 Rust 流处理引擎，提供强大的数据流处理能力，支持多种输入/输出源和处理器。 arkflow-rs.com
Topics
mysql rust flow sql kafka stream sqlite postgresql tokio rust-lang tokio-rs datafusion duckdb
Resources
License
Apache-2.0 license Activity Custom properties
Stars
Watchers
Forks
Releases 5
Release v0.2.1 Latest Apr 17, 2025 + 4 releases
Packages 0
Contributors 4
![@dependabot[bot]](https://avatars.githubusercontent.com/in/29110?s=64&v=4)
Languages
Footer
Footer navigation
You can’t perform that action at this time.