Abstract
Large - scale automotive engineering organizations generate telemetry and engineering data from a wide range of sources extending well beyond vehicles already in customer hands, including vehicle test rigs, dynamometer benches, environmental chambers, and pr e - production fleet vehicles, each producing continuous streams of sensor readings, diagnostic logs, and test result records that must be reliably collected, routed, and made available to downstream engineering and analytics systems. As the number and diver sity of these sources grows, a centralized, monolithic ingestion approach increasingly struggles to keep pace, motivating a shift toward a distributed, event - driven ingestion architecture capable of scaling horizontally across sources, processing stages, a nd consuming applications. This paper presents a distributed telemetry ingestion architecture for large - scale automotive engineering data built on Fluent Bit, a lightweight, high - performance log and telemetry forwarder deployed at the edge of each data source, combined with a partit ioned event broker and a layer of independently scalable stream processors feeding an engineering data lake and downstream analytics systems. The architecture was evaluated across benchmark configurations spanning telemetry volume, batch size, consumer par allelism, and buffer sizing, using throughput, latency, and backpressure risk as the primary evaluation dimensions. Fluent Bit achieved the highest sustained throughput per allocated CPU core among four ingestion frameworks benchmarked, reaching approximately 38 megabytes per second per core at a single - core allocation and scaling to over 200 megabytes per second at eig ht allocated cores, outperforming Fluentd, Logstash, and a comparison streaming - native forwarder across every core allocation tested. Partition - level analysis across eighteen broker partitions showed sustained throughput variation of up to 2.5 - fold between the least and most loaded partitions under a representative workload, motivating the partition rebalancing guidance presented later in the paper, and modeled backpressure risk rose sharply once ingestion rate exceeded available consumer buffer capacity, c onsistent with the architecture's event - driven design intentionally shifting load management from the ingestion edge toward consumer - side buffering and parallelism. The paper further presents a triangulated latency surface across irregular benchmark samples spanning event rate and batch size, a percentile - level latency breakdown by pipeline stage, and a performance profile of the architecture across four load levels r anging from light to peak load. The findings support Fluent Bit combined with partitioned, event - driven processing as a practical and well - performing foundation for ingesting automotive engineering data at the scale and diversity that large - scale vehicle t est, validation, and fleet programs now require, and the paper concludes with deployment guidance for engineering organizations adopting a similar architecture.
Keywords
Fluent Bit, event - driven architecture, telemetry ingestion, distributed systems, automotive engineering data, event streaming, backpressure, partitioned event broker, stream processing, log forwarding.
References
[1] Kreps, J., Narkhede, N., and Rao, J. Kafka, A Distributed Messaging System for Log Processing.
Proceedings of the ACM SIGMOD Workshop on Networking Meets Databases, Athens, 2011.
[2] Kleppmann, M. Designing Data-Intensive Applications. Sebastopol, CA: O'Reilly Media, 2017.
[3] Akidau, T., Chernyak, S., and Lax, R. Streaming Systems, The What, Where, When, and How of LargeScale Data Processing. Sebastopol, CA: O'Reilly Media, 2018.
[4] Narkhede, N., Shapira, G., and Palino, T. Kafka, The Definitive Guide, Real-Time Data and Stream
Processing at Scale. Sebastopol, CA: O'Reilly Media, 2017.
[5] Carbone, P., Katsifodimos, A., Ewen, S., Markl, V., Haridi, S., and Tzoumas, K. Apache Flink, Stream and
Batch Processing in a Single Engine. IEEE Data Engineering Bulletin, vol. 38, no. 4, pp. 28 to 38, 2015.
[6] Zaharia, M., Das, T., Li, H., Hunter, T., Shenker, S., and Stoica, I. Discretized Streams, Fault-Tolerant
Streaming Computation at Scale. Proceedings of the 24th ACM Symposium on Operating Systems
Principles, Farmington, PA, pp. 423 to 438, 2013.
[7] Toshniwal, A., Taneja, S., Shukla, A., Ramasamy, K., Patel, J. M., Kulkarni, S., Jackson, J., Gade, K., Fu,
M., Donham, J., et al. Storm at Twitter. Proceedings of the ACM SIGMOD International Conference on
Management of Data, Snowbird, UT, pp. 147 to 156, 2014.
[8] Hwang, J. H. and Xing, C. A Comparison of Stream Processing Frameworks. Proceedings of the IEEE
International Conference on Big Data, Seattle, WA, pp. 3885 to 3890, 2018.
[9] Ranjan, R. Streaming Big Data Processing in Datacenter Clouds. IEEE Cloud Computing, vol. 1, no. 1, pp.
78 to 83, 2014.
Proceedings of the ACM SIGMOD Workshop on Networking Meets Databases, Athens, 2011.
[2] Kleppmann, M. Designing Data-Intensive Applications. Sebastopol, CA: O'Reilly Media, 2017.
[3] Akidau, T., Chernyak, S., and Lax, R. Streaming Systems, The What, Where, When, and How of LargeScale Data Processing. Sebastopol, CA: O'Reilly Media, 2018.
[4] Narkhede, N., Shapira, G., and Palino, T. Kafka, The Definitive Guide, Real-Time Data and Stream
Processing at Scale. Sebastopol, CA: O'Reilly Media, 2017.
[5] Carbone, P., Katsifodimos, A., Ewen, S., Markl, V., Haridi, S., and Tzoumas, K. Apache Flink, Stream and
Batch Processing in a Single Engine. IEEE Data Engineering Bulletin, vol. 38, no. 4, pp. 28 to 38, 2015.
[6] Zaharia, M., Das, T., Li, H., Hunter, T., Shenker, S., and Stoica, I. Discretized Streams, Fault-Tolerant
Streaming Computation at Scale. Proceedings of the 24th ACM Symposium on Operating Systems
Principles, Farmington, PA, pp. 423 to 438, 2013.
[7] Toshniwal, A., Taneja, S., Shukla, A., Ramasamy, K., Patel, J. M., Kulkarni, S., Jackson, J., Gade, K., Fu,
M., Donham, J., et al. Storm at Twitter. Proceedings of the ACM SIGMOD International Conference on
Management of Data, Snowbird, UT, pp. 147 to 156, 2014.
[8] Hwang, J. H. and Xing, C. A Comparison of Stream Processing Frameworks. Proceedings of the IEEE
International Conference on Big Data, Seattle, WA, pp. 3885 to 3890, 2018.
[9] Ranjan, R. Streaming Big Data Processing in Datacenter Clouds. IEEE Cloud Computing, vol. 1, no. 1, pp.
78 to 83, 2014.