Empowering Cyber Security by Enabling 7 Times Faster Log Analysis>
Medium – Apache Doris
The cyber security service provider faced several issues with their old log storage and analysis system (LSAS), which used StarRocks as the analytic engine.
They experienced slow data writing and slow query execution, which hindered the system’s stability and performance.
To overcome these challenges, they decided to upgrade their architecture and implement Apache Doris as their new database tool.
The company tested Apache Doris and observed significant improvements in data writing speed, query execution speed, and storage costs.
Here are the key findings from their evaluation:
1) Data Writing Speed:
– With Apache Doris, they achieved a 300% increase in data writing speed compared to the old system.
– By using only 30% of CPU usage on a 3-server cluster, they were able to handle the writing workload, leading to savings of over 70% on hardware resources.
– Enabling inverted index for half of the fields further increased the writing speed by 50%.
2) Storage Cost:
– Apache Doris with inverted index enabled resulted in 60% lower storage space compared to the old system without inverted indexes.
– The data compression ratio was 1:5) 7, meaning that Apache Doris achieved efficient storage utilization.
– The adoption of columnar storage and the ZStandard compression algorithm contributed to space savings and enhanced log processing.
3) Query Speed:
– The new Apache Doris-based system improved query execution time by a factor of 7 on average for 79 frequently executed SQL statements.
– Apache Doris introduced features and optimizations specifically tailored for log analysis.
– Notable improvements were seen in queries that involved keyword searches using inverted index and searching using the LIKE operator with NGram BloomFilter.
– For example, a query was sped up 88) 2 times with inverted index and 44) 4 times with NGram BloomFilter.
4) Visualized Operation & Maintenance:
– Doris Manager, a visualized cluster management tool, was contributed by SelectDB, the commercial supporter of Apache Doris.
– Doris Manager enabled efficient cluster maintenance, including monitoring, inspection, configuration modification, scaling, and upgrading.
– It also provided a visualized WebUI for log analysis, similar to Kibana, allowing users to perform keyword searches, analyze trends, filter fields, and have a detailed view of log data.
Based on the successful trial run, the cyber security service provider officially replaced their old LSAS with the Apache Doris-based system in production.
They experienced faster data ingestion, improved query performance, and easier maintenance and analysis.
Their next goal is to expand the coverage of JSON data types and leverage the upcoming Apache Doris 2) 1, which will provide additional support for semi-structured data analysis with a new Variant data type and JSON data of any structures.
Link: https://medium.com/@ApacheDoris/empowering-cyber-security-by-enabling-7-times-faster-log-analysis-776beb847200