Hybrid machine learning for a real-time anomaly detection system in computer networks with the ELK stack using system logs and netflow data
- Pathum Thani, Thailand : Asian Institute of Technology, 2025
- 90 leaves : ill.+ 1 online resource
- Thesis ; no. DSAI-25-08 .
- Asian Institute of Technology. Thesis ; no. DSAI-25-08 .
A thesis submitted in partial fulfillment of the requirements for the degree of Master of Engineering in Data Science and Artificial Intelligence
Thesis (M. Eng.) - Asian Institute of Technology, 2025
The increasing intricacy and prevalence of cyber threats in modern computer networks highlight the need for effective anomaly detection systems to protect sensitive information. Traditional methods face challenges such as limited real-time processing, reliance on simple binary classification, and inadequate evaluation using realistic datasets. To address these issues, this research proposes a hybrid machine learning framework for anomaly detection. In the first stage, an autoencoder is used to learn latent represen tations of normal traffic, while an Isolation Forest algorithm detects anomalies based on anomaly scores. The Receiver Operating Characteristic (ROC) curve and Youdens Index are employed to determine thresholds, which are then validated against the test labels of the UNSW-NB15 dataset to obtain baseline performance metrics. In the sec ond stage, supervised models including Decision Tree, XGBoost, and Random Forest are trained on the latent features, reconstruction error of the autoencoder, and anomaly scores from the Isolation Forest. Among these, Random Forest achieved the best per formance, significantly improving upon the unsupervised baseline, with an accuracy of 98.81%, precision of 92.00%, recall of 99.25%, F1-score of 95.49%, and a false posi tive rate of only 1.25%. To enable real-time usage, the framework is deployed with the Elastic Stack (ELK), allowing automated alerting, continuous monitoring, and visual ization of security events. The deployed system is further evaluated using real-world NetFlow and Syslog data collected from the Operational Technology (OT) network of the Provincial Electricity Authority (PEA). Controlled attack scenarios including TCP, UDP, and ICMP flood attacks are generated using the Nping tool to validate real-time anomaly detection. The results confirm that the proposed framework performs effec tively under realistic operational conditions and is suitable for practical deployment in critical infrastructure environments.