Anomaly Detection in Security Logs Using Sequence Modeling

This project focused on self-supervised sequence modeling as a tool for performing anomaly detection in network authentication logs. Building on existing work on this topic (Tuor et al., Brown et al.), we investigated changes to the model learning method, the anomaly detection method, and to the model architecture.

Abstract

"As cyberattacks are becoming more sophisticated, automated activity logging and anomaly detection are becoming important tools for defending computer systems. Recent deep learning-based approaches have demonstrated promising results in cybersecurity contexts, typically using supervised learning combined with large amounts of labeled data. Self-supervised learning has seen growing interest as a method of training models because it does not require labeled training data, which can be difficult and expensive to collect. However, existing self-supervised approaches to anomaly detection in user authentication logs either suffer from low precision or rely on large pre-trained natural language models. This makes them slow and expensive both during training and inference. Building on previous works, we therefore propose an end-to-end trained self-supervised transformer-based sequence model for anomaly detection in user authentication events. Thanks in part to an adapted masked-language modeling (MLM) learning task and domain knowledge-based improvements to the anomaly detection method, our proposed model outperforms previous long short-term memory (LSTM)-based approaches at detecting red-team activity in the “Comprehensive, Multi-Source Cyber-Security Events” authentication event dataset, improving the area under the receiver operating characteristic curve (AUC) from 0.9760 to 0.9989 and achieving an average precision of 0.0410. Our work presents the first application of end-to-end trained self-supervised transformer models to user authentication data in a cybersecurity context, and demonstrates the potential of transformer-based approaches for anomaly detection."

Description

This project contains code for building, training, and evaluating/applying self-supervised transformer-based sequence models for anomaly detection on security log data. After the setup below has been carried out, a few different model configurations can be trained and evaluated on the linked dataset (on the models' ability to separate anomalous log entries from benevolent ones) using the options provided in the `makefile`. More advanced uses are also supported, e.g. defining a custom model architecture or using different datasets, requiring a deeper familiarisation with the project.

This project was built with Python 3.8. It was tested and successfully used on Ubuntu 20.04 LTS.

The module and all requirements can most easily be installed by running:

pip install .

If you want to be able to edit the code, instead run:

pip install -e .

(If you prefer not to install in this way, see `setup.cfg` for a list of required dependencies.)

The dataset can be found here. You will need to download the auth.txt.gz and redteam.txt.gz files and place them in data/.

Next the data must be preprocessed to the required format. Note that this may take a while. To do this most easily, with default settings, simply run:

make prepare_data

Or equivalently:

python log_analyzer/data_utils/log_file_utils.py

If you want to use a different subset of the dataset, or for other pre-processing options, have a look at the log_analyzer/data_utils/log_file_utils.py file.

The code is now ready to be run. The makefile provides a small set of useful shorthands for model training, e.g.:

make transformer_word-global CUDA=True

to train a transformer model using a global word-based vocabulary and default config (see config/) utilising the GPU (if available).

For more advanced uses have a look at the log_analyzer/train_model.py file. For example: changing the model type or architecture, training/evaluation configurations, or the dataset.