Foucault03 is a anomaly log monitoring system.
- Perl (> 5.20)
- Ansible
Foucault03 system monitors logs treated by fluentd and tagged "multilinelog.**". The system detects anomaly logs defined by pre-generated patterns. Patterns are builded from sample logs and build rules. If you hope to monitor /var/log/messages, you may use /var/log/messages for a sample log as is.
Build rules may specify variable words in the logs by regexp, like following:
\d+\.\d+\.\d+\.\d+
(IP address)(?:Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\x20[\x200-9][0-9]\x20[\x200-9][0-9]:[0-9][0-9]:[0-9][0-9]
(timestamp)
git clone https://github.com/frisky-gh/foucault03.git
cd foucault03
sudo setup.sh
vi conf/fluentd.conf
./bin/foucaultctl build_fluentd_conf
./bin/foucaultctl build_patterns
/etc/init.d/td-agent restart
foucaultctl <SUBCOMMAND>
SUBCOMMAND is one of following:
- build_fluentd_conf
- Build a conf file for fluentd.
- build_patterns
- Build all pattern files which related to updated rules or sample file.
- list_unmonitoredlog
- List up all unmonitoredlogs.
- capture_unmonitoredlog
- Caputure unmonitoredlogs into capturedlogs.
- capture_anomalylog
- Caputure anomalylogs into capturedlogs.
- show_capturedlog
- Show all caputuredlogs.
- strip_capturedlog
- Strip redundant capturedlogs.
- import_capturedlog
- Append all capturedlogs into samples.
- strip_samples
- Strip redundant samples.
- conf/fluentd.conf
- Configuration file for fluentd.
- conf/fluentd.tt
- Template file for a fluentd.conf.
- conf/deliver.conf
- Configuration file for report deliveries.
- conf/deliver_flash.tt
- Template file for a flash report of anomaly log by mail.
- conf/deliver_daily.tt
- conf/fluentd/fluentd_foucault03.conf
- fluentd configuration file. It's included by /etc/td-agent/td-agent.conf.
- conf/patterns/*.rules
- Build rules file. You may customize it to adjust to your VMs.
- conf/patterns/*.sample
- Sample log file. You may put log file you want to target. Its size is hoped to be less than < 1MB.
- conf/patterns/*.pattern
- Pattern file. It's builded from a sample log and build rules, by `foucaultctl build_patterns`.
- anomalylog/*
- File of anomaly logs detected by foucault03.
- unmonitoredlog/*
- File of logs that is not monitored.
- capturedlog/*
- File of logs that is caputured from anomalylog or unmonitoredlog by 'capture_anomalylog' or 'capture_unmonitoredlog' subcommand.
- deliveredevent/*
- File of events that is delivered to recipients.
- undeliveredevent/*
- File of events file that is not delivered to any recipients.