Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Moving logstash dockerfile and associated files to its own folder #302

Merged
merged 19 commits into from
Aug 5, 2024

Conversation

f-galland
Copy link
Member

Description

This PR moves the integrations' logstash dependencies to its own folder for ease of maintenance.

Issues Resolved

Resolves #299

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Failing checks are inspected and point to the corresponding known issue(s) (See: Troubleshooting Failing Builds)
  • Commits are signed per the DCO using --signoff
  • Commit changes are listed out in CHANGELOG.md file (See: Changelog)
  • Public documentation issue/PR created

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

@f-galland f-galland marked this pull request as ready for review July 10, 2024 12:21
@f-galland f-galland requested a review from a team as a code owner July 10, 2024 12:21
@f-galland f-galland linked an issue Jul 10, 2024 that may be closed by this pull request
2 tasks
@f-galland
Copy link
Member Author

Alerts are now being pushed to the S3 in json.gz format:

2024-07-11_09-01

The lambda function reads the files and generates the appropriate OCSF and parquet files:

2024-07-11_09-01_1

2024-07-11_09-01_2

@f-galland
Copy link
Member Author

The parquet file is properly populated:

$ parquet-tools show --format github --head 5 ext_wazuh_region\=us-east-1_accountId\=111111111111_eventDay\=20240711_986b64bbdbc54d73a38a42c7fc0957b9.parquet
|   activity_id | category_name   |   category_uid | class_name       |   class_uid |   count | message                                         | metadata                                                                                                                                                | raw_data                                                              | resources                           |   risk_score |   severity_id |   status_id |       time |   type_uid | unmapped                                                                                               | analytic                                                                                          | attacks                                                                                                     | finding                                                                                                                        |   state_id |
|---------------|-----------------|----------------|------------------|-------------|---------|-------------------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------|-------------------------------------|--------------|---------------|-------------|------------|------------|--------------------------------------------------------------------------------------------------------|---------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------|------------|
|             1 | Findings        |              2 | Security Finding |        2001 |       6 | Host-based anomaly detection event (rootcheck). | {'log_name': 'Security events', 'log_provider': 'Wazuh', 'product': {'lang': 'en', 'name': 'Wazuh', 'vendor_name': 'Wazuh, Inc,.'}, 'version': '1.1.0'} | Rootkit 'Slapper' detected by the presence of file '/tmp/.bugtraq.c'. | [{'name': 'RHEL7', 'uid': '001'}]   |            7 |             2 |          99 | 1720697518 |     200101 | {'data_sources': array(['rootcheck', 'wazuh-manager'], dtype=object), 'nist': array([], dtype=object)} | {'category': 'wazuh, rootcheck', 'name': 'rootcheck', 'type': 'Rule', 'type_id': 1, 'uid': '510'} | [{'tactic': {'name': 'N/A', 'uid': 'N/A'}, 'technique': {'name': 'N/A', 'uid': 'N/A'}, 'version': 'v13.1'}] | {'title': 'Host-based anomaly detection event (rootcheck).', 'types': array(['log'], dtype=object), 'uid': '1580123327.49031'} |          1 |
|             1 | Findings        |              2 | Security Finding |        2001 |      11 | Audit: Command: /usr/sbin/id                    | {'log_name': 'Security events', 'log_provider': 'Wazuh', 'product': {'lang': 'en', 'name': 'Wazuh', 'vendor_name': 'Wazuh, Inc,.'}, 'version': '1.1.0'} |                                                                       | [{'name': 'Amazon', 'uid': '002'}]  |            3 |             1 |          99 | 1720697423 |     200101 | {'data_sources': array(['', 'wazuh-manager'], dtype=object), 'nist': array([], dtype=object)}          | {'category': 'audit, audit_command', 'name': 'N/A', 'type': 'Rule', 'type_id': 1, 'uid': '80784'} | [{'tactic': {'name': 'N/A', 'uid': 'N/A'}, 'technique': {'name': 'N/A', 'uid': 'N/A'}, 'version': 'v13.1'}] | {'title': 'Audit: Command: /usr/sbin/id', 'types': array(['N/A'], dtype=object), 'uid': '1580123327.49031'}                    |          1 |
|             1 | Findings        |              2 | Security Finding |        2001 |      16 | Audit: Command: /usr/sbin/consoletype           | {'log_name': 'Security events', 'log_provider': 'Wazuh', 'product': {'lang': 'en', 'name': 'Wazuh', 'vendor_name': 'Wazuh, Inc,.'}, 'version': '1.1.0'} |                                                                       | [{'name': 'Debian', 'uid': '007'}]  |            3 |             1 |          99 | 1720697695 |     200101 | {'data_sources': array(['', 'wazuh-manager'], dtype=object), 'nist': array([], dtype=object)}          | {'category': 'audit, audit_command', 'name': 'N/A', 'type': 'Rule', 'type_id': 1, 'uid': '80784'} | [{'tactic': {'name': 'N/A', 'uid': 'N/A'}, 'technique': {'name': 'N/A', 'uid': 'N/A'}, 'version': 'v13.1'}] | {'title': 'Audit: Command: /usr/sbin/consoletype', 'types': array(['N/A'], dtype=object), 'uid': '1580123327.49031'}           |          1 |
|             1 | Findings        |              2 | Security Finding |        2001 |      13 | Audit: Command: /usr/sbin/hostname              | {'log_name': 'Security events', 'log_provider': 'Wazuh', 'product': {'lang': 'en', 'name': 'Wazuh', 'vendor_name': 'Wazuh, Inc,.'}, 'version': '1.1.0'} |                                                                       | [{'name': 'Windows', 'uid': '006'}] |            3 |             1 |          99 | 1720697604 |     200101 | {'data_sources': array(['', 'wazuh-manager'], dtype=object), 'nist': array([], dtype=object)}          | {'category': 'audit, audit_command', 'name': 'N/A', 'type': 'Rule', 'type_id': 1, 'uid': '80784'} | [{'tactic': {'name': 'N/A', 'uid': 'N/A'}, 'technique': {'name': 'N/A', 'uid': 'N/A'}, 'version': 'v13.1'}] | {'title': 'Audit: Command: /usr/sbin/hostname', 'types': array(['N/A'], dtype=object), 'uid': '1580123327.49031'}              |          1 |
|             1 | Findings        |              2 | Security Finding |        2001 |      17 | Audit: Command: /usr/sbin/sh                    | {'log_name': 'Security events', 'log_provider': 'Wazuh', 'product': {'lang': 'en', 'name': 'Wazuh', 'vendor_name': 'Wazuh, Inc,.'}, 'version': '1.1.0'} |                                                                       | [{'name': 'Ubuntu', 'uid': '004'}]  |            3 |             1 |          99 | 1720697690 |     200101 | {'data_sources': array(['', 'wazuh-manager'], dtype=object), 'nist': array([], dtype=object)}          | {'category': 'audit, audit_command', 'name': 'N/A', 'type': 'Rule', 'type_id': 1, 'uid': '80790'} | [{'tactic': {'name': 'N/A', 'uid': 'N/A'}, 'technique': {'name': 'N/A', 'uid': 'N/A'}, 'version': 'v13.1'}] | {'title': 'Audit: Command: /usr/sbin/sh', 'types': array(['N/A'], dtype=object), 'uid': '1580123327.49031'}                    |          1 |

@f-galland f-galland linked an issue Jul 11, 2024 that may be closed by this pull request
2 tasks
Copy link
Member

@AlexRuiz7 AlexRuiz7 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • Indexer to OpenSearch - ✔️
  • Indexer to Elastic - ✔️
  • Indexer to Splunk - ✔️
  • Indexer to Amazon Security Lake - ✔️

integrations/docker/compose.amazon-security-lake.yml Outdated Show resolved Hide resolved
integrations/docker/compose.indexer-opensearch.yml Outdated Show resolved Hide resolved
@AlexRuiz7 AlexRuiz7 merged commit d7aecfe into 4.9.0 Aug 5, 2024
7 checks passed
@AlexRuiz7 AlexRuiz7 deleted the 299-reuse-logstash-dockerfile branch August 5, 2024 13:40
AlexRuiz7 added a commit that referenced this pull request Aug 20, 2024
* Adding /var/run/wazuh-indexer to the list of ignored files (#339)

* Fix sysv files (#343)

* Change sysv script to reference opensearch script

* Correct permissions on sysv script

* Updating sysv service file permissions to match 4.8

* Moving logstash dockerfile and associated files to its own folder (#302)

* Moving logstash dockerfile and associated files to its own folder

* Removing unused context from logstash container in splunk compose

* Changing context in manager to splunk integration to newly created logstash folder

* Using common logstash container for Security Lake integration

* Removing unused dockerfiles

* Adding gzip to logstash pipeline

* Adding gzip decompression to lambda

* Changing aws REGION variable name

* Cleanup commented code

* Adding environment variables for docker images versions

* Adding logstash version to .env

* Fixing minor issues in compose files

* Removing certs configuration

* Format YAML and Readme file from 3rd-party integrations

* Remove unused code

* Upgrade integrations to the latest Wazuh version

* Fic environment variable

* Remove unused code and fix ASL documentation

---------

Co-authored-by: Álex Ruiz <[email protected]>

* Fixing chown command (#347)

---------

Co-authored-by: Federico Gustavo Galland <[email protected]>
AlexRuiz7 added a commit that referenced this pull request Aug 20, 2024
* Adding /var/run/wazuh-indexer to the list of ignored files (#339)

* Fix sysv files (#343)

* Change sysv script to reference opensearch script

* Correct permissions on sysv script

* Updating sysv service file permissions to match 4.8

* Moving logstash dockerfile and associated files to its own folder (#302)

* Moving logstash dockerfile and associated files to its own folder

* Removing unused context from logstash container in splunk compose

* Changing context in manager to splunk integration to newly created logstash folder

* Using common logstash container for Security Lake integration

* Removing unused dockerfiles

* Adding gzip to logstash pipeline

* Adding gzip decompression to lambda

* Changing aws REGION variable name

* Cleanup commented code

* Adding environment variables for docker images versions

* Adding logstash version to .env

* Fixing minor issues in compose files

* Removing certs configuration

* Format YAML and Readme file from 3rd-party integrations

* Remove unused code

* Upgrade integrations to the latest Wazuh version

* Fic environment variable

* Remove unused code and fix ASL documentation

---------

Co-authored-by: Álex Ruiz <[email protected]>

* Fixing chown command (#347)

---------

Co-authored-by: Federico Gustavo Galland <[email protected]>
AlexRuiz7 added a commit that referenced this pull request Sep 6, 2024
* Adding /var/run/wazuh-indexer to the list of ignored files (#339)

* Fix sysv files (#343)

* Change sysv script to reference opensearch script

* Correct permissions on sysv script

* Updating sysv service file permissions to match 4.8

* Moving logstash dockerfile and associated files to its own folder (#302)

* Moving logstash dockerfile and associated files to its own folder

* Removing unused context from logstash container in splunk compose

* Changing context in manager to splunk integration to newly created logstash folder

* Using common logstash container for Security Lake integration

* Removing unused dockerfiles

* Adding gzip to logstash pipeline

* Adding gzip decompression to lambda

* Changing aws REGION variable name

* Cleanup commented code

* Adding environment variables for docker images versions

* Adding logstash version to .env

* Fixing minor issues in compose files

* Removing certs configuration

* Format YAML and Readme file from 3rd-party integrations

* Remove unused code

* Upgrade integrations to the latest Wazuh version

* Fic environment variable

* Remove unused code and fix ASL documentation

---------

Co-authored-by: Álex Ruiz <[email protected]>

* Fixing chown command (#347)

* Add Changelog for 4.9.0 (#364)

---------

Co-authored-by: Federico Gustavo Galland <[email protected]>
AlexRuiz7 added a commit that referenced this pull request Sep 6, 2024
* Merge 4.9.0 into 4.9.1 (#376)

* Adding /var/run/wazuh-indexer to the list of ignored files (#339)

* Fix sysv files (#343)

* Change sysv script to reference opensearch script

* Correct permissions on sysv script

* Updating sysv service file permissions to match 4.8

* Moving logstash dockerfile and associated files to its own folder (#302)

* Moving logstash dockerfile and associated files to its own folder

* Removing unused context from logstash container in splunk compose

* Changing context in manager to splunk integration to newly created logstash folder

* Using common logstash container for Security Lake integration

* Removing unused dockerfiles

* Adding gzip to logstash pipeline

* Adding gzip decompression to lambda

* Changing aws REGION variable name

* Cleanup commented code

* Adding environment variables for docker images versions

* Adding logstash version to .env

* Fixing minor issues in compose files

* Removing certs configuration

* Format YAML and Readme file from 3rd-party integrations

* Remove unused code

* Upgrade integrations to the latest Wazuh version

* Fic environment variable

* Remove unused code and fix ASL documentation

---------

Co-authored-by: Álex Ruiz <[email protected]>

* Fixing chown command (#347)

* Add Changelog for 4.9.0 (#364)

---------

Co-authored-by: Federico Gustavo Galland <[email protected]>

* Merge 4.9.1 into 4.9.2

---------

Co-authored-by: Federico Gustavo Galland <[email protected]>
AlexRuiz7 added a commit that referenced this pull request Sep 6, 2024
* Merge 4.9.0 into 4.9.1 (#376)

* Adding /var/run/wazuh-indexer to the list of ignored files (#339)

* Fix sysv files (#343)

* Change sysv script to reference opensearch script

* Correct permissions on sysv script

* Updating sysv service file permissions to match 4.8

* Moving logstash dockerfile and associated files to its own folder (#302)

* Moving logstash dockerfile and associated files to its own folder

* Removing unused context from logstash container in splunk compose

* Changing context in manager to splunk integration to newly created logstash folder

* Using common logstash container for Security Lake integration

* Removing unused dockerfiles

* Adding gzip to logstash pipeline

* Adding gzip decompression to lambda

* Changing aws REGION variable name

* Cleanup commented code

* Adding environment variables for docker images versions

* Adding logstash version to .env

* Fixing minor issues in compose files

* Removing certs configuration

* Format YAML and Readme file from 3rd-party integrations

* Remove unused code

* Upgrade integrations to the latest Wazuh version

* Fic environment variable

* Remove unused code and fix ASL documentation

---------

Co-authored-by: Álex Ruiz <[email protected]>

* Fixing chown command (#347)

* Add Changelog for 4.9.0 (#364)

---------

Co-authored-by: Federico Gustavo Galland <[email protected]>

* Merge 4.9.1 into 4.9.2

---------

Co-authored-by: Federico Gustavo Galland <[email protected]>
AlexRuiz7 added a commit that referenced this pull request Sep 6, 2024
* Merge 4.9.0 into 4.9.1 (#376)

* Adding /var/run/wazuh-indexer to the list of ignored files (#339)

* Fix sysv files (#343)

* Change sysv script to reference opensearch script

* Correct permissions on sysv script

* Updating sysv service file permissions to match 4.8

* Moving logstash dockerfile and associated files to its own folder (#302)

* Moving logstash dockerfile and associated files to its own folder

* Removing unused context from logstash container in splunk compose

* Changing context in manager to splunk integration to newly created logstash folder

* Using common logstash container for Security Lake integration

* Removing unused dockerfiles

* Adding gzip to logstash pipeline

* Adding gzip decompression to lambda

* Changing aws REGION variable name

* Cleanup commented code

* Adding environment variables for docker images versions

* Adding logstash version to .env

* Fixing minor issues in compose files

* Removing certs configuration

* Format YAML and Readme file from 3rd-party integrations

* Remove unused code

* Upgrade integrations to the latest Wazuh version

* Fic environment variable

* Remove unused code and fix ASL documentation

---------

Co-authored-by: Álex Ruiz <[email protected]>

* Fixing chown command (#347)

* Add Changelog for 4.9.0 (#364)

---------

Co-authored-by: Federico Gustavo Galland <[email protected]>

* Merge 4.9.1 into 4.9.2

---------

Co-authored-by: Federico Gustavo Galland <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
2 participants