Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dtspo 18633 add data collection rule #1

Merged
merged 20 commits into from
Sep 4, 2024

Conversation

ieuanb74
Copy link
Contributor

@ieuanb74 ieuanb74 commented Sep 4, 2024

Jira link

https://tools.hmcts.net/jira/browse/DTSPO-18633

Change description

  • Decided to move the data collection rules to their own repo as they are static resources
  • This will deploy 2 per LA workspace (one Linux and one Windows)
  • Will use data sources in terraform-module-vm-bootstrap to associate VMs to the correct rule

Testing done

Checklist

  • commit messages are meaningful and follow good commit message guidelines
  • README and other documentation has been updated / added (if needed)
  • tests have been updated / new tests has been added (if needed)
  • Does this PR introduce a breaking change

@hmcts-platform-operations

Plan Result (sandbox)

Plan: 2 to add, 0 to change, 0 to destroy.
  • Create
    • azurerm_monitor_data_collection_rule.linux_data_collection_rule
    • azurerm_monitor_data_collection_rule.windows_data_collection_rule
Change Result (Click me)
  # azurerm_monitor_data_collection_rule.linux_data_collection_rule will be created
  + resource "azurerm_monitor_data_collection_rule" "linux_data_collection_rule" {
      + description         = "A data collection rule for collecting Linux syslog and performance counters and sending them to the appropriate log analytics workspace."
      + id                  = (known after apply)
      + immutable_id        = (known after apply)
      + kind                = "Linux"
      + location            = "uksouth"
      + name                = "ama-linux-vm-logs"
      + resource_group_name = "oms-automation"

      + data_flow {
          + destinations = [
              + "hmcts-sandbox",
            ]
          + streams      = [
              + "Microsoft-Perf",
            ]
        }
      + data_flow {
          + destinations = [
              + "hmcts-sandbox",
            ]
          + streams      = [
              + "Microsoft-Syslog",
            ]
        }

      + data_sources {
          + performance_counter {
              + counter_specifiers            = [
                  + "% Used Inodes",
                  + "% Used Space",
                  + "Disk Reads/sec",
                  + "Disk Transfers/sec",
                  + "Disk Writes/sec",
                  + "Free Megabytes",
                ]
              + name                          = "ms-performance"
              + sampling_frequency_in_seconds = 300
              + streams                       = [
                  + "Microsoft-Perf",
                ]
            }
          + syslog {
              + facility_names = [
                  + "auth",
                  + "cron",
                  + "daemon",
                  + "local0",
                  + "local1",
                  + "local2",
                  + "local3",
                  + "local4",
                  + "local5",
                  + "local6",
                  + "local7",
                  + "user",
                ]
              + log_levels     = [
                  + "Emergency",
                  + "Alert",
                  + "Critical",
                  + "Error",
                ]
              + name           = "ms-syslog-error"
              + streams        = [
                  + "Microsoft-Syslog",
                ]
            }
          + syslog {
              + facility_names = [
                  + "syslog",
                ]
              + log_levels     = [
                  + "Emergency",
                  + "Alert",
                  + "Critical",
                  + "Error",
                  + "Warning",
                  + "Notice",
                  + "Info",
                ]
              + name           = "ms-syslog-info"
              + streams        = [
                  + "Microsoft-Syslog",
                ]
            }
        }

      + destinations {
          + log_analytics {
              + name                  = "hmcts-sandbox"
              + workspace_resource_id = "/subscriptions/bf308a5c-0624-4334-8ff8-8dca9fd43783/resourceGroups/oms-automation/providers/Microsoft.OperationalInsights/workspaces/hmcts-sandbox"
            }
        }
    }

  # azurerm_monitor_data_collection_rule.windows_data_collection_rule will be created
  + resource "azurerm_monitor_data_collection_rule" "windows_data_collection_rule" {
      + description         = "A data collection rule for collecting Windows event logs and sending them to the appropriate log analytics workspace."
      + id                  = (known after apply)
      + immutable_id        = (known after apply)
      + kind                = "Windows"
      + location            = "uksouth"
      + name                = "ama-windows-vm-logs"
      + resource_group_name = "oms-automation"

      + data_flow {
          + destinations = [
              + "hmcts-sandbox",
            ]
          + streams      = [
              + "Microsoft-WindowsEvent",
            ]
        }

      + data_sources {
          + windows_event_log {
              + name           = "ms-windows-event-log"
              + streams        = [
                  + "Microsoft-WindowsEvent",
                ]
              + x_path_queries = [
                  + "Application!*[System[(Level = 1 or Level = 2 or Level = 3)]]",
                  + "System!*[System[(Level = 1 or Level = 2 or Level = 3)]]",
                ]
            }
          + windows_event_log {
              + name           = "ms-event-log"
              + streams        = [
                  + "Microsoft-Event",
                ]
              + x_path_queries = [
                  + "Application!*[System[(Level = 1 or Level = 2 or Level = 3)]]",
                  + "System!*[System[(Level = 1 or Level = 2 or Level = 3)]]",
                ]
            }
          + windows_event_log {
              + name           = "ms-security-event-log"
              + streams        = [
                  + "Microsoft-SecurityEvent",
                ]
              + x_path_queries = [
                  + "Application!*[System[(Level = 1 or Level = 2 or Level = 3)]]",
                  + "System!*[System[(Level = 1 or Level = 2 or Level = 3)]]",
                ]
            }
        }

      + destinations {
          + log_analytics {
              + name                  = "hmcts-sandbox"
              + workspace_resource_id = "/subscriptions/bf308a5c-0624-4334-8ff8-8dca9fd43783/resourceGroups/oms-automation/providers/Microsoft.OperationalInsights/workspaces/hmcts-sandbox"
            }
        }
    }

Plan: 2 to add, 0 to change, 0 to destroy.

@hmcts-platform-operations

Plan Result (nonprod)

Plan: 2 to add, 0 to change, 0 to destroy.
  • Create
    • azurerm_monitor_data_collection_rule.linux_data_collection_rule
    • azurerm_monitor_data_collection_rule.windows_data_collection_rule
Change Result (Click me)
  # azurerm_monitor_data_collection_rule.linux_data_collection_rule will be created
  + resource "azurerm_monitor_data_collection_rule" "linux_data_collection_rule" {
      + description         = "A data collection rule for collecting Linux syslog and performance counters and sending them to the appropriate log analytics workspace."
      + id                  = (known after apply)
      + immutable_id        = (known after apply)
      + kind                = "Linux"
      + location            = "uksouth"
      + name                = "ama-linux-vm-logs"
      + resource_group_name = "oms-automation"

      + data_flow {
          + destinations = [
              + "hmcts-nonprod",
            ]
          + streams      = [
              + "Microsoft-Perf",
            ]
        }
      + data_flow {
          + destinations = [
              + "hmcts-nonprod",
            ]
          + streams      = [
              + "Microsoft-Syslog",
            ]
        }

      + data_sources {
          + performance_counter {
              + counter_specifiers            = [
                  + "% Used Inodes",
                  + "% Used Space",
                  + "Disk Reads/sec",
                  + "Disk Transfers/sec",
                  + "Disk Writes/sec",
                  + "Free Megabytes",
                ]
              + name                          = "ms-performance"
              + sampling_frequency_in_seconds = 300
              + streams                       = [
                  + "Microsoft-Perf",
                ]
            }
          + syslog {
              + facility_names = [
                  + "auth",
                  + "cron",
                  + "daemon",
                  + "local0",
                  + "local1",
                  + "local2",
                  + "local3",
                  + "local4",
                  + "local5",
                  + "local6",
                  + "local7",
                  + "user",
                ]
              + log_levels     = [
                  + "Emergency",
                  + "Alert",
                  + "Critical",
                  + "Error",
                ]
              + name           = "ms-syslog-error"
              + streams        = [
                  + "Microsoft-Syslog",
                ]
            }
          + syslog {
              + facility_names = [
                  + "syslog",
                ]
              + log_levels     = [
                  + "Emergency",
                  + "Alert",
                  + "Critical",
                  + "Error",
                  + "Warning",
                  + "Notice",
                  + "Info",
                ]
              + name           = "ms-syslog-info"
              + streams        = [
                  + "Microsoft-Syslog",
                ]
            }
        }

      + destinations {
          + log_analytics {
              + name                  = "hmcts-nonprod"
              + workspace_resource_id = "/subscriptions/1c4f0704-a29e-403d-b719-b90c34ef14c9/resourceGroups/oms-automation/providers/Microsoft.OperationalInsights/workspaces/hmcts-nonprod"
            }
        }
    }

  # azurerm_monitor_data_collection_rule.windows_data_collection_rule will be created
  + resource "azurerm_monitor_data_collection_rule" "windows_data_collection_rule" {
      + description         = "A data collection rule for collecting Windows event logs and sending them to the appropriate log analytics workspace."
      + id                  = (known after apply)
      + immutable_id        = (known after apply)
      + kind                = "Windows"
      + location            = "uksouth"
      + name                = "ama-windows-vm-logs"
      + resource_group_name = "oms-automation"

      + data_flow {
          + destinations = [
              + "hmcts-nonprod",
            ]
          + streams      = [
              + "Microsoft-WindowsEvent",
            ]
        }

      + data_sources {
          + windows_event_log {
              + name           = "ms-windows-event-log"
              + streams        = [
                  + "Microsoft-WindowsEvent",
                ]
              + x_path_queries = [
                  + "Application!*[System[(Level = 1 or Level = 2 or Level = 3)]]",
                  + "System!*[System[(Level = 1 or Level = 2 or Level = 3)]]",
                ]
            }
          + windows_event_log {
              + name           = "ms-event-log"
              + streams        = [
                  + "Microsoft-Event",
                ]
              + x_path_queries = [
                  + "Application!*[System[(Level = 1 or Level = 2 or Level = 3)]]",
                  + "System!*[System[(Level = 1 or Level = 2 or Level = 3)]]",
                ]
            }
          + windows_event_log {
              + name           = "ms-security-event-log"
              + streams        = [
                  + "Microsoft-SecurityEvent",
                ]
              + x_path_queries = [
                  + "Application!*[System[(Level = 1 or Level = 2 or Level = 3)]]",
                  + "System!*[System[(Level = 1 or Level = 2 or Level = 3)]]",
                ]
            }
        }

      + destinations {
          + log_analytics {
              + name                  = "hmcts-nonprod"
              + workspace_resource_id = "/subscriptions/1c4f0704-a29e-403d-b719-b90c34ef14c9/resourceGroups/oms-automation/providers/Microsoft.OperationalInsights/workspaces/hmcts-nonprod"
            }
        }
    }

Plan: 2 to add, 0 to change, 0 to destroy.

@hmcts-platform-operations

Plan Result (prod)

Plan: 2 to add, 0 to change, 0 to destroy.
  • Create
    • azurerm_monitor_data_collection_rule.linux_data_collection_rule
    • azurerm_monitor_data_collection_rule.windows_data_collection_rule
Change Result (Click me)
  # azurerm_monitor_data_collection_rule.linux_data_collection_rule will be created
  + resource "azurerm_monitor_data_collection_rule" "linux_data_collection_rule" {
      + description         = "A data collection rule for collecting Linux syslog and performance counters and sending them to the appropriate log analytics workspace."
      + id                  = (known after apply)
      + immutable_id        = (known after apply)
      + kind                = "Linux"
      + location            = "uksouth"
      + name                = "ama-linux-vm-logs"
      + resource_group_name = "oms-automation"

      + data_flow {
          + destinations = [
              + "hmcts-prod",
            ]
          + streams      = [
              + "Microsoft-Perf",
            ]
        }
      + data_flow {
          + destinations = [
              + "hmcts-prod",
            ]
          + streams      = [
              + "Microsoft-Syslog",
            ]
        }

      + data_sources {
          + performance_counter {
              + counter_specifiers            = [
                  + "% Used Inodes",
                  + "% Used Space",
                  + "Disk Reads/sec",
                  + "Disk Transfers/sec",
                  + "Disk Writes/sec",
                  + "Free Megabytes",
                ]
              + name                          = "ms-performance"
              + sampling_frequency_in_seconds = 300
              + streams                       = [
                  + "Microsoft-Perf",
                ]
            }
          + syslog {
              + facility_names = [
                  + "auth",
                  + "cron",
                  + "daemon",
                  + "local0",
                  + "local1",
                  + "local2",
                  + "local3",
                  + "local4",
                  + "local5",
                  + "local6",
                  + "local7",
                  + "user",
                ]
              + log_levels     = [
                  + "Emergency",
                  + "Alert",
                  + "Critical",
                  + "Error",
                ]
              + name           = "ms-syslog-error"
              + streams        = [
                  + "Microsoft-Syslog",
                ]
            }
          + syslog {
              + facility_names = [
                  + "syslog",
                ]
              + log_levels     = [
                  + "Emergency",
                  + "Alert",
                  + "Critical",
                  + "Error",
                  + "Warning",
                  + "Notice",
                  + "Info",
                ]
              + name           = "ms-syslog-info"
              + streams        = [
                  + "Microsoft-Syslog",
                ]
            }
        }

      + destinations {
          + log_analytics {
              + name                  = "hmcts-prod"
              + workspace_resource_id = "/subscriptions/8999dec3-0104-4a27-94ee-6588559729d1/resourceGroups/oms-automation/providers/Microsoft.OperationalInsights/workspaces/hmcts-prod"
            }
        }
    }

  # azurerm_monitor_data_collection_rule.windows_data_collection_rule will be created
  + resource "azurerm_monitor_data_collection_rule" "windows_data_collection_rule" {
      + description         = "A data collection rule for collecting Windows event logs and sending them to the appropriate log analytics workspace."
      + id                  = (known after apply)
      + immutable_id        = (known after apply)
      + kind                = "Windows"
      + location            = "uksouth"
      + name                = "ama-windows-vm-logs"
      + resource_group_name = "oms-automation"

      + data_flow {
          + destinations = [
              + "hmcts-prod",
            ]
          + streams      = [
              + "Microsoft-WindowsEvent",
            ]
        }

      + data_sources {
          + windows_event_log {
              + name           = "ms-windows-event-log"
              + streams        = [
                  + "Microsoft-WindowsEvent",
                ]
              + x_path_queries = [
                  + "Application!*[System[(Level = 1 or Level = 2 or Level = 3)]]",
                  + "System!*[System[(Level = 1 or Level = 2 or Level = 3)]]",
                ]
            }
          + windows_event_log {
              + name           = "ms-event-log"
              + streams        = [
                  + "Microsoft-Event",
                ]
              + x_path_queries = [
                  + "Application!*[System[(Level = 1 or Level = 2 or Level = 3)]]",
                  + "System!*[System[(Level = 1 or Level = 2 or Level = 3)]]",
                ]
            }
          + windows_event_log {
              + name           = "ms-security-event-log"
              + streams        = [
                  + "Microsoft-SecurityEvent",
                ]
              + x_path_queries = [
                  + "Application!*[System[(Level = 1 or Level = 2 or Level = 3)]]",
                  + "System!*[System[(Level = 1 or Level = 2 or Level = 3)]]",
                ]
            }
        }

      + destinations {
          + log_analytics {
              + name                  = "hmcts-prod"
              + workspace_resource_id = "/subscriptions/8999dec3-0104-4a27-94ee-6588559729d1/resourceGroups/oms-automation/providers/Microsoft.OperationalInsights/workspaces/hmcts-prod"
            }
        }
    }

Plan: 2 to add, 0 to change, 0 to destroy.

@ieuanb74 ieuanb74 marked this pull request as ready for review September 4, 2024 10:56
@ieuanb74 ieuanb74 merged commit f97dc03 into master Sep 4, 2024
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants