[Crowdstrike] Add ability to filter api calls

[btrieger]

Currently the crowdstrike integration does not allow the user to specify a filter to apply to the apis that are called. IE If the host api returns millions of hosts most of which are ephemeral it would be desirable to add a filter to only return hosts that have a certain tag applied. This will reduce the amount of data returned by the api and sent to elasticsearch.

In the cel.yml.hbs we can see the below:

        ?"filter": filter.hasValue() ?
          optional.of(['modified_timestamp:>"'+filter.value()+'"'])

It would be beneficial to append to the modified timestamp filter additional filters with an input provided by the user.

[elasticmachine]

Pinging @elastic/security-service-integrations (Team:Security-Service Integrations)

[efd6]

@btrieger I have this as a minimal approach (modelled in alert):

diff --git a/packages/crowdstrike/data_stream/alert/agent/stream/cel.yml.hbs b/packages/crowdstrike/data_stream/alert/agent/stream/cel.yml.hbs
index fe2d51ce00..57ffbd12a2 100644
--- a/packages/crowdstrike/data_stream/alert/agent/stream/cel.yml.hbs
+++ b/packages/crowdstrike/data_stream/alert/agent/stream/cel.yml.hbs
@@ -23,6 +23,9 @@ state:
   want_more: false
   offset: 0
   batch_size: {{batch_size}}
+{{#if query}}
+  query: {{query}}
+{{/if}}
 redact:
   fields: ~
 program: |
@@ -37,8 +40,13 @@ program: |
         "sort": ["timestamp|asc"],
         "offset": [string(state.offset)],
         "limit": [string(state.batch_size)],
-        ?"filter": filter.hasValue() ?
-          optional.of(['timestamp:>"'+filter.value()+'"'])
+        ?"filter": filter.hasValue() || state.?query.orValue("") != "" ?
+          optional.of([
+            [
+              ?filter.optMap(f, 'timestamp:>"'+f+'"'),
+              ?state.?query.optMap(q, "("+q+")"),
+            ].join("+")
+          ])
         :
           optional.none(),
       }.format_query()
diff --git a/packages/crowdstrike/data_stream/alert/manifest.yml b/packages/crowdstrike/data_stream/alert/manifest.yml
index 1d6866d48b..1015e1e306 100644
--- a/packages/crowdstrike/data_stream/alert/manifest.yml
+++ b/packages/crowdstrike/data_stream/alert/manifest.yml
@@ -39,6 +39,13 @@ streams:
         required: true
         show_user: false
         default: 30s
+      - name: query
+        type: text
+        title: FQL Query
+        description: This is an additional FQL query that may be included in requests to the API. You should not include any reference to the timestamp. See the [FalconPy documentation](https://www.falconpy.io/Usage/Falcon-Query-Language.html) for details.
+        multi: false
+        required: false
+        show_user: false
       - name: enable_request_tracer
         type: bool
         title: Enable request tracing

We cannot validate the query the user adds, so I am concerned about the possible confusion that it may bring. The best we can do is instruct and make sure any valid query that they provide is treated as a conjunction with the timestamp filtering.

[btrieger]

Working on getting the api ref for you. Apologies I was out sick yesterday. I understand the confusion that it could cause wish the api ref was public/had a way to validate it. Guess just add some info to the documentation on the new field explaining that it gets the timestamp appended to it. Will let you know what I am able to find on the api ref.

[btrieger]

Looks like the link you link to lines up pretty well with what is available. You will get a 400 if the filter is invalid with the message invalid filter expression supplied

[efd6]

Guess just add some info to the documentation on the new field explaining that it gets the timestamp appended to it.

I think this is covered by the "You should not include any reference to the timestamp." text. I find that a warning not to do something is often better than explaining how it works.