[SOAR-17874] Mimecast #2897

ablakley-r7 · 2024-10-25T10:47:08Z

Proposed Changes

Description

Describe the proposed changes:

Add 429 error handling task
Now wait 5 minutes before resume
Update Snyk vulnerablity https://www.cve.org/CVERecord?id=CVE-2024-49767

PR Requirements

Developers, verify you have completed the following items by checking them off:

Testing

Unit Tests

Review our documentation on generating and writing plugin unit tests

Unit tests written for any new or updated code

In-Product Tests

If you are an InsightConnect customer or have access to an InsightConnect instance, the following in-product tests should be done:

Screenshot of job output with the plugin changes
Screenshot of the changed connection, actions, or triggers input within the InsightConnect workflow builder

Style

Review the style guide

For dependencies, pin OS package and Python package versions
For security, set least privileged account with USER nobody in the Dockerfile when possible
For size, use the slim SDK images when possible: rapid7/insightconnect-python-3-38-slim-plugin:{sdk-version-num} and rapid7/insightconnect-python-3-38-plugin:{sdk-version-num}
For error handling, use of PluginException and ConnectionTestException
For logging, use self.logger
For docs, use changelog style
For docs, validate markdown with insight-plugin validate which calls icon_validate to lint help.md

Functional Checklist

Work fully completed
Functional
- Any new actions/triggers include JSON test files in the tests/ directory created with insight-plugin samples
- Tests should all pass unless it's a negative test. Negative tests have a naming convention of tests/$action_bad.json
- Unsuccessful tests should fail by raising an exception causing the plugin to die and an object should be returned on successful test
- Add functioning test results to PR, sanitize any output if necessary
  - Single action/trigger insight-plugin run -T tests/example.json --debug --jq
  - All actions/triggers shortcut insight-plugin run -T all --debug --jq (use PR format at end)
- Add functioning run results to PR, sanitize any output if necessary
  - Single action/trigger insight-plugin run -R tests/example.json --debug --jq
  - All actions/triggers shortcut insight-plugin run --debug --jq (use PR format at end)

Assessment

You must validate your work to reviewers:

Run insight-plugin validate and make sure everything passes
Run the assessment tool: insight-plugin run -A. For single action validation: insight-plugin run tests/{file}.json -A
Copy (insight-plugin ... | pbcopy) and paste the output in a new post on this PR
Add required screenshots from the In-Product Tests section

nmccullagh-r7 · 2024-10-25T10:58:33Z

plugins/mimecast/komand_mimecast/tasks/monitor_siem_logs/task.py

+            if rate_limited > now:
+                log_msg += "Still within rate limiting period, skipping task execution..."
+                self.logger.info(log_msg)
+                return [], state, False, 200, None


The only thing about this is that the customer won't have any idea they're rate limited 🤔

unless we put it the task into error state, we currently have no way to display any task info back to idr

If we return a recoverable error message/status code, these should get bubbled in the 'warning' text colour and the integration is still scheduled for the next run

mIght be a PM question to confirm as we haven't done this before, but I agree with Nathanael it would be good to let the customer know they're being rate limited. If we return status code 429, and then the error with cause with something like 'Rate limiting enforced from Mimecast' would show on the UI but let the integration be scheduled on the 5 minute interval, and it could prevent tickets raised for why no data has been pulled?

The only thing is that if we put the task into an error state we won't store the fact that we're rate limited 🤔

I've updated this now. The state should include a new rate limit value.

joneill-r7

Current logic looks good, but I'm just wondering if we could improve it a slightly so we base the time on what Mimecast returns and surface this to the customer?

joneill-r7 · 2024-10-29T11:54:15Z

plugins/mimecast/komand_mimecast/tasks/monitor_siem_logs/task.py

+
+    def check_rate_limit_error(self, status_code: int, state: dict) -> None:
+        if status_code == 429:
+            new_run_time = time() + 300


On their docs it does look like they might return a header that contains when their rate limit is reset: https://developer.services.mimecast.com/api-overview - X-Rate-Limit-Reset . It might be better to base the time on this value if it exists?

joneill-r7 · 2024-10-29T11:56:26Z

plugins/mimecast/komand_mimecast/tasks/monitor_siem_logs/task.py

+            if rate_limited > now:
+                log_msg += "Still within rate limiting period, skipping task execution..."
+                self.logger.info(log_msg)
+                return [], state, False, 200, None


mIght be a PM question to confirm as we haven't done this before, but I agree with Nathanael it would be good to let the customer know they're being rate limited. If we return status code 429, and then the error with cause with something like 'Rate limiting enforced from Mimecast' would show on the UI but let the integration be scheduled on the 5 minute interval, and it could prevent tickets raised for why no data has been pulled?

* Add 401 exception handling * Fix it up * Rename response to new response

nmccullagh-r7 reviewed Oct 25, 2024

View reviewed changes

ablakley-r7 force-pushed the soar-17874_mimecast branch from 30b0cc2 to 196d039 Compare October 29, 2024 08:51

ablakley-r7 requested review from nmccullagh-r7, rbowden-r7 and joneill-r7 October 29, 2024 08:52

joneill-r7 reviewed Oct 29, 2024

View reviewed changes

rmurray-r7 force-pushed the develop branch 4 times, most recently from 86876a9 to 9cb0c4a Compare November 4, 2024 15:21

cmcnally-r7 and others added 4 commits November 4, 2024 15:46

Palo alto retry creds 401 (#2826)

71a515f

* Add 401 exception handling * Fix it up * Rename response to new response

Add 429 handler to task | update sdk | add unit test

d3d80c5

Update requirements werkzeug

c613bdd

Update rate limit error

87a4595

ablakley-r7 force-pushed the soar-17874_mimecast branch from 196d039 to 87a4595 Compare November 4, 2024 15:46

ablakley-r7 added 3 commits November 4, 2024 16:02

Updare bandit

955b4aa

Update error logging

64d9f87

Update task.py

6899131

ablakley-r7 requested a review from joneill-r7 November 4, 2024 16:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SOAR-17874] Mimecast #2897

[SOAR-17874] Mimecast #2897

ablakley-r7 commented Oct 25, 2024 •

edited

Loading

nmccullagh-r7 Oct 25, 2024

rbowden-r7 Oct 25, 2024

joneill-r7 Oct 28, 2024

joneill-r7 Oct 29, 2024

nmccullagh-r7 Nov 1, 2024

ablakley-r7 Nov 4, 2024

joneill-r7 left a comment

joneill-r7 Oct 29, 2024

joneill-r7 Oct 29, 2024

[SOAR-17874] Mimecast #2897

Are you sure you want to change the base?

[SOAR-17874] Mimecast #2897

Conversation

ablakley-r7 commented Oct 25, 2024 • edited Loading

Proposed Changes

Description

PR Requirements

Testing

Unit Tests

In-Product Tests

Style

Functional Checklist

Assessment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

joneill-r7 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ablakley-r7 commented Oct 25, 2024 •

edited

Loading