optimize execution of workflow consisting of bucket-level followed by doc-level monitors #1729
Conversation
… doc-level monitors Signed-off-by: Subhobrata Dey <[email protected]>
sbcd90
requested review from
lezzago,
AWSHurneyt,
eirsep,
getsaurabh02,
praveensameneni,
bowenlan-amzn,
rishabhmaurya,
engechas,
riysaxen-amzn,
jowg-amazon,
amsiglan and
goyamegh
as code owners
|
plz update description with what the optimization is and what is the change in the PR? |
| @@ -349,6 +363,50 @@ class TransportDocLevelMonitorFanOutAction | |||
| } | |||
| } | |||
|
|
|||
| private suspend fun runForEachDocTriggerIgnoringFindingsAndAlerts( | |||
Signed-off-by: Subhobrata Dey <[email protected]>
| @@ -479,7 +480,7 @@ object BucketLevelMonitorRunner : MonitorRunner() { | |||
| val queryBuilder = if (input.query.query() == null) BoolQueryBuilder() | |||
| else QueryBuilders.boolQuery().must(source.query()) | |||
| queryBuilder.filter(QueryBuilders.termsQuery(fieldName, bucketValues)) | |||
| sr.source().query(queryBuilder) | |||
| sr.source().query(queryBuilder).sort("_seq_no", SortOrder.DESC) | |||
There was a problem hiding this comment.
why are we sorting based on _seq_no?
there is already a range query based on period_end variable
this seems incorrect
There was a problem hiding this comment.
we need to sort this because without sort, we get random 10 docs in period of last 15 minutes by default for a workflow running every 1 min. Now, the aggregation may have grouped 1000 docs.
So, 10 out of 1000 docs generated may not be the latest ones. We pass these 10 docs to the delegated doc-level monitor which has already moved its seq_no past these 10 random docs and hence do not generate an alert.
sort by seq_no ensures we always get the latest 10 docs out of the 1000 docs considered for aggregation. Thus, when the doc-level monitor runs next time, it gets latest 10 docs and it goes on to geenrate an alert.
Description
The PR proposes following changes to optimize execution of workflow consisting of bucket-level followed by match-all doc-level monitors.
Based on a flag
monitor.ignoreFindingsAndAlertsthe doc-level monitor is able to ignore storage of findings and alerts and triggering subsequentpublishFindingcalls. https://github.com/opensearch-project/alerting/pull/1729/files#diff-64dadab7578092d0871a6d87833637fcd3c3e56dae448222ec57476921c9707eR300Based on a flag
monitor.ignoreFindingsAndAlertsthe doc-level monitor is able to just match the trigger condition and generate a single alert and subsequently trigger notification. https://github.com/opensearch-project/alerting/pull/1729/files#diff-64dadab7578092d0871a6d87833637fcd3c3e56dae448222ec57476921c9707eR366This code considers all documents even if
indexExecutionContext.docIdsis empty.https://github.com/opensearch-project/alerting/pull/1729/files#diff-64dadab7578092d0871a6d87833637fcd3c3e56dae448222ec57476921c9707eR948
This pr addresses this issue for workflows where bucket-level monitor sends an empty
indexExecutionContext.docIdslist.10 documentsas part ofindexExecutionContext.docIds. This results in inconsistent alert generation and triggering of notifications. https://github.com/opensearch-project/alerting/pull/1729/files#diff-756df2dfd50dd9cc484313c8d6db1a052ab18e7fc962829ad56e982c0e6a5c56R483This pr addresses this issue by sorting the sequence numbers in descending order.
Related Issues
Resolves #[Issue number to be closed when this PR is merged]
Check List
--signoff.By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.