Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

FOPTS-5124 Added batch processing for some Policies with high memory consumption #2722

Open
wants to merge 5 commits into
base: master
Choose a base branch
from
Open

FOPTS-5124 Added batch processing for some Policies with high memory consumption #2722

wants to merge 5 commits into from

Conversation

JesusParada
Copy link
Contributor

Added batch processing in AWS Object Storage Optimization, Azure Databricks Rightsize Compute Instances and Azure Reserved Instance Recommendations Policies

Description

Following the release of batch processing for all organizations, it was decided to modify the templates with the highest memory consumption to make use of batch processing. These policies are:

  • AWS Object Storage Optimization
  • Azure Databricks Rightsize Compute Instances
  • Azure Reserved Instance Recommendations

NOTE: For AWS Object Storage Optimization, I think there was a mistake in the parameters to run script
$js_aws_s3_objects_with_storage_class. It was using $ds_aws_list_s3_objects but I think it should use, $ds_aws_s3_objects because it is the data source that has the objects filtered by tags if we use that param, and $ds_aws_s3_objects was not being used by any other datasource. I changed the data source used as input, but let me know if it was correct

Issues Resolved

FOPTS-5124

Link to Example Applied Policy

Contribution Check List

  • New functionality includes testing.
  • New functionality has been documented in the README if applicable
  • New functionality has been documented in CHANGELOG.MD

@JesusParada JesusParada requested a review from a team as a code owner October 4, 2024 20:08
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Oct 4, 2024 •
edited
Loading

2 Warnings
⚠️

cost/azure/databricks/rightsize_compute/azure_databricks_rightsize_compute.pt

Policy template updated but associated README.md file has not been. Please verify that any necessary changes have been made to the README.

Recommendation policy has export that is missing recommended fields. These fields are scraped by the Flexera platform for dashboards:

Line 1198: resourceName, tags
Line 1370: resourceName, tags

Possible invalid spacing between comma-separated items found:

Line 602: "aggregation": "minimum,average,maximum",

Line 605: "metricnames": "Percentage CPU,Available Memory Bytes",

Comma separated items should be organized as follows, with a single space following each comma: apple, banana, pear
⚠️

cost/azure/reserved_instances/recommendations/azure_reserved_instance_recommendations.pt

Policy template updated but associated README.md file has not been. Please verify that any necessary changes have been made to the README.

Generated by 🚫 Danger

@JesusParada JesusParada added enhancement New feature or request READY-FOR-REVIEW labels Oct 7, 2024
@JesusParada JesusParada force-pushed the FOPTS-5124 branch 2 times, most recently from 753973a to f9a4886 Compare October 7, 2024 21:23
@nia-vf1 nia-vf1 self-assigned this Oct 9, 2024
nia-vf1
nia-vf1 reviewed Oct 14, 2024
View reviewed changes
cost/azure/databricks/rightsize_compute/azure_databricks_rightsize_compute.pt
@@ -1026,7 +1054,7 @@ script "js_idle_and_underutil_instances", type: "javascript" do

applied_policy_name = ds_applied_policy["name"]

_.each(ds_filtered_workspace_virtualmachines_with_costs, function(vm) {
getIterator(ds_filtered_workspace_virtualmachines_with_costs).Each(function(vm, idx) {
// Set the Applied Policy Name which is used in the summary_template/detail_template output
vm["policy_name"] = applied_policy_name
// Get the utilization metrics that we are going to use for comparison
Copy link
Contributor

@nia-vf1 nia-vf1 Oct 14, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Line 1193 and Line 1354 -

Not caused by this change but see Errors -

Recommendation policy has export that is missing required fields. These fields are scraped by the Flexera platform for dashboards:

Line 1193: accountID, accountName, resourceID, service, savingsCurrency
Line 1354: accountID, accountName, resourceID, service, savingsCurrency

e.g., line 1201 should be accountID instead of subscriptionId (and same for subscriptionName); resourceID, service and savingsCurrency fields need to be added

@nia-vf1 nia-vf1 assigned JesusParada and unassigned JesusParada and nia-vf1 Oct 14, 2024
@JesusParada JesusParada force-pushed the FOPTS-5124 branch 2 times, most recently from d82742c to 05c60e7 Compare October 16, 2024 19:29
@nia-vf1
Copy link
Contributor

nia-vf1 commented Oct 31, 2024

Just one outstanding change requested, then happy to approve

@JesusParada JesusParada force-pushed the FOPTS-5124 branch 2 times, most recently from 10f39fe to 6dfc641 Compare November 12, 2024 20:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@nia-vf1 nia-vf1 nia-vf1 left review comments

@bryankaraffa bryankaraffa Awaiting requested review from bryankaraffa

@XOmniverse XOmniverse Awaiting requested review from XOmniverse

At least 1 approving review is required to merge this pull request.

Assignees

@JesusParada JesusParada

Labels
enhancement New feature or request READY-FOR-REVIEW
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@JesusParada @nia-vf1