[aws-eks] Metrics Query works but Metrics Search does not

[sarasensible]

I am getting no data across the aws-eks dashboard. I have Container Insights installed on my cluster and I can see them in the AWS console. I cloned the EKS dashboard and fiddled around and found two issues - one is that if I do an Explore on cluster_failed_node_count with the MatchExact option toggled to false I can see data, the second is that if I change the cluster_failed_node_count panel to use a "Metrics Query" instead of a "Metrics Search" I can see data. I don't understand why the dashboard isn't working as is. Any guidance would be greatly appreciated.

Metrics Search vs Metrics Query

Match exact toggled off

Match exact toggled on

"panels": [
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 7,
        "w": 6,
        "x": 0,
        "y": 0
      },
      "hiddenSeries": false,
      "id": 2,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": true,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "nullPointMode": "null",
      "options": {
        "alertThreshold": true
      },
      "percentage": false,
      "pluginVersion": "8.4.2",
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "alias": "",
          "dimensions": {
            "ClusterName": "$clustername"
          },
          "expression": "",
          "id": "",
          "matchExact": false,
          "metricEditorMode": 0,
          "metricName": "cluster_failed_node_count",
          "metricQueryType": 0,
          "namespace": "ContainerInsights",
          "period": "",
          "refId": "A",
          "region": "default",
          "statistic": "Average"
        }
      ],
      "thresholds": [],
      "timeRegions": [],
      "title": "cluster_failed_node_count",
      "tooltip": {
        "shared": true,
        "sort": 0,
        "value_type": "individual"
      },
      "type": "graph",
      "xaxis": {
        "mode": "time",
        "show": true,
        "values": []
      },
      "yaxes": [
        {
          "$$hashKey": "object:73",
          "format": "short",
          "logBase": 1,
          "show": true
        },
        {
          "$$hashKey": "object:74",
          "format": "short",
          "logBase": 1,
          "show": true
        }
      ],
      "yaxis": {
        "align": false
      }
    },
    {
      "aliasColors": {},
      "bars": false,
      "dashLength": 10,
      "dashes": false,
      "datasource": {
        "type": "cloudwatch",
        "uid": "${datasource}"
      },
      "fill": 1,
      "fillGradient": 0,
      "gridPos": {
        "h": 7,
        "w": 6,
        "x": 6,
        "y": 0
      },
      "hiddenSeries": false,
      "id": 43,
      "legend": {
        "avg": false,
        "current": false,
        "max": false,
        "min": false,
        "show": false,
        "total": false,
        "values": false
      },
      "lines": true,
      "linewidth": 1,
      "nullPointMode": "null",
      "options": {
        "alertThreshold": true
      },
      "percentage": false,
      "pluginVersion": "8.4.2",
      "pointradius": 2,
      "points": false,
      "renderer": "flot",
      "seriesOverrides": [],
      "spaceLength": 10,
      "stack": false,
      "steppedLine": false,
      "targets": [
        {
          "alias": "",
          "datasource": {
            "type": "cloudwatch",
            "uid": "${datasource}"
          },
          "dimensions": {},
          "expression": "",
          "id": "",
          "matchExact": true,
          "metricEditorMode": 0,
          "metricName": "",
          "metricQueryType": 1,
          "namespace": "ContainerInsights",
          "period": "",
          "queryMode": "Metrics",
          "refId": "A",
          "region": "default",
          "sql": {
            "from": {
              "property": {
                "name": "ContainerInsights",
                "type": "string"
              },
              "type": "property"
            },
            "select": {
              "name": "AVG",
              "parameters": [
                {
                  "name": "cluster_failed_node_count",
                  "type": "functionParameter"
                }
              ],
              "type": "function"
            }
          },
          "sqlExpression": "SELECT AVG(cluster_failed_node_count) FROM ContainerInsights",
          "statistic": "Average"
        }
      ]

[tomsozolins]

Similar problem, added cloudwatch agent and iam role, but no data for all panels. EKS 1.19.

[jangaraj]

From Grafana doc:

If Match Exact is enabled, you also need to specify all the dimensions of the metric you’re querying, so that the metric schema matches exactly.

From AWS doc is clear that metrics, e.g. cluster_failed_node_count have dimensions. So dimensions must be specified in the query, when Match Exact is enabled.

[sarasensible]

Thanks for the feedback @jangaraj . What is the actual resolution for using the dashboard that you propose in this case? I can either disable Match Exact or specify some dimensions. I'm not sure how to do either since when I go to edit one of the panels the query does not populate in the search bar. Additionally I see no option for "Match Exact". Is it required to paste the query from the raw JSON into the panel and then edit each one to have some dimension? I prefer my own solution of changing them to Metric Query in that case.

I have created a gist with the dashboard using Metric Queries - https://gist.github.com/sarasensible/a775378a2d3786c61a706b2496772e75 . @tomsozolins you are welcome to try it and see if it works any better.

[jangaraj]

@sarasensible Thank you. Actually, I never used this dashboard - it was a community contribution. Feel free to create pull request if you can improve it.

[sarasensible]

My revised dashboard works for me but I would like some confirmation that it works for someone else before putting up a PR. @tomsozolins would you be willing to give it a try and post here whether my gist works for you out of the box?