Implement nec.com/ve-seconds metric

[wmeddie]

Very useful when trying to see if the VE is being used properly.

[saudet]

https://nec.com/ve-seconds doesn't exist, what's the correct URL for this?

[saudet]

From what I understand, here is where those seconds get computed, including NECVEPlugin as one of the "resources":

  private AggregateAppResourceUsage getRunningAggregateAppResourceUsage() {
    long currentTimeMillis = System.currentTimeMillis();
    // Don't walk the whole container list if the resources were computed
    // recently.
    if ((currentTimeMillis - lastMemoryAggregateAllocationUpdateTime)
        > MEM_AGGREGATE_ALLOCATION_CACHE_MSECS) {
      Map<String, Long> resourceSecondsMap = new HashMap<>();
      for (RMContainer rmContainer : this.liveContainers.values()) {
        long usedMillis = currentTimeMillis - rmContainer.getCreationTime();
        Resource resource = rmContainer.getContainer().getResource();
        for (ResourceInformation entry : resource.getResources()) {
          long value = RMServerUtils
              .getOrDefault(resourceSecondsMap, entry.getName(), 0L);
          value += entry.getValue() * usedMillis
              / DateUtils.MILLIS_PER_SECOND;
          resourceSecondsMap.put(entry.getName(), value);
        }
      }

https://github.com/apache/hadoop/blob/rel/release-3.3.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java#L1097-L1114

And then eventually down the pipeline the strings for the UI get put together here:

  public static String getResourceSecondsString(Map<String, Long> targetMap) {
    List<String> strings = new ArrayList<>(targetMap.size());
    //completed app report in the timeline server doesn't have usage report
    Long memorySeconds = 0L;
    Long vcoreSeconds = 0L;
    if (targetMap.containsKey(ResourceInformation.MEMORY_MB.getName())) {
      memorySeconds = targetMap.get(ResourceInformation.MEMORY_MB.getName());
    }
    if (targetMap.containsKey(ResourceInformation.VCORES.getName())) {
      vcoreSeconds = targetMap.get(ResourceInformation.VCORES.getName());
    }
    strings.add(memorySeconds + " MB-seconds");
    strings.add(vcoreSeconds + " vcore-seconds");
    Map<String, ResourceInformation> tmp = ResourceUtils.getResourceTypes();
    if (targetMap.size() > 2) {
      for (Map.Entry<String, Long> entry : targetMap.entrySet()) {
        if (!entry.getKey().equals(ResourceInformation.MEMORY_MB.getName())
            && !entry.getKey().equals(ResourceInformation.VCORES.getName())) {
          String units = "";
          if (tmp.containsKey(entry.getKey())) {
            units = tmp.get(entry.getKey()).getUnits();
          }
          strings.add(entry.getValue() + " " + entry.getKey() + "-" + units
              + "seconds");
        }
      }
    }
    return String.join(", ", strings);
  }

https://github.com/apache/hadoop/blob/rel/release-3.3.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/StringHelper.java#L184-L212

So it looks like it's merely reporting how much time those resources are allocated. It doesn't seem possible to set arbitrary values there.