Operation failures corrupt moving average buckets

[werkt]

The average decorations in PutOperationStage are not resilient to the state of all sources of putOperation, particularly early in the pipeline when an error occurs. The following is observed continuously if, say, the disk is full and errors an operation out of fetchInputs:

2024-02-14 10:53:23.532 | Feb 14, 2024 3:53:23 PM build.buildfarm.worker.InputFetcher failOperation |  
-- | -- | --
  |   | 2024-02-14 10:53:23.532 | SEVERE: Cannot report failed operation shard/operations/074d0fcf-e77e-4806-9918-a60045fbaae1 |  
  |   | 2024-02-14 10:53:23.532 | java.lang.IllegalArgumentException: Duration is not valid. See proto definition for valid values. Seconds (-315956351598) must be in range [-315,576,000,000, +315,576,000,000]. Nanos (-237000000) must be in range [-999,999,999, +999,999,999]. Nanos must have the same sign as seconds |  
  |   | 2024-02-14 10:53:23.532 | at com.google.protobuf.util.Durations.checkValid(Durations.java:190) |  
  |   | 2024-02-14 10:53:23.532 | at com.google.protobuf.util.Durations.normalizedDuration(Durations.java:479) |  
  |   | 2024-02-14 10:53:23.532 | at com.google.protobuf.util.Durations.add(Durations.java:452) |  
  |   | 2024-02-14 10:53:23.532 | at build.buildfarm.worker.PutOperationStage$OperationStageDurations.addOperations(PutOperationStage.java:200) |  
  |   | 2024-02-14 10:53:23.532 | at build.buildfarm.worker.PutOperationStage$AverageTimeCostOfLastPeriod.addOperation(PutOperationStage.java:135) |  
  |   | 2024-02-14 10:53:23.532 | at build.buildfarm.worker.PutOperationStage.put(PutOperationStage.java:49) |  
  |   | 2024-02-14 10:53:23.532 | at build.buildfarm.worker.InputFetcher.failOperation(InputFetcher.java:334) |  
  |   | 2024-02-14 10:53:23.532 | at build.buildfarm.worker.InputFetcher.fetchPolled(InputFetcher.java:211) |  
  |   | 2024-02-14 10:53:23.532 | at build.buildfarm.worker.InputFetcher.runInterruptibly(InputFetcher.java:106) |  
  |   | 2024-02-14 10:53:23.532 | at build.buildfarm.worker.InputFetcher.run(InputFetcher.java:293) |  
  |   | 2024-02-14 10:53:23.532 | at java.lang.Thread.run(Thread.java:748)

@jacobmou Please have a look at this trace and figure out how we can avoid using invalid (likely 0) timestamps subtracted from the worker timestamp. I recommend only using Stopwatch outputs to determine durations, not time subtraction with possibly 0 endpoints.

[RituRajSingh878]

I'm also receiving the somewhat same issue. Any updates or workaround on this?

CC @werkt