Some records fail to export in long-running exports

[barbeau]

Describe the bug
If I try to export data for all users unfiltered, while the overall export is successful there are occasional interruptions with Firebase that result in this error being occasionally printed to the log:

java.util.concurrent.ExecutionException: com.google.api.gax.rpc.UnavailableException: io.grpc.StatusRuntimeException: UNAVAILABLE: The datastore opera
tion timed out, or the data was temporarily unavailable.
        at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:500)
        at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:479)
        at com.google.api.core.AbstractApiFuture.get(AbstractApiFuture.java:56)
        at edu.usf.cutr.tba.utils.FirebaseIOUtils.getQueryDocumentSnapshots(FirebaseIOUtils.java:72)
        at edu.usf.cutr.tba.utils.FirebaseIOUtils.getAllRecordIdsByUserIdAndFolder(FirebaseIOUtils.java:48)
        at edu.usf.cutr.tba.io.FirebaseReader.getAllUserInfoById(FirebaseReader.java:70)
        at edu.usf.cutr.tba.manager.TravelBehaviorDataAnalysisManager.processUserById(TravelBehaviorDataAnalysisManager.java:141)
        at edu.usf.cutr.tba.manager.TravelBehaviorDataAnalysisManager.analyzeAllTravelBehaviorData(TravelBehaviorDataAnalysisManager.java:118)
        at edu.usf.cutr.tba.manager.TravelBehaviorDataAnalysisManager.processData(TravelBehaviorDataAnalysisManager.java:84)
        at edu.usf.cutr.tba.ProcessorMain.main(ProcessorMain.java:142)
Caused by: com.google.api.gax.rpc.UnavailableException: io.grpc.StatusRuntimeException: UNAVAILABLE: The datastore operation timed out, or the data wa
s temporarily unavailable.
        at com.google.api.gax.rpc.ApiExceptionFactory.createException(ApiExceptionFactory.java:69)
        at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:72)
        at com.google.api.gax.grpc.GrpcApiExceptionFactory.create(GrpcApiExceptionFactory.java:60)
        at com.google.api.gax.grpc.ExceptionResponseObserver.onErrorImpl(ExceptionResponseObserver.java:82)
        at com.google.api.gax.rpc.StateCheckingResponseObserver.onError(StateCheckingResponseObserver.java:86)
        at com.google.api.gax.grpc.GrpcDirectStreamController$ResponseObserverAdapter.onClose(GrpcDirectStreamController.java:149)
        at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
        at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
        at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
        at io.grpc.internal.CensusStatsModule$StatsClientInterceptor$1$1.onClose(CensusStatsModule.java:684)
        at io.grpc.PartialForwardingClientCallListener.onClose(PartialForwardingClientCallListener.java:39)
        at io.grpc.ForwardingClientCallListener.onClose(ForwardingClientCallListener.java:23)
        at io.grpc.ForwardingClientCallListener$SimpleForwardingClientCallListener.onClose(ForwardingClientCallListener.java:40)
        at io.grpc.internal.CensusTracingModule$TracingClientInterceptor$1$1.onClose(CensusTracingModule.java:403)
        at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:459)
        at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:63)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.close(ClientCallImpl.java:546)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl.access$600(ClientCallImpl.java:467)
        at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:584)
        at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: io.grpc.StatusRuntimeException: UNAVAILABLE: The datastore operation timed out, or the data was temporarily unavailable.
        at io.grpc.Status.asRuntimeException(Status.java:526)
        ... 22 more

I believe the currently behavior would just be to skip this record and output the next one. It would be nice if we could retry that record so we weren't missing any data in exports due to occasional connection failures.

It looks like in the last export of all data it happened 59 times, out of records for ~17,700 users (which would mean millions of records). So not a big problem, but something we should eventually fix.

To Reproduce
Steps to reproduce the behavior:

1. Run an export of a huge amount of user data

Expected behavior
Occasional interruptions in connection happen during export resulting in error message

Actual behavior
Upon these exceptions, we should retry exporting the same record again.

Desktop (please complete the following information):

• Windows 7 Enterprise w/ JDK 11