You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
After recording ~130 experiments, attempting to list them from the Python client fails with an oversized message exception. Adding a filter does not work around the issue.
Traceback when calling keepsake.experiments.list():
_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.RESOURCE_EXHAUSTED
details = "Received message larger than max (4237525 vs. 4194304)"
debug_error_string = "{"created":"@1645296119.443456300","description":"Received message larger than max (4237525 vs. 4194304)","file":"src/core/ext/filters/message_size/message_size_filter.cc","file_line":205,"grpc_status":8}"
During handling of the above exception, another exception occurred:
Exception Traceback (most recent call last)
/tmp/ipykernel_4297/1455666872.py in
----> 1 experiments = keepsake.experiments.list(lambda e: "autotar" in e.params["run_group"])
2 len(experiments)
~/auto-stop-tar/.venv/lib/python3.9/site-packages/keepsake/experiment.py in list(self, filter)
387 Return all experiments for a project, sorted by creation date.
388 """
--> 389 experiments = self.project._daemon().list_experiments()
390 result: ExperimentList = ExperimentList()
391 for exp in experiments:
~/auto-stop-tar/.venv/lib/python3.9/site-packages/keepsake/daemon.py in wrapped(*args, **kwargs)
38 if status_code:
39 raise handle_exception(status_code, details)
---> 40 raise Exception(details)
41
42 return wrapped
Exception: Received message larger than max (4237525 vs. 4194304)
The text was updated successfully, but these errors were encountered:
For anyone else encountering this, the work-around change that's in the PR can be done locally (at your own risk). Eventually it probably starts to run into the 15 second timeout, so that would need to be bumped/configurable. A longer-term solution might be implementing a paging protocol, or filtering that can be passed to the daemon like the CLI (but with more robust filtering).
After recording ~130 experiments, attempting to list them from the Python client fails with an oversized message exception. Adding a filter does not work around the issue.
Traceback when calling
keepsake.experiments.list()
:_InactiveRpcError Traceback (most recent call last)
~/auto-stop-tar/.venv/lib/python3.9/site-packages/keepsake/daemon.py in wrapped(*args, **kwargs)
31 try:
---> 32 return f(*args, **kwargs)
33 except grpc.RpcError as e:
~/auto-stop-tar/.venv/lib/python3.9/site-packages/keepsake/daemon.py in list_experiments(self)
208 def list_experiments(self) -> List[Experiment]:
--> 209 ret = self.stub.ListExperiments(pb.ListExperimentsRequest())
210 return pb_convert.experiments_from_pb(self.project, ret.experiments)
~/auto-stop-tar/.venv/lib/python3.9/site-packages/grpc/_channel.py in call(self, request, timeout, metadata, credentials, wait_for_ready, compression)
945 wait_for_ready, compression)
--> 946 return _end_unary_response_blocking(state, call, False, None)
947
~/auto-stop-tar/.venv/lib/python3.9/site-packages/grpc/_channel.py in _end_unary_response_blocking(state, call, with_call, deadline)
848 else:
--> 849 raise _InactiveRpcError(state)
850
_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.RESOURCE_EXHAUSTED
details = "Received message larger than max (4237525 vs. 4194304)"
debug_error_string = "{"created":"@1645296119.443456300","description":"Received message larger than max (4237525 vs. 4194304)","file":"src/core/ext/filters/message_size/message_size_filter.cc","file_line":205,"grpc_status":8}"
During handling of the above exception, another exception occurred:
Exception Traceback (most recent call last)
/tmp/ipykernel_4297/1455666872.py in
----> 1 experiments = keepsake.experiments.list(lambda e: "autotar" in e.params["run_group"])
2 len(experiments)
~/auto-stop-tar/.venv/lib/python3.9/site-packages/keepsake/experiment.py in list(self, filter)
387 Return all experiments for a project, sorted by creation date.
388 """
--> 389 experiments = self.project._daemon().list_experiments()
390 result: ExperimentList = ExperimentList()
391 for exp in experiments:
~/auto-stop-tar/.venv/lib/python3.9/site-packages/keepsake/daemon.py in wrapped(*args, **kwargs)
38 if status_code:
39 raise handle_exception(status_code, details)
---> 40 raise Exception(details)
41
42 return wrapped
Exception: Received message larger than max (4237525 vs. 4194304)
The text was updated successfully, but these errors were encountered: