Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Received message larger than max (4237525 vs. 4194304) - Python client cannot list experiments #849

Open
CharlesPDX opened this issue Feb 19, 2022 · 2 comments

Comments

@CharlesPDX
Copy link

After recording ~130 experiments, attempting to list them from the Python client fails with an oversized message exception. Adding a filter does not work around the issue.

Traceback when calling keepsake.experiments.list():

_InactiveRpcError Traceback (most recent call last)
~/auto-stop-tar/.venv/lib/python3.9/site-packages/keepsake/daemon.py in wrapped(*args, **kwargs)
31 try:
---> 32 return f(*args, **kwargs)
33 except grpc.RpcError as e:

~/auto-stop-tar/.venv/lib/python3.9/site-packages/keepsake/daemon.py in list_experiments(self)
208 def list_experiments(self) -> List[Experiment]:
--> 209 ret = self.stub.ListExperiments(pb.ListExperimentsRequest())
210 return pb_convert.experiments_from_pb(self.project, ret.experiments)

~/auto-stop-tar/.venv/lib/python3.9/site-packages/grpc/_channel.py in call(self, request, timeout, metadata, credentials, wait_for_ready, compression)
945 wait_for_ready, compression)
--> 946 return _end_unary_response_blocking(state, call, False, None)
947

~/auto-stop-tar/.venv/lib/python3.9/site-packages/grpc/_channel.py in _end_unary_response_blocking(state, call, with_call, deadline)
848 else:
--> 849 raise _InactiveRpcError(state)
850

_InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.RESOURCE_EXHAUSTED
details = "Received message larger than max (4237525 vs. 4194304)"
debug_error_string = "{"created":"@1645296119.443456300","description":"Received message larger than max (4237525 vs. 4194304)","file":"src/core/ext/filters/message_size/message_size_filter.cc","file_line":205,"grpc_status":8}"

During handling of the above exception, another exception occurred:

Exception Traceback (most recent call last)
/tmp/ipykernel_4297/1455666872.py in
----> 1 experiments = keepsake.experiments.list(lambda e: "autotar" in e.params["run_group"])
2 len(experiments)

~/auto-stop-tar/.venv/lib/python3.9/site-packages/keepsake/experiment.py in list(self, filter)
387 Return all experiments for a project, sorted by creation date.
388 """
--> 389 experiments = self.project._daemon().list_experiments()
390 result: ExperimentList = ExperimentList()
391 for exp in experiments:

~/auto-stop-tar/.venv/lib/python3.9/site-packages/keepsake/daemon.py in wrapped(*args, **kwargs)
38 if status_code:
39 raise handle_exception(status_code, details)
---> 40 raise Exception(details)
41
42 return wrapped

Exception: Received message larger than max (4237525 vs. 4194304)

@CharlesPDX
Copy link
Author

For anyone else encountering this, the work-around change that's in the PR can be done locally (at your own risk). Eventually it probably starts to run into the 15 second timeout, so that would need to be bumped/configurable. A longer-term solution might be implementing a paging protocol, or filtering that can be passed to the daemon like the CLI (but with more robust filtering).

@zeke
Copy link
Member

zeke commented Mar 15, 2022

Hi @CharlesPDX 👋🏼

The Keepsake project is no longer actively maintained. If you're interested in helping maintain it, please let us know.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants