Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: client side metrics data model #923

Open
wants to merge 60 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from 51 commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
b5b62c8
feat!: add new v3.0.0 API skeleton (#745)
daniel-sanche Mar 14, 2023
7d51eeb
chore: merge branch 'main' into v3
daniel-sanche Mar 15, 2023
507da99
feat: improve rows filters (#751)
daniel-sanche Mar 23, 2023
71b0312
feat: read rows query model class (#752)
daniel-sanche Apr 3, 2023
c55099f
feat: implement row and cell model classes (#753)
daniel-sanche Apr 5, 2023
f9a1907
feat: add pooled grpc transport (#748)
daniel-sanche Apr 24, 2023
3de7a68
feat: implement read_rows (#762)
daniel-sanche May 24, 2023
9b81289
feat: implement mutate rows (#769)
daniel-sanche Jun 6, 2023
ec3fd01
feat: literal value filter (#767)
daniel-sanche Jun 6, 2023
5d65703
feat: row_exists and read_row (#778)
daniel-sanche Jun 16, 2023
432d159
feat: read_modify_write and check_and_mutate_row (#780)
daniel-sanche Jun 16, 2023
ec2b983
feat: sharded read rows (#766)
daniel-sanche Jun 23, 2023
ceaf598
feat: ping and warm with metadata (#810)
daniel-sanche Jun 26, 2023
1ecf65f
feat: mutate rows batching (#770)
daniel-sanche Jun 26, 2023
eedde1e
chore: restructure module paths (#816)
daniel-sanche Jun 28, 2023
07438ca
feat: improve timeout structure (#819)
daniel-sanche Jun 29, 2023
0d92a84
fix: api errors apply to all bulk mutations
daniel-sanche Jul 24, 2023
a8cdf7c
chore: reduce public api surface (#820)
daniel-sanche Aug 1, 2023
aa760b2
feat: improve error group tracebacks on < py11 (#825)
daniel-sanche Aug 16, 2023
0323dde
feat: optimize read_rows (#852)
daniel-sanche Aug 17, 2023
0b3606f
chore: add user agent suffix (#842)
daniel-sanche Aug 17, 2023
b6d232a
feat: optimize retries (#854)
daniel-sanche Aug 17, 2023
8708a25
feat: add test proxy (#836)
daniel-sanche Aug 18, 2023
1d3a7c1
chore(tests): add conformance tests to CI for v3 (#870)
daniel-sanche Oct 16, 2023
50531e5
chore(tests): turn off fast fail for conformance tets (#882)
daniel-sanche Oct 26, 2023
8ff1216
feat: add TABLE_DEFAULTS enum for table method arguments (#880)
daniel-sanche Oct 26, 2023
94bfe66
fix: pass None for retry in gapic calls (#881)
daniel-sanche Oct 27, 2023
3ac80a9
feat: replace internal dictionaries with protos in gapic calls (#875)
daniel-sanche Nov 22, 2023
b191451
chore: optimize gapic calls (#863)
daniel-sanche Dec 1, 2023
285cdd3
feat: expose retryable error codes to users (#879)
daniel-sanche Dec 1, 2023
9342e27
chore: update api_core submodule (#897)
daniel-sanche Dec 15, 2023
858b93a
chore: merge main into experimental_v3 (#900)
daniel-sanche Dec 15, 2023
cc79d8c
chore: pin conformance tests to v0.0.2 (#903)
daniel-sanche Dec 15, 2023
f0d75de
fix: bulk mutation eventual success (#909)
daniel-sanche Dec 19, 2023
11532ef
feat: pulled in data model for client side metrics
daniel-sanche Jan 26, 2024
874a9a8
stripped out handlers for future PR
daniel-sanche Jan 26, 2024
d905462
added metric controls to table
daniel-sanche Jan 26, 2024
5f42046
removed api_core from mypy install
daniel-sanche Jan 27, 2024
4369d57
improved comments
daniel-sanche Jan 27, 2024
cc79cde
always log metric errors
daniel-sanche Feb 1, 2024
f6831bb
ran blacken
daniel-sanche Feb 1, 2024
3ffc7c0
use ResponseParams proto to parse blob
daniel-sanche Feb 1, 2024
2ef2c3c
convert all time variables to explicitly use ms
daniel-sanche Feb 1, 2024
00d70db
pulled backoff generator into PR
daniel-sanche Feb 1, 2024
24432c8
fixed checks
daniel-sanche Feb 1, 2024
619d982
Merge branch 'main' into client_side_metrics_data_model
daniel-sanche Feb 8, 2024
2c8506d
removed exception flag
daniel-sanche Feb 9, 2024
7408722
added comments
daniel-sanche Feb 16, 2024
e1fe3ab
improved server timing regex
daniel-sanche Feb 16, 2024
6d2e9f4
change float ms to int ns for timestamps
daniel-sanche Feb 16, 2024
6bf526e
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Feb 16, 2024
600ef0b
renamed variable
daniel-sanche Feb 23, 2024
10ef5d1
lint
daniel-sanche Feb 23, 2024
f9dab5d
turned backoff_generator into a class
daniel-sanche Mar 15, 2024
1986b06
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Mar 15, 2024
035dab1
added backoff generator test
daniel-sanche Mar 15, 2024
5a63965
🦉 Updates from OwlBot post-processor
gcf-owl-bot[bot] Mar 15, 2024
a5aceef
removed unused import
daniel-sanche Mar 20, 2024
24fe643
Merge branch 'main' into client_side_metrics_data_model
daniel-sanche Apr 22, 2024
b1d08c9
fixed test
daniel-sanche Apr 22, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 9 additions & 0 deletions google/cloud/bigtable/data/_async/client.py
Original file line number Diff line number Diff line change
Expand Up @@ -79,6 +79,8 @@
from google.cloud.bigtable.data.row_filters import CellsRowLimitFilter
from google.cloud.bigtable.data.row_filters import RowFilterChain

from google.cloud.bigtable.data._metrics import BigtableClientSideMetricsController


if TYPE_CHECKING:
from google.cloud.bigtable.data._helpers import RowKeySamples
Expand Down Expand Up @@ -532,6 +534,13 @@ def __init__(
)
self.default_mutate_rows_attempt_timeout = default_mutate_rows_attempt_timeout

self._metrics = BigtableClientSideMetricsController(
project_id=self.client.project,
instance_id=instance_id,
table_id=table_id,
app_profile_id=app_profile_id,
)

self.default_read_rows_retryable_errors = (
default_read_rows_retryable_errors or ()
)
Expand Down
20 changes: 20 additions & 0 deletions google/cloud/bigtable/data/_helpers.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
from google.cloud.bigtable.data.read_rows_query import ReadRowsQuery

from google.api_core import exceptions as core_exceptions
from google.api_core.retry import exponential_sleep_generator
from google.api_core.retry import RetryFailureReason
from google.cloud.bigtable.data.exceptions import RetryExceptionGroup

Expand Down Expand Up @@ -97,6 +98,25 @@ def _attempt_timeout_generator(
yield max(0, min(per_request_timeout, deadline - time.monotonic()))


def backoff_generator(initial=0.01, multiplier=2, maximum=60):
"""
Build a generator for exponential backoff sleep times.

This implementation builds on top of api_core.retries.exponential_sleep_generator,
adding the ability to retrieve previous values using the send(idx) method. This is
used by the Metrics class to track the sleep times used for each attempt.
"""
history = []
subgenerator = exponential_sleep_generator(initial, multiplier, maximum)
while True:
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dumb question: when will it break out of the loop?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a python generator function. It gives up control on each yield line

The idea is that you get an instance like generator = backoff_generator(...), and then you can call next(generator) or generator.send(idx) on it every time you want to retrieve a value. This will run the internal code until it reaches the next yield, and then pause execution again until next time a value is requested

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This seems a bit too surprising of an api. I think it would be a lot cleaner and easier to read if Attempt.start() took a delay parameter

next_backoff = next(subgenerator)
history.append(next_backoff)
sent_idx = yield next_backoff
while sent_idx is not None:
# requesting from history
sent_idx = yield history[sent_idx]


def _retry_exception_factory(
exc_list: list[Exception],
reason: RetryFailureReason,
Expand Down
25 changes: 25 additions & 0 deletions google/cloud/bigtable/data/_metrics/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# Copyright 2023 Google LLC
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
from google.cloud.bigtable.data._metrics.metrics_controller import (
BigtableClientSideMetricsController,
)

from google.cloud.bigtable.data._metrics.data_model import OperationType
from google.cloud.bigtable.data._metrics.data_model import ActiveOperationMetric

__all__ = (
"BigtableClientSideMetricsController",
"OperationType",
"ActiveOperationMetric",
)
Loading
Loading