Add history log encoder for dashboards #3424

JCZuurmond · 2024-12-10T13:11:58Z

Changes

Add history log encoder for dashboards

Linked issues

Resolves #3368
Resolves #3369

Functionality

modified existing workflow: experimental-migration-progress

Tests

manually tested
added unit tests
added integration tests

github-actions · 2024-12-16T13:51:08Z

❌ 100/101 passed, 9 flaky, 1 failed, 7 skipped, 5h24m27s total

❌ test_redash_dashboard_ownership_is_me: AssertionError: Invalid owner for dashboard: Dashboard(id='b7199ffc-e28c-41e7-8bbf-edf3e64cd63d', name='ucx_D7n9L_ra78a5a6a0', parent='folders/3865756826903956', query_ids=['1cb92323-637b-487b-94ef-22724c98cab4'], tags=['original_dashboard_tag'], creator_id='481119220561874') (4m13.495s)

AssertionError: Invalid owner for dashboard: Dashboard(id='b7199ffc-e28c-41e7-8bbf-edf3e64cd63d', name='ucx_D7n9L_ra78a5a6a0', parent='folders/3865756826903956', query_ids=['1cb92323-637b-487b-94ef-22724c98cab4'], tags=['original_dashboard_tag'], creator_id='481119220561874')
assert 'alexander.ru...atabricks.com' == '0a330eb5-dd5...-c474356b1d5d'
  
  - 0a330eb5-dd51-4d97-b6e4-c474356b1d5d
  + [email protected]
[gw9] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python
14:37 INFO [tests.integration.conftest] Dashboard Created ucx_D7n9L_ra78a5a6a0: https://DATABRICKS_HOST/sql/dashboards/b7199ffc-e28c-41e7-8bbf-edf3e64cd63d
14:37 WARNING [databricks.labs.ucx.assessment.dashboards] Could not retrieve user: 481119220561874
Traceback (most recent call last):
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/assessment/dashboards.py", line 415, in _get_user_name
    user = self._ws.users.get(user_id)
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/service/iam.py", line 3435, in get
    res = self._api.do('GET', f'/api/2.0/preview/scim/v2/Users/{id}', query=query, headers=headers)
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/core.py", line 77, in do
    return self._api_client.do(method=method,
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/_base_client.py", line 172, in do
    response = call(method,
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/retries.py", line 54, in wrapper
    raise err
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/retries.py", line 33, in wrapper
    return func(*args, **kwargs)
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/_base_client.py", line 278, in _perform
    raise error from None
databricks.sdk.errors.platform.NotFound: None User with id 481119220561874 not found.
14:37 WARNING [databricks.labs.ucx.framework.owners] Cannot retrieve status for: folders/3865756826903956
14:37 DEBUG [databricks.labs.ucx.framework.owners] Enumerating users to locate active workspace administrators...
14:37 DEBUG [databricks.labs.ucx.framework.owners] Verifying membership of the 'admins' workspace group for users: [User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.remorph.write', primary=None, ref='Groups/332446109793000', type='direct', value='332446109793000'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730'), ComplexValue(display='labs.scope.tool', primary=None, ref='Groups/531996560706268', type='indirect', value='531996560706268'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239'), ComplexValue(display='role.labs.ucx.write', primary=None, ref='Groups/558249464776489', type='direct', value='558249464776489')], id='24123045404007', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.dbignite.write', primary=None, ref='Groups/880797712442732', type='direct', value='880797712442732'), ComplexValue(display='role.labs.splunk-app.maintain', primary=None, ref='Groups/867146854789835', type='direct', value='867146854789835'), ComplexValue(display='role.labs.smolder.write', primary=None, ref='Groups/972285451353531', type='direct', value='972285451353531'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730'), ComplexValue(display='role.labs.lsql.write', primary=None, ref='Groups/1090561499178599', type='direct', value='1090561499178599'), ComplexValue(display='labs.scope.runtime', primary=None, ref='Groups/300667344111082', type='indirect', value='300667344111082'), ComplexValue(display='role.labs.TEST_CATALOG.write', primary=None, ref='Groups/1054410134247190', type='direct', value='1054410134247190'), ComplexValue(display='labs.scope.tool', primary=None, ref='Groups/531996560706268', type='indirect', value='531996560706268'), ComplexValue(display='role.labs.overwatch.write', primary=None, ref='Groups/126055207032596', type='direct', value='126055207032596'), ComplexValue(display='role.labs.blueprint.write', primary=None, ref='Groups/956728708332625', type='direct', value='956728708332625'), ComplexValue(display='role.labs.watchdog.write', primary=None, ref='Groups/1025833911894044', type='direct', value='1025833911894044')], id='661448457191611', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.overwatch.write', primary=None, ref='Groups/126055207032596', type='direct', value='126055207032596'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239'), ComplexValue(display='labs.scope.tool', primary=None, ref='Groups/531996560706268', type='indirect', value='531996560706268'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='role.labs.remorph.write', primary=None, ref='Groups/332446109793000', type='direct', value='332446109793000')], id='683434022113008', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='labs.scope.runtime', primary=None, ref='Groups/300667344111082', type='indirect', value='300667344111082'), ComplexValue(display='role.labs.delta-oms.write', primary=None, ref='Groups/925885353519506', type='direct', value='925885353519506'), ComplexValue(display='role.labs.feature-factory.maintain', primary=None, ref='Groups/237719608535457', type='direct', value='237719608535457'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='role.labs.overwatch.maintain', primary=None, ref='Groups/929610554024196', type='direct', value='929610554024196'), ComplexValue(display='role.labs.dfrules.maintain', primary=None, ref='Groups/1002224683468154', type='direct', value='1002224683468154'), ComplexValue(display='role.labs.datagen.maintain', primary=None, ref='Groups/347106088269125', type='direct', value='347106088269125')], id='1225709386181028', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.sdk-r.write', primary=None, ref='Groups/179198484394146', type='direct', value='179198484394146'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='1256651402323171', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.overwatch.write', primary=None, ref='Groups/126055207032596', type='direct', value='126055207032596'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='1728866782013041', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='role.labs.TEST_CATALOG.write', primary=None, ref='Groups/1054410134247190', type='direct', value='1054410134247190'), ComplexValue(display='labs.scope.tool', primary=None, ref='Groups/531996560706268', type='indirect', value='531996560706268'), ComplexValue(display='role.labs.ucx.write', primary=None, ref='Groups/558249464776489', type='direct', value='558249464776489')], id='2296939550881230', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.overwatch.write', primary=None, ref='Groups/126055207032596', type='direct', value='126055207032596'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='2324459211697819', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.ucx.write', primary=None, ref='Groups/558249464776489', type='direct', value='558249464776489'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='2562508731442717', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.ucx.write', primary=None, ref='Groups/558249464776489', type='direct', value='558249464776489'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='2614179680770278', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.sdk-rust.write', primary=None, ref='Groups/1011887706650232', type='direct', value='1011887706650232'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='2697155103502290', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='labs.scope.runtime', primary=None, ref='Groups/300667344111082', type='indirect', value='300667344111082'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='role.labs.delta-oms.write', primary=None, ref='Groups/925885353519506', type='direct', value='925885353519506'), ComplexValue(display='role.labs.dbsync.write', primary=None, ref='Groups/890738974305402', type='direct', value='890738974305402')], id='3272374305027664', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.tool', primary=None, ref='Groups/531996560706268', type='indirect', value='531996560706268'), ComplexValue(display='role.labs.TEST_CATALOG.write', primary=None, ref='Groups/1054410134247190', type='direct', value='1054410134247190'), ComplexValue(display='role.labs.ucx.write', primary=None, ref='Groups/558249464776489', type='direct', value='558249464776489')], id='3297263947987260', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.ucx.write', primary=None, ref='Groups/558249464776489', type='direct', value='558249464776489'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='3619520043928476', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.sdk-dotnet.write', primary=None, ref='Groups/518828574352171', type='direct', value='518828574352171'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='4085253546229304', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.ucx.write', primary=None, ref='Groups/558249464776489', type='direct', value='558249464776489'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='4142558850594997', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.sdk-dotnet.maintain', primary=None, ref='Groups/422903443390924', type='direct', value='422903443390924'), ComplexValue(display='role.labs.sdk-r.maintain', primary=None, ref='Groups/69514106516856', type='direct', value='69514106516856'), ComplexValue(display='role.labs.TEST_CATALOG.maintain', primary=None, ref='Groups/891852804756554', type='direct', value='891852804756554'), ComplexValue(display='role.labs.remorph.maintain', primary=None, ref='Groups/75982969078436', type='direct', value='75982969078436'), ComplexValue(display='labs.scope.runtime', primary=None, ref='Groups/300667344111082', type='indirect', value='300667344111082'), ComplexValue(display='role.labs.watchdog.maintain', primary=None, ref='Groups/362656320875527', type='direct', value='362656320875527'), ComplexValue(display='labs-oss', primary=None, ref='Groups/852860106691249', type='direct', value='852860106691249'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='direct', value='194385937528161'), ComplexValue(display='role.labs.sdk-rust.maintain', primary=None, ref='Groups/400100643391091', type='direct', value='400100643391091'), ComplexValue(display='role.labs.lsql.maintain', primary=None, ref='Groups/583508855493027', type='direct', value='583508855493027'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730'), ComplexValue(display='role.labs.transpiler.maintain', primary=None, ref='Groups/51203042855777', type='direct', value='51203042855777'), ComplexValue(display='labs.scope.tool', primary=None, ref='Groups/531996560706268', type='indirect', value='531996560706268'), ComplexValue(display='role.labs.blueprint.maintain', primary=None, ref='Groups/1118462350168987', type='direct', value='1118462350168987'), ComplexValue(display='role.labs.ucx.maintain', primary=None, ref='Groups/658020652963371', type='direct', value='658020652963371'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='4183391249163402', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.ucx.write', primary=None, ref='Groups/558249464776489', type='direct', value='558249464776489'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='4256879669741685', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.overwatch.write', primary=None, ref='Groups/126055207032596', type='direct', value='126055207032596'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='4313162842336807', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='labs.scope.tool', primary=None, ref='Groups/531996560706268', type='indirect', value='531996560706268'), ComplexValue(display='role.labs.remorph.write', primary=None, ref='Groups/332446109793000', type='direct', value='332446109793000'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239'), ComplexValue(display='role.labs.overwatch.write', primary=None, ref='Groups/126055207032596', type='direct', value='126055207032596'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161')], id='5220576099010372', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.overwatch.write', primary=None, ref='Groups/126055207032596', type='direct', value='126055207032596'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='5629836015416408', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='role.labs.overwatch.write', primary=None, ref='Groups/126055207032596', type='direct', value='126055207032596'), ComplexValue(display='labs.scope.tool', primary=None, ref='Groups/531996560706268', type='indirect', value='531996560706268'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239'), ComplexValue(display='role.labs.dlt-meta.write', primary=None, ref='Groups/408004270888312', type='direct', value='408004270888312')], id='5818361206914695', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.ucx.write', primary=None, ref='Groups/558249464776489', type='direct', value='558249464776489'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='5934952062729548', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.overwatch.maintain', primary=None, ref='Groups/929610554024196', type='direct', value='929610554024196'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='6472610608479780', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239'), ComplexValue(display='role.labs.TEST_CATALOG.write', primary=None, ref='Groups/1054410134247190', type='direct', value='1054410134247190'), ComplexValue(display='role.labs.remorph.write', primary=None, ref='Groups/332446109793000', type='direct', value='332446109793000'), ComplexValue(display='role.labs.ucx.write', primary=None, ref='Groups/558249464776489', type='direct', value='558249464776489'), ComplexValue(display='labs.scope.tool', primary=None, ref='Groups/531996560706268', type='indirect', value='531996560706268'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161')], id='6779888502363704', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='labs-oss', primary=None, ref='Groups/852860106691249', type='direct', value='852860106691249'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730')], id='7344182020334936', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239'), ComplexValue(display='role.labs.remorph.maintain', primary=None, ref='Groups/75982969078436', type='direct', value='75982969078436'), ComplexValue(display='role.labs.blueprint.write', primary=None, ref='Groups/956728708332625', type='direct', value='956728708332625'), ComplexValue(display='labs.scope.tool', primary=None, ref='Groups/531996560706268', type='indirect', value='531996560706268'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='role.labs.overwatch.write', primary=None, ref='Groups/126055207032596', type='direct', value='126055207032596'), ComplexValue(display='labs.scope.runtime', primary=None, ref='Groups/300667344111082', type='indirect', value='300667344111082')], id='7534000974286519', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.migrate.maintain', primary=None, ref='Groups/174848272515934', type='direct', value='174848272515934'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='7603434113437775', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.ucx.write', primary=None, ref='Groups/558249464776489', type='direct', value='558249464776489'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='7816683983824279', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.ucx.write', primary=None, ref='Groups/558249464776489', type='direct', value='558249464776489'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='8077637869895037', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.watchdog.write', primary=None, ref='Groups/1025833911894044', type='direct', value='1025833911894044'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730')], id='8609205844338373', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.migrate.maintain', primary=None, ref='Groups/174848272515934', type='direct', value='174848272515934'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='8922191418148017', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.sdk-r.write', primary=None, ref='Groups/179198484394146', type='direct', value='179198484394146'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='8957403854939895', name=None, roles=[], schemas=None, user_name='[email protected]')]
14:37 INFO [tests.integration.conftest] Dashboard Created ucx_D7n9L_ra78a5a6a0: https://DATABRICKS_HOST/sql/dashboards/b7199ffc-e28c-41e7-8bbf-edf3e64cd63d
14:37 WARNING [databricks.labs.ucx.assessment.dashboards] Could not retrieve user: 481119220561874
Traceback (most recent call last):
  File "/home/runner/work/ucx/ucx/src/databricks/labs/ucx/assessment/dashboards.py", line 415, in _get_user_name
    user = self._ws.users.get(user_id)
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/service/iam.py", line 3435, in get
    res = self._api.do('GET', f'/api/2.0/preview/scim/v2/Users/{id}', query=query, headers=headers)
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/core.py", line 77, in do
    return self._api_client.do(method=method,
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/_base_client.py", line 172, in do
    response = call(method,
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/retries.py", line 54, in wrapper
    raise err
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/retries.py", line 33, in wrapper
    return func(*args, **kwargs)
  File "/home/runner/work/ucx/ucx/.venv/lib/python3.10/site-packages/databricks/sdk/_base_client.py", line 278, in _perform
    raise error from None
databricks.sdk.errors.platform.NotFound: None User with id 481119220561874 not found.
14:37 WARNING [databricks.labs.ucx.framework.owners] Cannot retrieve status for: folders/3865756826903956
14:37 DEBUG [databricks.labs.ucx.framework.owners] Enumerating users to locate active workspace administrators...
14:37 DEBUG [databricks.labs.ucx.framework.owners] Verifying membership of the 'admins' workspace group for users: [User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.remorph.write', primary=None, ref='Groups/332446109793000', type='direct', value='332446109793000'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730'), ComplexValue(display='labs.scope.tool', primary=None, ref='Groups/531996560706268', type='indirect', value='531996560706268'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239'), ComplexValue(display='role.labs.ucx.write', primary=None, ref='Groups/558249464776489', type='direct', value='558249464776489')], id='24123045404007', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.dbignite.write', primary=None, ref='Groups/880797712442732', type='direct', value='880797712442732'), ComplexValue(display='role.labs.splunk-app.maintain', primary=None, ref='Groups/867146854789835', type='direct', value='867146854789835'), ComplexValue(display='role.labs.smolder.write', primary=None, ref='Groups/972285451353531', type='direct', value='972285451353531'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730'), ComplexValue(display='role.labs.lsql.write', primary=None, ref='Groups/1090561499178599', type='direct', value='1090561499178599'), ComplexValue(display='labs.scope.runtime', primary=None, ref='Groups/300667344111082', type='indirect', value='300667344111082'), ComplexValue(display='role.labs.TEST_CATALOG.write', primary=None, ref='Groups/1054410134247190', type='direct', value='1054410134247190'), ComplexValue(display='labs.scope.tool', primary=None, ref='Groups/531996560706268', type='indirect', value='531996560706268'), ComplexValue(display='role.labs.overwatch.write', primary=None, ref='Groups/126055207032596', type='direct', value='126055207032596'), ComplexValue(display='role.labs.blueprint.write', primary=None, ref='Groups/956728708332625', type='direct', value='956728708332625'), ComplexValue(display='role.labs.watchdog.write', primary=None, ref='Groups/1025833911894044', type='direct', value='1025833911894044')], id='661448457191611', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.overwatch.write', primary=None, ref='Groups/126055207032596', type='direct', value='126055207032596'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239'), ComplexValue(display='labs.scope.tool', primary=None, ref='Groups/531996560706268', type='indirect', value='531996560706268'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='role.labs.remorph.write', primary=None, ref='Groups/332446109793000', type='direct', value='332446109793000')], id='683434022113008', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='labs.scope.runtime', primary=None, ref='Groups/300667344111082', type='indirect', value='300667344111082'), ComplexValue(display='role.labs.delta-oms.write', primary=None, ref='Groups/925885353519506', type='direct', value='925885353519506'), ComplexValue(display='role.labs.feature-factory.maintain', primary=None, ref='Groups/237719608535457', type='direct', value='237719608535457'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='role.labs.overwatch.maintain', primary=None, ref='Groups/929610554024196', type='direct', value='929610554024196'), ComplexValue(display='role.labs.dfrules.maintain', primary=None, ref='Groups/1002224683468154', type='direct', value='1002224683468154'), ComplexValue(display='role.labs.datagen.maintain', primary=None, ref='Groups/347106088269125', type='direct', value='347106088269125')], id='1225709386181028', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.sdk-r.write', primary=None, ref='Groups/179198484394146', type='direct', value='179198484394146'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='1256651402323171', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.overwatch.write', primary=None, ref='Groups/126055207032596', type='direct', value='126055207032596'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='1728866782013041', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='role.labs.TEST_CATALOG.write', primary=None, ref='Groups/1054410134247190', type='direct', value='1054410134247190'), ComplexValue(display='labs.scope.tool', primary=None, ref='Groups/531996560706268', type='indirect', value='531996560706268'), ComplexValue(display='role.labs.ucx.write', primary=None, ref='Groups/558249464776489', type='direct', value='558249464776489')], id='2296939550881230', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.overwatch.write', primary=None, ref='Groups/126055207032596', type='direct', value='126055207032596'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='2324459211697819', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.ucx.write', primary=None, ref='Groups/558249464776489', type='direct', value='558249464776489'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='2562508731442717', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.ucx.write', primary=None, ref='Groups/558249464776489', type='direct', value='558249464776489'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='2614179680770278', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.sdk-rust.write', primary=None, ref='Groups/1011887706650232', type='direct', value='1011887706650232'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='2697155103502290', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='labs.scope.runtime', primary=None, ref='Groups/300667344111082', type='indirect', value='300667344111082'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='role.labs.delta-oms.write', primary=None, ref='Groups/925885353519506', type='direct', value='925885353519506'), ComplexValue(display='role.labs.dbsync.write', primary=None, ref='Groups/890738974305402', type='direct', value='890738974305402')], id='3272374305027664', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.tool', primary=None, ref='Groups/531996560706268', type='indirect', value='531996560706268'), ComplexValue(display='role.labs.TEST_CATALOG.write', primary=None, ref='Groups/1054410134247190', type='direct', value='1054410134247190'), ComplexValue(display='role.labs.ucx.write', primary=None, ref='Groups/558249464776489', type='direct', value='558249464776489')], id='3297263947987260', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.ucx.write', primary=None, ref='Groups/558249464776489', type='direct', value='558249464776489'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='3619520043928476', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.sdk-dotnet.write', primary=None, ref='Groups/518828574352171', type='direct', value='518828574352171'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='4085253546229304', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.ucx.write', primary=None, ref='Groups/558249464776489', type='direct', value='558249464776489'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='4142558850594997', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.sdk-dotnet.maintain', primary=None, ref='Groups/422903443390924', type='direct', value='422903443390924'), ComplexValue(display='role.labs.sdk-r.maintain', primary=None, ref='Groups/69514106516856', type='direct', value='69514106516856'), ComplexValue(display='role.labs.TEST_CATALOG.maintain', primary=None, ref='Groups/891852804756554', type='direct', value='891852804756554'), ComplexValue(display='role.labs.remorph.maintain', primary=None, ref='Groups/75982969078436', type='direct', value='75982969078436'), ComplexValue(display='labs.scope.runtime', primary=None, ref='Groups/300667344111082', type='indirect', value='300667344111082'), ComplexValue(display='role.labs.watchdog.maintain', primary=None, ref='Groups/362656320875527', type='direct', value='362656320875527'), ComplexValue(display='labs-oss', primary=None, ref='Groups/852860106691249', type='direct', value='852860106691249'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='direct', value='194385937528161'), ComplexValue(display='role.labs.sdk-rust.maintain', primary=None, ref='Groups/400100643391091', type='direct', value='400100643391091'), ComplexValue(display='role.labs.lsql.maintain', primary=None, ref='Groups/583508855493027', type='direct', value='583508855493027'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730'), ComplexValue(display='role.labs.transpiler.maintain', primary=None, ref='Groups/51203042855777', type='direct', value='51203042855777'), ComplexValue(display='labs.scope.tool', primary=None, ref='Groups/531996560706268', type='indirect', value='531996560706268'), ComplexValue(display='role.labs.blueprint.maintain', primary=None, ref='Groups/1118462350168987', type='direct', value='1118462350168987'), ComplexValue(display='role.labs.ucx.maintain', primary=None, ref='Groups/658020652963371', type='direct', value='658020652963371'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='4183391249163402', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.ucx.write', primary=None, ref='Groups/558249464776489', type='direct', value='558249464776489'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='4256879669741685', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.overwatch.write', primary=None, ref='Groups/126055207032596', type='direct', value='126055207032596'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='4313162842336807', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='labs.scope.tool', primary=None, ref='Groups/531996560706268', type='indirect', value='531996560706268'), ComplexValue(display='role.labs.remorph.write', primary=None, ref='Groups/332446109793000', type='direct', value='332446109793000'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239'), ComplexValue(display='role.labs.overwatch.write', primary=None, ref='Groups/126055207032596', type='direct', value='126055207032596'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161')], id='5220576099010372', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.overwatch.write', primary=None, ref='Groups/126055207032596', type='direct', value='126055207032596'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='5629836015416408', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='role.labs.overwatch.write', primary=None, ref='Groups/126055207032596', type='direct', value='126055207032596'), ComplexValue(display='labs.scope.tool', primary=None, ref='Groups/531996560706268', type='indirect', value='531996560706268'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239'), ComplexValue(display='role.labs.dlt-meta.write', primary=None, ref='Groups/408004270888312', type='direct', value='408004270888312')], id='5818361206914695', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.ucx.write', primary=None, ref='Groups/558249464776489', type='direct', value='558249464776489'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='5934952062729548', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.overwatch.maintain', primary=None, ref='Groups/929610554024196', type='direct', value='929610554024196'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='6472610608479780', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239'), ComplexValue(display='role.labs.TEST_CATALOG.write', primary=None, ref='Groups/1054410134247190', type='direct', value='1054410134247190'), ComplexValue(display='role.labs.remorph.write', primary=None, ref='Groups/332446109793000', type='direct', value='332446109793000'), ComplexValue(display='role.labs.ucx.write', primary=None, ref='Groups/558249464776489', type='direct', value='558249464776489'), ComplexValue(display='labs.scope.tool', primary=None, ref='Groups/531996560706268', type='indirect', value='531996560706268'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161')], id='6779888502363704', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='labs-oss', primary=None, ref='Groups/852860106691249', type='direct', value='852860106691249'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730')], id='7344182020334936', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239'), ComplexValue(display='role.labs.remorph.maintain', primary=None, ref='Groups/75982969078436', type='direct', value='75982969078436'), ComplexValue(display='role.labs.blueprint.write', primary=None, ref='Groups/956728708332625', type='direct', value='956728708332625'), ComplexValue(display='labs.scope.tool', primary=None, ref='Groups/531996560706268', type='indirect', value='531996560706268'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='role.labs.overwatch.write', primary=None, ref='Groups/126055207032596', type='direct', value='126055207032596'), ComplexValue(display='labs.scope.runtime', primary=None, ref='Groups/300667344111082', type='indirect', value='300667344111082')], id='7534000974286519', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.migrate.maintain', primary=None, ref='Groups/174848272515934', type='direct', value='174848272515934'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='7603434113437775', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.ucx.write', primary=None, ref='Groups/558249464776489', type='direct', value='558249464776489'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='7816683983824279', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.ucx.write', primary=None, ref='Groups/558249464776489', type='direct', value='558249464776489'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='8077637869895037', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.watchdog.write', primary=None, ref='Groups/1025833911894044', type='direct', value='1025833911894044'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.account-admin', primary=None, ref='Groups/688239313962730', type='indirect', value='688239313962730')], id='8609205844338373', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.migrate.maintain', primary=None, ref='Groups/174848272515934', type='direct', value='174848272515934'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='8922191418148017', name=None, roles=[], schemas=None, user_name='[email protected]'), User(active=True, display_name=None, emails=[], entitlements=[], external_id=None, groups=[ComplexValue(display='role.labs.sdk-r.write', primary=None, ref='Groups/179198484394146', type='direct', value='179198484394146'), ComplexValue(display='admins', primary=None, ref='Groups/194385937528161', type='indirect', value='194385937528161'), ComplexValue(display='labs.scope.admin', primary=None, ref='Groups/847659649002239', type='indirect', value='847659649002239')], id='8957403854939895', name=None, roles=[], schemas=None, user_name='[email protected]')]
[gw9] linux -- Python 3.10.15 /home/runner/work/ucx/ucx/.venv/bin/python

Flaky tests:

🤪 test_udf_ownership (5m17.608s)
🤪 test_migrate_managed_tables_with_acl (7m59.413s)
🤪 test_migrate_external_table_hiveserde_ctas (7m11.167s)
🤪 test_hiveserde_table_in_place_migration_job[migrate-external-hiveserde-tables-in-place-experimental] (14m47.344s)
🤪 test_table_migration_job_refreshes_migration_status[regular-migrate-tables] (11m30.804s)
🤪 test_table_migration_job_publishes_remaining_tables (14m0.238s)
🤪 test_table_migration_convert_manged_to_external (12m52.461s)
🤪 test_table_migration_job_refreshes_migration_status[hiveserde-migrate-external-tables-ctas] (9m20.595s)
🤪 test_running_real_migration_progress_job (7m50.791s)

_{Running from acceptance #7806}

The directionality will be the other way around: code points to used tables

FastLee

Small Changes

FastLee · 2024-12-20T15:34:28Z

src/databricks/labs/ucx/assessment/dashboards.py

@@ -410,7 +413,7 @@ def _maybe_direct_owner(self, record: Dashboard) -> str | None:
    def _get_user_name(self, user_id: str) -> str | None:
        try:
            user = self._ws.users.get(user_id)
-            return user.display_name or user.user_name


Are we sure about that?

FastLee · 2024-12-20T15:36:02Z

src/databricks/labs/ucx/contexts/application.py

@@ -266,6 +267,10 @@ def udfs_crawler(self) -> UdfsCrawler:
    def udf_ownership(self) -> UdfOwnership:
        return UdfOwnership(self.administrator_locator)

+    @cached_property


Should we consider combining all ownerships to a single method?

FastLee · 2024-12-20T15:39:27Z

src/databricks/labs/ucx/framework/owners.py

+            logger.warning(f"Cannot retrieve status for: {path}")
+            return None
+        if not (object_info.object_id and object_info.object_type):
+            return None
        object_id = str(object_info.object_id)
        match object_info.object_type:


Change to return object_info.object_type.lower(), object_id

FastLee · 2024-12-20T15:43:05Z

src/databricks/labs/ucx/progress/dashboards.py

+        self._inventory_database = inventory_database
+        self._used_tables_crawlers = used_tables_crawlers
+
+    def append_inventory_snapshot(self, snapshot: Iterable[Dashboard]) -> None:


Suggested change

def append_inventory_snapshot(self, snapshot: Iterable[Dashboard]) -> None:

def append_dashboards(self, snapshot: dashboards[Dashboard]) -> None:

FastLee · 2024-12-20T15:45:51Z

src/databricks/labs/ucx/progress/dashboards.py

+
+    def _get_query_problems(self) -> DashboardIdToFailuresType:
+        index = collections.defaultdict(list)
+        for row in self._sql_backend.fetch(


Shouldn't we use a crawler for that?

FastLee · 2024-12-20T15:46:31Z

src/databricks/labs/ucx/progress/dashboards.py

+
+    def _get_tables_failures(self) -> DashboardIdToFailuresType:
+        table_failures = {}
+        for row in self._sql_backend.fetch(


Same as above

JCZuurmond added the feat/migration-progress Issues related to the migration progress workflow label Dec 10, 2024

JCZuurmond requested a review from asnare December 10, 2024 13:11

JCZuurmond self-assigned this Dec 10, 2024

JCZuurmond force-pushed the feat/add-history-log-encoder-for-dashboards branch from f417e8d to 3bf9a07 Compare December 16, 2024 08:34

JCZuurmond requested review from FastLee and pritishpai December 16, 2024 13:21

JCZuurmond marked this pull request as ready for review December 16, 2024 13:21

JCZuurmond requested a review from a team as a code owner December 16, 2024 13:21

JCZuurmond had a problem deploying to account-admin December 16, 2024 13:21 — with GitHub Actions Failure

JCZuurmond had a problem deploying to account-admin December 17, 2024 10:35 — with GitHub Actions Error

JCZuurmond force-pushed the feat/add-history-log-encoder-for-dashboards branch from 0386b5c to 54c4442 Compare December 17, 2024 10:36

JCZuurmond had a problem deploying to account-admin December 17, 2024 10:37 — with GitHub Actions Failure

JCZuurmond mentioned this pull request Dec 17, 2024

Add dashboards to migration progress dashboard #3314

Open

6 tasks

JCZuurmond had a problem deploying to account-admin December 20, 2024 10:17 — with GitHub Actions Error

JCZuurmond added 15 commits December 20, 2024 11:38

Set LineageAtom.other to dict by default

b9902db

Add dashboard progress encoder

e26a3ba

Get table failures from historical table snapshot

d2e2560

Allow other to be None

6f2e215

Remove cached properties from dashboard progress encoder

9d0cae8

Add first integration test for dashboard progress encoder

d1fd047

Test dashboard progress encoder without failures

f997440

Test dashboard failure coming from query problem

377a2de

Test dashboard failure coming from dfsa

8b5b2ba

Rewrite tests to assert on historical rows

a86e854

Test used table failure from hive metastore table

74a655d

Merge tests

b93518a

Test Table.from_historical

e16a26d

Format

83ce063

Assert row in integration test

a1f0b06

JCZuurmond added 22 commits December 20, 2024 11:38

Remove used tables crawler from TableProgressEncoder

b7b6a29

The directionality will be the other way around: code points to used tables

Pass failure from used table to dashboard

0a88e9d

Persist dashboard migration progress in workflow

1af0e47

Force run integration test in CI

c12dfe1

Test Redash ownership is me

5fd8c8c

Test ownership of directory

c09714e

Support ownership of directory

ec76986

Test owner of directory

6eae00e

Test retrieving ownership for invalid path

89e5743

Handle invalid path

ae7a9e4

Add missing type hints

c7916a4

Test warn about unsupported object type

4e85182

Warn about unsupported object type

849079e

Test Lakeview dashboard ownership is me

51b7a0d

Test getting user name

cd2c051

Handle resource does not exists

180ab07

Handle None types

f399ff5

Remove legacy test

34f566f

Update unit test

2e45735

Rename me to current_user

ecdef76

The owner of an invalid path should fallback on the workspace admin

474ee7d

Improve assert message

8beff2c

JCZuurmond force-pushed the feat/add-history-log-encoder-for-dashboards branch from 78e563b to 8beff2c Compare December 20, 2024 10:40

JCZuurmond had a problem deploying to account-admin December 20, 2024 10:40 — with GitHub Actions Failure

Skip test when running in debug

7a50eff

JCZuurmond had a problem deploying to account-admin December 20, 2024 14:19 — with GitHub Actions Error

JCZuurmond had a problem deploying to account-admin December 20, 2024 14:36 — with GitHub Actions Error

JCZuurmond force-pushed the feat/add-history-log-encoder-for-dashboards branch from 02282a0 to 7a50eff Compare December 20, 2024 14:36

JCZuurmond had a problem deploying to account-admin December 20, 2024 14:36 — with GitHub Actions Failure

FastLee reviewed Dec 20, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add history log encoder for dashboards #3424

Add history log encoder for dashboards #3424

JCZuurmond commented Dec 10, 2024 •

edited

Loading

github-actions bot commented Dec 16, 2024 •

edited

Loading

FastLee left a comment

FastLee Dec 20, 2024

FastLee Dec 20, 2024

FastLee Dec 20, 2024

FastLee Dec 20, 2024

FastLee Dec 20, 2024

FastLee Dec 20, 2024

	def append_inventory_snapshot(self, snapshot: Iterable[Dashboard]) -> None:
	def append_dashboards(self, snapshot: dashboards[Dashboard]) -> None:

Add history log encoder for dashboards #3424

Are you sure you want to change the base?

Add history log encoder for dashboards #3424

Conversation

JCZuurmond commented Dec 10, 2024 • edited Loading

Changes

Linked issues

Functionality

Tests

github-actions bot commented Dec 16, 2024 • edited Loading

FastLee left a comment

Choose a reason for hiding this comment

FastLee Dec 20, 2024

Choose a reason for hiding this comment

FastLee Dec 20, 2024

Choose a reason for hiding this comment

FastLee Dec 20, 2024

Choose a reason for hiding this comment

FastLee Dec 20, 2024

Choose a reason for hiding this comment

FastLee Dec 20, 2024

Choose a reason for hiding this comment

FastLee Dec 20, 2024

Choose a reason for hiding this comment

JCZuurmond commented Dec 10, 2024 •

edited

Loading

github-actions bot commented Dec 16, 2024 •

edited

Loading