Dashboard Support (#439)

* Initial commit for Dashboards feature (#394)

* Initial Dashboard commit
- Added a DashboardPage and route
- Added basic getDashboard reducer/saga/API
- Added basic tests for dashboard page
- Cleaned up tabs component styles

* Add chart and query component to dashboard page (#403)

* Fix rebase syntax error in _layouts.scss

* Initial dashboard search (#411)

* Dummy python code

* Python updates; React App support

* Some code cleanup

* Python tests

* Add some tests

* Lint/test fix

* Update ProfilePage & My Bookmarks to new design (#416)

* Dummy python code

* Python updates; React App support

* Some code cleanup

* Python tests

* Add some tests

* Lint/test fix

* Update ProfilePage & My Bookmarks to new design

* Add some tests

* Update Dashboard Search UI (#424)

* Update UI

* Cleanup; Add some tests; Update doc

* Lint fix

* Update search payload used

* Lint fix; Test fix;

* Some last tests

* Add Dashboard Preview UI Support (#425)

* WIP

* Default loading error message; Resolve titleNode vs title

* Cleanup ducks

* Component tests

* Ducks tests

* Log exception omn file not found error

* Dashboard Bookmarks & Dashboard ProfilePage support (#427)

* Initial implementation

* Some tests; Some cleanup; Add owners

* Code cleanup; Tests

* Connect dashboard page with API (#426)

* Connected the dashboard get API
- Added several metadata fields
- Added dashboard Tables
* Added dashboard tag api
* Refactor tags to support multiple resource types
* Fixed tag refresh logic
* Removed tag extraction helper methods

* Update dashboard preview UI (#430)

* Update endpoint logic; Update ImagePreview

* Remove preview redux logic

* Tests

* Some code cleanup

* Add dashboard url to preview error message

* Switch order of People & Dashboards across search UI (#431)

* Dashboard Page Updates (#432)

* Update last run state ui; Update badge colors

* Add support for 'No owner' UI

* Add icon to DashboardPage; Add update description UI

* Update description UI according to direction from Design

* Code cleanup based on review comments

* List Item UI Updates (#433)

* Update last run state ui; Update badge colors

* Add support for 'No owner' UI

* Add icon to DashboardPage; Add update description UI

* Update description UI according to direction from Design

* Code cleanup based on review comments

* Update dashboard group css

* Center list items; Update queryitem height; Add chartitem keys

* TabsComponent border always under tabs

* Do not render ChartList if no charts

* Center UserListItem

* Add count to dashboard right panel tabs (#438)

* Cleanup: Delete unused code

* Cleanup: Remove unsued import

* Dashboard Logging Updates (#436)

* Support logging dashboard source & index; Consolidate logic into utility

* log interaction with description edit

* Update method name for a less ambiguous command name in logs

* Revert add description logging -- there is a new design

* Cleanup

* Improve coverage under DashboardPage

* Update ImagePreview; Fix imports

* Add a dashboardsummary fixture; Fix indentation

* ducks/dashboard tests

* Update application_config doc

* Added a 'beta' flag for dashboards (#441)

* Added a 'beta' flag for dashboards

* Reorder imports

* Remove config for hard coding beta tags

* Cleanup

* Dashboard -> DashboardMetadata

* Remove obsolete TODO; Use DashboardSummary schema

* Remove other TODO

* Fix python tests & lint

Co-authored-by: Daniel <[email protected]>

frontend/amundsen_application/api/metadata/v0.py

@@ -11,7 +11,8 @@
 
 from amundsen_application.models.user import load_user, dump_user
 
-from amundsen_application.api.utils.metadata_utils import marshall_table_partial, marshall_table_full
+from amundsen_application.api.utils.metadata_utils import marshall_table_partial, marshall_table_full,\
+    marshall_dashboard_partial, marshall_dashboard_full
 from amundsen_application.api.utils.request_utils import get_query_param, request_metadata
 
 
@@ -25,6 +26,7 @@
 POPULAR_TABLES_ENDPOINT = '/popular_tables/'
 TAGS_ENDPOINT = '/tags/'
 USER_ENDPOINT = '/user'
+DASHBOARD_ENDPOINT = '/dashboard'
 
 
 def _get_table_endpoint() -> str:
@@ -34,6 +36,13 @@ def _get_table_endpoint() -> str:
     return table_endpoint
 
 
+def _get_dashboard_endpoint() -> str:
+    dashboard_endpoint = app.config['METADATASERVICE_BASE'] + DASHBOARD_ENDPOINT
+    if dashboard_endpoint is None:
+        raise Exception('An request endpoint for dashboard resources must be configured')
+    return dashboard_endpoint
+
+
 @metadata_blueprint.route('/popular_tables', methods=['GET'])
 def popular_tables() -> Response:
     """
@@ -366,7 +375,7 @@ def _log_update_table_tags(*, table_key: str, method: str, tag: str) -> None:
 
         tag = get_query_param(args, 'tag')
 
-        url = '{0}/{1}/tag/{2}'.format(table_endpoint, table_key, tag)
+        url = f'{table_endpoint}/{table_key}/tag/{tag}'
 
         _log_update_table_tags(table_key=table_key, method=method, tag=tag)
 
@@ -376,7 +385,43 @@ def _log_update_table_tags(*, table_key: str, method: str, tag: str) -> None:
         if status_code == HTTPStatus.OK:
             message = 'Success'
         else:
-            message = 'Encountered error: {0} tag failed'.format(method)
+            message = f'Encountered error: {method} table tag failed'
+            logging.error(message)
+
+        payload = jsonify({'msg': message})
+        return make_response(payload, status_code)
+    except Exception as e:
+        message = 'Encountered exception: ' + str(e)
+        logging.exception(message)
+        payload = jsonify({'msg': message})
+        return make_response(payload, HTTPStatus.INTERNAL_SERVER_ERROR)
+
+
+@metadata_blueprint.route('/update_dashboard_tags', methods=['PUT', 'DELETE'])
+def update_dashboard_tags() -> Response:
+
+    @action_logging
+    def _log_update_dashboard_tags(*, uri_key: str, method: str, tag: str) -> None:
+        pass  # pragma: no cover
+
+    try:
+        args = request.get_json()
+        method = request.method
+
+        dashboard_endpoint = _get_dashboard_endpoint()
+        uri_key = get_query_param(args, 'key')
+        tag = get_query_param(args, 'tag')
+        url = f'{dashboard_endpoint}/{uri_key}/tag/{tag}'
+
+        _log_update_dashboard_tags(uri_key=uri_key, method=method, tag=tag)
+
+        response = request_metadata(url=url, method=method)
+        status_code = response.status_code
+
+        if status_code == HTTPStatus.OK:
+            message = 'Success'
+        else:
+            message = f'Encountered error: {method} dashboard tag failed'
             logging.error(message)
 
         payload = jsonify({'msg': message})
@@ -449,12 +494,19 @@ def get_bookmark() -> Response:
             message = 'Success'
             tables = response.json().get('table')
             table_bookmarks = [marshall_table_partial(table) for table in tables]
+            dashboards = response.json().get('dashboard', [])
+            dashboard_bookmarks = [marshall_dashboard_partial(dashboard) for dashboard in dashboards]
         else:
             message = f'Encountered error: failed to get bookmark for user_id: {user_id}'
             logging.error(message)
             table_bookmarks = []
+            dashboard_bookmarks = []
 
-        return make_response(jsonify({'msg': message, 'bookmarks': table_bookmarks}), status_code)
+        all_bookmarks = {
+            'table': table_bookmarks,
+            'dashboard': dashboard_bookmarks
+        }
+        return make_response(jsonify({'msg': message, 'bookmarks': all_bookmarks}), status_code)
     except Exception as e:
         message = 'Encountered exception: ' + str(e)
         logging.exception(message)
@@ -543,9 +595,42 @@ def get_user_own() -> Response:
         status_code = response.status_code
         owned_tables_raw = response.json().get('table')
         owned_tables = [marshall_table_partial(table) for table in owned_tables_raw]
-        return make_response(jsonify({'msg': 'success', 'own': owned_tables}), status_code)
-
+        dashboards = response.json().get('dashboard', [])
+        owned_dashboards = [marshall_dashboard_partial(dashboard) for dashboard in dashboards]
+        all_owned = {
+            'table': owned_tables,
+            'dashboard': owned_dashboards
+        }
+        return make_response(jsonify({'msg': 'success', 'own': all_owned}), status_code)
     except Exception as e:
         message = 'Encountered exception: ' + str(e)
         logging.exception(message)
         return make_response(jsonify({'msg': message}), HTTPStatus.INTERNAL_SERVER_ERROR)
+
+
+@metadata_blueprint.route('/dashboard', methods=['GET'])
+def get_dashboard_metadata() -> Response:
+    """
+    Call metadata service endpoint to fetch specified dashboard metadata
+    :return:
+    """
+    @action_logging
+    def _get_dashboard_metadata(*, uri: str, index: int, source: str) -> None:
+        pass  # pragma: no cover
+
+    try:
+        uri = get_query_param(request.args, 'uri')
+        index = request.args.get('index', None)
+        source = request.args.get('source', None)
+        _get_dashboard_metadata(uri=uri, index=index, source=source)
+
+        url = f'{app.config["METADATASERVICE_BASE"]}{DASHBOARD_ENDPOINT}/{uri}'
+
+        response = request_metadata(url=url, method=request.method)
+        dashboard = marshall_dashboard_full(response.json())
+        status_code = response.status_code
+        return make_response(jsonify({'msg': 'success', 'dashboard': dashboard}), status_code)
+    except Exception as e:
+        message = 'Encountered exception: ' + str(e)
+        logging.exception(message)
+        return make_response(jsonify({'dashboard': {}, 'msg': message}), HTTPStatus.INTERNAL_SERVER_ERROR)

frontend/amundsen_application/api/preview/dashboard/v0.py

@@ -45,6 +45,7 @@ def get_preview_image(uri: str) -> Response:
                          mimetype='image/jpeg',
                          cache_timeout=app.config['DASHBOARD_PREVIEW_IMAGE_CACHE_MAX_AGE_SECONDS'])
     except FileNotFoundError as fne:
+        LOGGER.exception('FileNotFoundError on get_preview_image')
         return make_response(jsonify({'msg': fne.args[0]}), HTTPStatus.NOT_FOUND)
     except Exception as e:
         LOGGER.exception('Unexpected failure on get_preview_image')

frontend/amundsen_application/api/search/v0.py

@@ -10,6 +10,7 @@
 from flask.blueprints import Blueprint
 
 from amundsen_application.log.action_log import action_logging
+from amundsen_application.api.utils.metadata_utils import marshall_dashboard_partial
 from amundsen_application.api.utils.request_utils import get_query_param, request_search
 from amundsen_application.api.utils.search_utils import generate_query_json, has_filters, \
     map_table_result, transform_filters
@@ -21,6 +22,7 @@
 
 search_blueprint = Blueprint('search', __name__, url_prefix='/api/search/v0')
 
+SEARCH_DASHBOARD_ENDPOINT = '/search_dashboard'
 SEARCH_TABLE_ENDPOINT = '/search'
 SEARCH_TABLE_FILTER_ENDPOINT = '/search_table'
 SEARCH_USER_ENDPOINT = '/search_user'
@@ -119,7 +121,7 @@ def search_user() -> Response:
         page_index = get_query_param(request.args, 'page_index', 'Endpoint takes a "page_index" parameter')
         search_type = request.args.get('search_type')
 
-        results_dict = _search_user(search_term=search_term, page_index=page_index, search_type=search_type)
+        results_dict = _search_user(search_term=search_term, page_index=int(page_index), search_type=search_type)
 
         return make_response(jsonify(results_dict), results_dict.get('status_code', HTTPStatus.INTERNAL_SERVER_ERROR))
     except Exception as e:
@@ -144,7 +146,7 @@ def _map_user_result(result: Dict) -> Dict:
         return user_result
 
     users = {
-        'page_index': int(page_index),
+        'page_index': page_index,
         'results': [],
         'total_results': 0,
     }
@@ -157,9 +159,8 @@ def _map_user_result(result: Dict) -> Dict:
     }
 
     try:
-        url = '{0}?query_term={1}&page_index={2}'.format(app.config['SEARCHSERVICE_BASE'] + SEARCH_USER_ENDPOINT,
-                                                         search_term,
-                                                         page_index)
+        url_base = app.config['SEARCHSERVICE_BASE'] + SEARCH_USER_ENDPOINT
+        url = f'{url_base}?query_term={search_term}&page_index={page_index}'
 
         response = request_search(url=url)
         status_code = response.status_code
@@ -184,6 +185,68 @@ def _map_user_result(result: Dict) -> Dict:
         return results_dict
 
 
-# TODO - Implement
-def _search_dashboard(*, search_term: str, page_index: int, filters: Dict, search_type: str) -> Dict[str, Any]:
-    return {}
+@search_blueprint.route('/dashboard', methods=['GET'])
+def search_dashboard() -> Response:
+    """
+    Parse the request arguments and call the helper method to execute a dashboard search
+    :return: a Response created with the results from the helper method
+    """
+    try:
+        search_term = get_query_param(request.args, 'query', 'Endpoint takes a "query" parameter')
+        page_index = get_query_param(request.args, 'page_index', 'Endpoint takes a "page_index" parameter')
+        search_type = request.args.get('search_type')
+
+        results_dict = _search_dashboard(search_term=search_term, page_index=int(page_index), search_type=search_type)
+
+        return make_response(jsonify(results_dict), results_dict.get('status_code', HTTPStatus.INTERNAL_SERVER_ERROR))
+    except Exception as e:
+        message = 'Encountered exception: ' + str(e)
+        logging.exception(message)
+        return make_response(jsonify(results_dict), HTTPStatus.INTERNAL_SERVER_ERROR)
+
+
+@action_logging
+def _search_dashboard(*, search_term: str, page_index: int, search_type: str) -> Dict[str, Any]:
+    """
+    Call the search service endpoint and return matching results
+    Search service logic defined here:
+    https://github.com/lyft/amundsensearchlibrary/blob/master/search_service/api/dashboard.py
+
+    :return: a json output containing search results array as 'results'
+    """
+    # Default results
+    dashboards = {
+        'page_index': page_index,
+        'results': [],
+        'total_results': 0,
+    }
+
+    results_dict = {
+        'search_term': search_term,
+        'msg': '',
+        'dashboards': dashboards,
+    }
+
+    try:
+        url_base = app.config['SEARCHSERVICE_BASE'] + SEARCH_DASHBOARD_ENDPOINT
+        url = f'{url_base}?query_term={search_term}&page_index={page_index}'
+        response = request_search(url=url)
+
+        status_code = response.status_code
+        if status_code == HTTPStatus.OK:
+            results_dict['msg'] = 'Success'
+            results = response.json().get('results')
+            dashboards['results'] = [marshall_dashboard_partial(result) for result in results]
+            dashboards['total_results'] = response.json().get('total_results')
+        else:
+            message = 'Encountered error: Search request failed'
+            results_dict['msg'] = message
+            logging.error(message)
+
+        results_dict['status_code'] = status_code
+        return results_dict
+    except Exception as e:
+        message = 'Encountered exception: ' + str(e)
+        results_dict['msg'] = message
+        logging.exception(message)
+        return results_dict

frontend/amundsen_application/api/utils/metadata_utils.py

@@ -2,6 +2,7 @@
 
 from typing import Any, Dict, List
 
+from amundsen_common.models.dashboard import DashboardSummary, DashboardSummarySchema
 from amundsen_common.models.popular_table import PopularTable, PopularTableSchema
 from amundsen_common.models.table import Table, TableSchema
 from amundsen_application.models.user import load_user, dump_user
@@ -74,6 +75,37 @@ def marshall_table_full(table_dict: Dict) -> Dict:
     return results
 
 
+def marshall_dashboard_partial(dashboard_dict: Dict) -> Dict:
+    """
+    Forms a short version of dashboard metadata, with selected fields and an added 'key'
+    and 'type'
+    :param dashboard_dict: Dict of partial dashboard metadata
+    :return: partial dashboard Dict
+    """
+    schema = DashboardSummarySchema(strict=True)
+    dashboard: DashboardSummary = schema.load(dashboard_dict).data
+    results = schema.dump(dashboard).data
+    results['type'] = 'dashboard'
+    # TODO: Bookmark logic relies on key, opting to add this here to avoid messy logic in
+    # React app and we have to clean up later.
+    results['key'] = results.get('uri', '')
+    return results
+
+
+def marshall_dashboard_full(dashboard_dict: Dict) -> Dict:
+    """
+    Cleanup some fields in the dashboard response
+    :param dashboard_dict: Dashboard response from metadata service.
+    :return: Dashboard dictionary with sanitized fields, particularly the tables and owners.
+    """
+    # TODO - Cleanup https://github.com/lyft/amundsen/issues/296
+    #  This code will try to supplement some missing data since the data here is incomplete.
+    #  Once the metadata service response provides complete user objects we can remove this.
+    dashboard_dict['owners'] = [_map_user_object_to_schema(owner) for owner in dashboard_dict['owners']]
+    dashboard_dict['tables'] = [marshall_table_partial(table) for table in dashboard_dict['tables']]
+    return dashboard_dict
+
+
 def _update_prog_descriptions(prog_descriptions: List) -> None:
     # We want to make sure there is a display title that is just source
     for desc in prog_descriptions:

frontend/amundsen_application/dashboard_preview/mode_preview.py

@@ -69,4 +69,8 @@ def _get_preview_image_url(self, *, uri: str) -> str:
         if web_preview_image_key not in result:
             raise FileNotFoundError('No preview image available on {}'.format(uri))
 
-        return result[web_preview_image_key]
+        image_url = result[web_preview_image_key]
+        if image_url is None:
+            raise FileNotFoundError('No preview image available on {}'.format(uri))
+
+        return image_url

frontend/amundsen_application/static/css/_icons.scss

@@ -9,8 +9,8 @@ img.icon {
   mask-repeat: no-repeat;
   min-width: 24px;
   width: 24px;
-  -webkit-mask-size: 24px 24px;
-  mask-size: 24px 24px;
+  -webkit-mask-size: contain;
+  mask-size: contain;
 
   &.icon-small {
     height: 16px;
@@ -28,9 +28,6 @@ img.icon {
     background-color: $icon-bg-dark;
   }
 
-  // TODO - Add other icons here
-
-
   &.icon-alert {
     -webkit-mask-image: url('/static/images/icons/Alert-Triangle.svg');
     mask-image: url('/static/images/icons/Alert-Triangle.svg');
@@ -66,6 +63,11 @@ img.icon {
     mask-image: url('/static/images/icons/DataQualityWarning.svg');
   }
 
+  &.icon-dashboard {
+    -webkit-mask-image: url('/static/images/icons/dashboard.svg');
+    mask-image: url('/static/images/icons/dashboard.svg');
+  }
+
   &.icon-down {
     -webkit-mask-image: url('/static/images/icons/Down.svg');
     mask-image: url('/static/images/icons/Down.svg');
@@ -111,6 +113,11 @@ img.icon {
     mask-image: url('/static/images/icons/mail.svg');
   }
 
+  &.icon-mode {
+    content: url('/static/images/icons/logo-mode.svg');
+    background-color: transparent;
+  }
+
   &.icon-plus {
     -webkit-mask-image: url('/static/images/icons/plus.svg');
     mask-image: url('/static/images/icons/plus.svg');

frontend/amundsen_application/static/css/_labels.scss

@@ -5,6 +5,11 @@
   background-color: $badge-danger-color;
 }
 
+.label-default {
+  color: $badge-text-color;
+  background-color: $badge-default-color;
+}
+
 .label-primary {
   color: $badge-text-color;
   background-color: $badge-primary-color;