Analytics Dashboard for EvalAI Admin, Challenge hosts & Participants

[Sanji515]

Project Title: Analytics Dashboard for EvalAI Admin, Challenge hosts & Participants

Description:

As the number of compute-intensive challenges on EvalAI are increasing, we want to focus on improving the performance of our services. As a first step, we will focus on monitoring and measuring all key metrics of our services. Insights from these will allow us to efficiently utilize our infrastructure, improve uptime and reduce costs. The project will concentrate on writing REST API’s, plotting pretty graphs and building analytics dashboards to cater for all three types of users on EvalAI i.e. admin, challenge hosts and participants. The analytics will help challenge hosts to view the progress of participants in their challenge for instance, comparing the trends of the accuracy from participant submissions over the period of time, etc. The final goal is to provide users with several analytics so as to display their progress on the platform and utilize the resources efficiently to reduce costs.

Deliverable:

1. EvalAI Admin -

   • Add backend APIs & graphs on UI for number of daily, monthly, & yearly signups of users, number of challenges on EvalAI & number of submissions on EvalAI.
   • Integration of the backend with open-source softwares like Grafana so to report metrics for the load on APIs
   • Integrate the backend with the AWS cloudwatch to get the metrics for server load, CPU utilization, storage, etc.
   • Write tests for the APIs and frontend

2. Challenge Host -

   • Add backend APIs and UI graphs for the performance of a participant in the challenge over a period of time.
   • Add a plot for showing the number of submissions in a challenge by a participant.
   • Add plots for showing the number of submissions in a challenge in a day, month, and yearly.
   • Add a plot to show the ranks of various participants on the leaderboard over time.
   • Write tests for the APIs and frontend

3. Participants -

   • Add backend APIs and graphs for submissions in a challenge and the corresponding rank.
   • Add a plot for the number of submissions in a challenge over time.
   • Write tests for the APIs and frontend

Mentor: Shekhar Prasad Rajak @Shekharrajak, Rishabh Jain @RishabhJain2018

Skills: Angular 7, Django, Django Rest Framework, D3.js

Skill Level: Medium

Get started: Try to fix some issues in EvalAI (note that there are some issues labeled with GSOC-2020)

Tutorials:

• https://angular.io/tutorial
• https://www.django-rest-framework.org/tutorial/quickstart/
• https://www.tutorialspoint.com/angular7/index.htm

Important Links:

• EvalAI Website: evalai.cloudcv.org
• EvalAI Github repository: Cloud-CV/EvalAI
• EvalAI-ngx Github repository: Cloud-CV/EvalAI-ngx
• EvalAI Docs: http://evalai.readthedocs.io/en/latest
• GSoC Proposal Template: Cloud-CV/GSoC-Ideas/wiki/GSOC-2020-Proposal-Template
• Gitter Channel: gitter.im/Cloud-CV
• Mailing list: groups.google.com/forum/#!forum/cloudcv

[Shekharrajak]

Points that can be discussed in front end side (can be added in idea list: #28 as per this enhancement ):

• If I am correct then currently we have the roles (users, host users, admin), so increasing number of roles come with security issues and the application should be capable enough to authenticate each user, get role for the user and authorize to show the features/page.

First, we can implement Authentication guard, generalizing this issue: Cloud-CV/EvalAI-ngx#271

Then we can have authorization guard which will check the authenticated user role, then the route can be activated. Refer canActivate.

Extending this idea, we should use various router guards as per the requirement:

CanActivate, CanActivateChild, CanDeactivate, CanLoad

It will improve the performance because we don't have to check the roles (reducing number of if , else conditions in template and services).

• Group discussion or good proposal on which visualization tool/library to use. I see that Google Charts , HighCharts have Angular wrapper and they all can generate almost all kind of charts with many customizations. But maybe Angular Materialize itself is good enough for this requirement which is better than using 3rd dependency.

[Aakash1822]

Interested to work on this project. Please guide me where to start

[Shekharrajak]

@Aakash1822 , Go through the contribution & development guidelines. To understand the codebase I would suggest you to start fixing small easy to fix bugs in issue list then continue with task list.

[Shekharrajak]

Hello, Students!

I am continuously getting messages in different platforms(linkedin/gitter, etc) about the application review or discussion. It would be better if student create a mail thread in google group: https://groups.google.com/forum/#!forum/cloudcv and cc all the mentors. It will help to know everyone about the idea, proposal and we can easily correct you or provide the right direction to your approach.

Keep committing! :)

[Shekharrajak]

@adityaketkar, Since I am not able to comment on your PDF application, I am pasting the comments here:

• You can write some sample definition along with test method signature.

• Is there only those APIs or there is more?

• For which role which dashboard will be shown? Admin, candidate, challenge host user ?

• Improve the overall task list - you can provide your ideas

• Your any Open PRs or any demo work for your approach could be added

• Your plan after GSoC period

• Your plan in summer , any other commitment

• Your timeline for each week can be improved

[AkashDPA]

Hello members,
I can't able to reach at mailing list so i am writing here.
I have some experience with API and also have submitted research paper to optimize it.
I want to contribute for Eval Admin, currently doing installation but I have some question regarding feature and technology stack, can anyone guide me?

[Shekharrajak]

Hi @AkashDPA , Feel free to ask questions and let us know your research on the idea.

[AkashDPA]

Research is about optimizing API signature validation, regular API uses the timestamp method to generate a unique signature and to validate it takes more server resources. for this, I have come up with a solution.

[AkashDPA]

Can I have docker image, I am getting following error each time, may be because heavy load (due to lockdown) but it lasts 2 days

root@kali:/evalai# docker-compose up --build
Pulling sqs (pakohan/elasticmq:)...
latest: Pulling from pakohan/elasticmq
Image docker.io/pakohan/elasticmq:latest uses outdated schema1 manifest format. Please upgrade to a schema2 image for better future compatibility. More information at https://docs.docker.com/registry/spec/deprecated-schema-v1/
a3ed95caeb02: Pull complete
798775e4391c: Downloading
3585d15c3f22: Download complete
aa076096cff1: Download complete
5d67e39adf5b: Downloading
b1eb7b91e407: Download complete
4a69946c8688: Download complete
f9c265c6cb17: Download complete
cd3c13c60972: Download complete
e7b3ea95788c: Download complete
ERROR: dial tcp 3.224.11.4:443: i/o timeout