Automated Download Speed Testing for download.kiwix.org

[rgaudin]

This ticket tracks the Kiwix GSoC 20232024 Project Automated Download Speed Testing for download.kiwix.org until code contributions and/or specific tickets requires creating its own dedicated repository.

Candidates, contributors, this ticket is the preferred location to discuss this project. Ask your questions here and mentors (@rgaudin, @kelson42) shall respond.

Mandatory reads:

• https://github.com/kiwix/overview/blob/main/CONTRIBUTING.md
• https://github.com/kiwix/overview/blob/main/CODE_OF_CONDUCT.md

---

Automated Download Speed Testing for download.kiwix.org

Objective: create a solution that automatically tests the download speed of download.kiwix.org from various locations on Earth. This will help Kiwix to ensure fast and reliable downloads for all users and improve the overall user experience, as 80% of our users are located in the Global South. By automating the testing process, Kiwix can save time and resources (we’re a non-profit, after all).

Technologies: Python, VPN, datacenters

Description:

Kiwix provides offline access to Wikipedia and other educational content to millions of users worldwide. Ensuring fast and reliable downloads from download.kiwix.org is critical to delivering a good user experience. However, it is challenging to accurately test download speeds from different locations on earth due to network latency and other factors.

The objective of this project is to create an automated solution to test the download speed of download.kiwix.org from various locations on earth using VPN or datacenters. The solution will use python to run the tests and record the results. The tests will be run at regular intervals and the results will be stored in a database. This will allow Kiwix to monitor download speeds over time and identify any issues that may be affecting performance.

Key Deliverables:

• A python script that tests download speed from various locations on earth.
• A database to store the test results.
• A web-based interface to view the results.
• Documentation for how to set up and use the solution.

Skills required:

• Strong knowledge of python programming language
• Familiarity with network testing and performance analysis.
• Experience with VPNs and datacenters.
• Knowledge of databases and web development is a plus.

Difficulty: Easy, but probably a longer (350 hours) project

[FledgeXu]

Thank you for your work!
I would like to ask a few questions:

1. What data should be collected? Besides the download speed of a certain file, do we need to include some other data? For example, DNS resolution records (used to analyze if there is network congestion), ping records (used to analyze latency and connectivity), and traceroute records (used to optimize the download path afterwards).
2. Does it need to run on a cluster?
3. Does it need to be familiar with VPNs and datacenters, what exactly does that mean? Does it mean that you need to connect to a datacenter around the world with a VPN to test the download speed?
4. What specific files need to be tested for speed?

[rgaudin]

• What data should be collected? Besides the download speed of a certain file, do we need to include some other data? For example, DNS resolution records (used to analyze if there is network congestion), ping records (used to analyze latency and connectivity), and traceroute records (used to optimize the download path afterwards).

That's interesting. Maybe we could run speedtest first so we'd have reference, general data to compare with.

• Does it need to run on a cluster?

No. It will need to be packaged as a docker image.

• Does it need to be familiar with VPNs and datacenters, what exactly does that mean? Does it mean that you need to connect to a datacenter around the world with a VPN to test the download speed?

Let me copy-paste a comment I sent to someone else before opning this ticket

Yes that’s the idea. download.kiwix.org is actually not serving up files but delivers redirections to those files on mirrors. You can append .mirrorlist to any file on the server (ex: https://download.kiwix.org/zim/wikipedia/wikipedia_fr_test_2023-01.zim.mirrorlist) and you’ll get the list of links to the various mirrors holding that particular file.
When you download a file, it just picks one mirror based on its algorithm: priority and location for instance. btw this is all handled by the mirrorbrain tool.
Now those priorities on the mirrors, we had set them arbitrarily .
Also, we have no idea how mirrors performs or what download speed users from around the world can expect.
So the idea of this project is to build something that would download from various locations accross the globe and report metrics so that:
we know what to generally expect from each mirror
we can adjust the priorities on our mirrors list
we know what locations are underserved (where we should look for a new mirror)
we know when a mirror is misbehaving/overloaded
This tool is not critical to our infrastructure so it should be as lightweight to setup/operate/maintain as possible. That’s why we suggested using VPN services so we could download from different locations yet, from a single server.
We understand there are limitations to this approach: we’d be downloading from a VPN service’s datacenter so that would not reflect end-user’s experience but that should at least allow us to compare location-based access

• What specific files need to be tested for speed?

Doesn't matter much at this point.

[FledgeXu]

Thanks for your reply! It’s very enlightening.

[FledgeXu]

I have some other questions I'd like to ask. Do we really need to use the VPN technology, like OpenVPN and WireGuard? If the download speed is the only thing we want to collect, I think using proxy, like socks5, is a more reasonable choice. Using proxy makes it easier for the software to change "location".

[santhoshtk01]

Summary
So, In our kiwix software, we need a feature that automatically tests the downloading speed of download.kiwix.org around different locations in the world using a VPN service at a reasonable regular interval. Then we need to create a database to store the data and we have to create a web-based interface to visualize the downloading speed at different locations. Finally, we have to pack it as a docker image.

Is that the case?

[Sarthak-Biswas]

Hello @rgaudin by looking at the project idea, it seemed interesting to me. I have done Full-stack and networking projects so I am comfortable with the tech stack.

I have some questions about this project.

1. Since on VPN the speedtest will test speed between client and server but not between server and internet ( here the kiwix mirrors ). So can I use the model in the diagram depicted below ?
2. For making the web-based interface are we allowed to use React/Nextjs/Angular ?
3. Are there any restriction of database ? or can we use any . For example : MySQL, Mongodb, PostgreSQL etc.

[FledgeXu]

@Sarthak-Biswas

For making the web-based interface are we allowed to use React/Nextjs/Angular ?

Somewhat yes, because we'll have to maintain what will be built and we tend to limit the number of technologies we need across our projects. So most likely Python via Flask or FastAPI and the frontend using Vue.js
kiwix/overview#77 (comment)

For this question, I just check the comment under the another project, I think Flask or FastAPI and Vue.js are good choices.

[Sarthak-Biswas]

@FledgeXu thanks for the quick reply 😊

[Sarthak-Biswas]

Hi @rgaudin my question is:
Will the python script be running on machines of every individual user of kiwix ? Or it will be running on an independent server ?

[santhoshtk01]

Hello, @rgaudin
Could you please review the attachment and given feedback on the plan ?

Instead of using VPN can we use free proxy services https://free-proxy-list.net/ like this one. They have a bunch of IP from different locations and updates every 10 minutes. We can scrape through their website to get the updated list and then we can use the valid proxies to test the download speed from various location around the world.

Also take a look at this https://geonode.com/free-proxy-list they provide free proxies with different protocols and there are so many filter options for us.

[Devanshu-17]

Hello @rgaudin

From my understanding, I believe that I need to follow these steps to implement this project:

1. Identify the locations around the world where we want to test the download speed. We can use VPN services or Data Centers located in those regions for testing purposes.

2. Install Python on the testing machines in each location. We can use popular Python libraries like Speedtest-cli, Ping, and Requests to perform download speed tests.

3. Create a Python script that runs the download speed tests and records the results. The script should run at regular intervals, and the results should be stored in a database for further analysis.

4. Choose a suitable database to store the test results. We can use a relational database like MySQL, PostgreSQL, SQLite or use a NoSQL database like MongoDB or CouchDB.

5. Develop a web-based interface to view the test results. We can use popular web development frameworks like Django, Flask, or Pyramid to create the interface. The interface should provide detailed information about the download speeds, test locations, and time of the tests.

On looking at the project idea and the required technical stack, I am confident that I will be able to implement this.

And as per above discussion, I have previously worked extensively on Python, Django, Docker and React.

[rgaudin]

All, thank you for your interest in the project. Allow me to catch-up with the discussion by responding to all questions in a single comment

@FledgeXu

I have some other questions I'd like to ask. Do we really need to use the VPN technology, like OpenVPN and WireGuard? If the download speed is the only thing we want to collect, I think using proxy, like socks5, is a more reasonable choice. Using proxy makes it easier for the software to change "location".

Yes, we want to use VPN and more specifically Wireguard because:

• It's a network tool (layer 3) meaning the software doesn't know or care about the connection.
• It's modern.
• Plenty of paid-for services are offering it and it's increasing. On other hand, proxies are declining.
• We can easily pick a VPN service provider and have reasonnable expectation about quality and speed. Harder with proxies.

---

@santhoshtk01

So, In our kiwix software, we need a feature that automatically tests the downloading speed of download.kiwix.org around different locations in the world using a VPN service at a reasonable regular interval. Then we need to create a database to store the data and we have to create a web-based interface to visualize the downloading speed at different locations. Finally, we have to pack it as a docker image.
Is that the case?

Yes

---

@Sarthak-Biswas

Since on VPN the speedtest will test speed between client and server but not between server and internet ( here the kiwix mirrors ). So can I use the model in the diagram depicted below ?

Somewhat. Not sure at this point if it will be multiple instances running in parallel or a single one sequentially switching location but that's the idea.

For making the web-based interface are we allowed to use React/Nextjs/Angular ?

Flask or FastAPI either serving frontend or complemented by Vue.js

Are there any restriction of database ? or can we use any . For example : MySQL, Mongodb, PostgreSQL etc.

Ideally DB-agnostic (using an ORM). We tend to deploy production on PostgreSQL and develop with SQLite. We're moving away from MongoDB which is difficult to scale (all stored in RAM)

---

@Sarthak-Biswas

Will the python script be running on machines of every individual user of kiwix ? Or it will be running on an independent server ?

No. While it would provide more realistic data, we just can't support this. It would require a larger infrastructure, more complex code that would not be centralized and require managing the community of reporters. We thus decided on the lesser-value VPN option

---

@santhoshtk01

Instead of using VPN can we use free proxy services https://free-proxy-list.net/ like this one. They have a bunch of IP from different locations and updates every 10 minutes. We can scrape through their website to get the updated list and then we can use the valid proxies to test the download speed from various location around the world.
Also take a look at this https://geonode.com/free-proxy-list they provide free proxies with different protocols and there are so many filter options for us.

See answer above.

---

@Devanshu-17

Identify the locations around the world where we want to test the download speed. We can use VPN services or Data Centers located in those regions for testing purposes.
Install Python on the testing machines in each location. We can use popular Python libraries like Speedtest-cli, Ping, and Requests to perform download speed tests.
Create a Python script that runs the download speed tests and records the results. The script should run at regular intervals, and the results should be stored in a database for further analysis.
Choose a suitable database to store the test results. We can use a relational database like MySQL, PostgreSQL, SQLite or use a NoSQL database like MongoDB or CouchDB.

Yes but there won't be testing machines in each location. That's the point of the project, a single machine (in the cloud) will connect (using wireguard) to various locations using the VPN provider's list and perfom the tests: speedtest, download from kiwix, and insert this data into the DB

Develop a web-based interface to view the test results. We can use popular web development frameworks like Django, Flask, or Pyramid to create the interface. The interface should provide detailed information about the download speeds, test locations, and time of the tests.

Yes, see above for specific frameworks.

[Sarthak-Biswas]

Hello @rgaudin I want to fix some issues of Kiwix. Since there are many kiwix repositories, can you suggest me repo's on whose issues I can work on ?

[rgaudin]

Hello @rgaudin I want to fix some issues of Kiwix. Since there are many kiwix repositories, can you suggest me repo's on whose issues I can work on ?

No. it's part of the process.

[Sarthak-Biswas]

So that means I can fix the issues on any repo ?

[Sarthak-Biswas]

So that means I can fix the issues on any repo ?

@rgaudin ?

[rgaudin]

@Sarthak-Biswas this is not an interactive chat and that has already been replied. Read previous comments

[mini-M-py]

I am an extreme beginner in open source contribution, so are there any issues I can contribute?

[HimanshuMahto]

hey there, I want to contribute to this project and would like to know a bit more about this project.

[rgaudin]

@abhijeet10052002 this is a GSoC ticket. GSoc rules applies

[jairajmahadev]

Can I get an update on this project? How much of the objectives were completed/what objectives are yet to be completed?

[rgaudin]

Can I get an update on this project? How much of the objectives were completed/what objectives are yet to be completed?

We did not affect any GSoC student for this project in 2023.

[Greeshmanth1909]

Why can't we use a server less infrastructure like Aws-lambda? Assuming the speed tests happen a few times a day (once every 30 minutes or so), this option seems much simpler and cheaper compared to using dedicated VPN servers.

[ImperaLuna]

Here's my understanding of the project scope. Please let me know if I'm on the right track or if misunderstood something:

We have a central machine that will most likely store: the database and a docker image.
Inside the docker image we will have the testing script, some kind of automation and the vpn configuration.

This Docker setup will connect to the Kiwix's mirrors via VPN, run tests, and save the data in the database.

We'll also create a frontend that connects to the database using either FastAPI or Flask.

A couple of tech clarifications: Are we still using PostgreSQL for deployment and SQLite for development? Are the any preferred ORMs in the organization? And is WireGuard still our go-to VPN?

[rgaudin]

Why can't we use a server less infrastructure like Aws-lambda? Assuming the speed tests happen a few times a day (once every 30 minutes or so), this option seems much simpler and cheaper compared to using dedicated VPN servers.

Feel free to propose something that's simpler and cheaper, as long as the mandatory features are present: download speed tests from different locations around the globe.
If you propose something significantly different, make sure to include all the details we'd need to make a choice –like actual price comparison–, etc

[rgaudin]

We have a central machine that will most likely store: the database and a docker image. Inside the docker image we will have the testing script, some kind of automation and the vpn configuration.

This Docker setup will connect to the Kiwix's mirrors via VPN, run tests, and save the data in the database.

There are multiple ways to orchestrate this but indeed, in the end, there's a container running, connecting to a VPN for a specified location, downloading stuff from Kiwix mirror then inserting this to the central DB (probably via an API so we could easily add additional contributors)

We'll also create a frontend that connects to the database using either FastAPI or Flask.

Yes

A couple of tech clarifications: Are we still using PostgreSQL for deployment and SQLite for development? Are the any preferred ORMs in the organization? And is WireGuard still our go-to VPN?

Yes (PG is fine for both as well), SQLAlchemy, and yes but feel free to propose a better alternative (with arguments!)

[Vishnurr2k01]

@rgaudin
Here is my basic understanding and idea for implementation.

• Setup docker container with linux environment
• Install wireguard packages and speedtest-cli
• Get the range of ip addresses organized by country using any of the following [ IP2Location database ( paid ) , ipshu , ipinfo.io]
• Create configuration files for these
• Find the download speed from kiwix from these servers , over a period of time
• Populate these values in db using fast api in any RDB
• Implement web interface to view this result

Correct me if I am wrong.

I just tried using wireguard cli and an upvote for choosing that.

Few suggestions:
I think we should test more than once per region and calculate the average.

for the webinterface part, what options should it contain ? Basically to get the download speed of a particular location.
should we add more feature like getting history of download speed, comparison ?

[jairajmahadev]

Why can't we use a server less infrastructure like Aws-lambda? Assuming the speed tests happen a few times a day (once every 30 minutes or so), this option seems much simpler and cheaper compared to using dedicated VPN servers.

You would still need VPN servers. I don't think AWS Lambda functions natively support VPN connections, you would need to set up a VPN connection at the VPC level, or use a third-party service (wireguard for example) that provides an API for establishing VPN connections.

[rgaudin]

@Vishnurr2k01 AFAIK speedtest-cli is for testing one's internet connection (using speedtest-partner servers).
That's an interesting additional value but what's most important is download-speed from various locations to Kiwix server targeting random files (smalls, larger) so we get a sense of experience of actual users in a given region.

[Vishnurr2k01]

@Vishnurr2k01 AFAIK speedtest-cli is for testing one's internet connection (using speedtest-partner servers). That's an interesting additional value but what's most important is download-speed from various locations to Kiwix server targeting random files (smalls, larger) so we get a sense of experience of actual users in a given region.

you are right, so we can compute the throughput normally by dividing the file size by the time taken for download. Is that okay ?

[rgaudin]

you are right, so we can compute the throughput normally by dividing the file size by the time taken for download. Is that okay ?

Yes but that sounds like a detail at this stage

[Vishnurr2k01]

you are right, so we can compute the throughput normally by dividing the file size by the time taken for download. Is that okay ?

Yes but that sounds like a detail at this stage

I would like to work on this as part of Gsoc. Is there anything more I should look into before writing a proposal for this.

[rgaudin]

That's all we have at the moment

[omthakare16]

@rgaudin @kelson42 ,

As far as I understood the issue, we have to create a solution/tool for testing the download speed in the various regions provided by kiwix mirrors using vpn to take a look into the status of the server and improve/fix if there is any issue.
I want to express my interest in this project for Automated Download Speed Testing! Below, I outline my proposed steps to tackle this project:

Step 1: Initial Setup and Environment Preparation

• Set up a development environment with necessary tools like Python and VPN configurations.
• Familiarize with Wireguard for VPN connectivity.

Step 2: Python Script Development

• Implement functionality to connect to various VPN locations using Python library Wireguard.
• Utilize Python libraries like Requests to conduct download speed tests directly from Kiwix servers by downloading some files.
• Design the script to run tests at regular intervals and record results.

Step 3: Database Integration

• Select an appropriate database system, considering the project's requirements and preferences (ideally PostgreSQL for production and SQLite for development).
• Develop database schemas for storing test results efficiently.
• Implement database connectivity within the Python script to store test data.

Step 4: Web Interface Development

• the web framework based on the provided suggestions (Flask or FastAPI complemented by Vue.js for frontend).
• Develop a web interface to visualize test results.
• Incorporate features to display detailed information about download speeds, test locations, and timestamps.

Step 5: Documentation and Dockerization

• Create comprehensive documentation outlining the setup process, usage instructions, and project architecture.
• Package the entire solution into a Docker image for easier deployment and scalability.

Am I missing anything in above steps, I'm open to any specific preferences or modifications to these steps.

[rgaudin]

Looks good except we package with docker early to ease testing by reviewers and gather actual data.

[omthakare16]

Looks good except we package with docker early to ease testing by reviewers and gather actual data.

Hey @rgaudin, thanks for the quick response! Right I agree that dockerizing the image early on will centralize everything and streamline the process.

[andy0712dik]

Hello, @rgaudin I am new here and I want to contribute to Quix's project. Can you tell me how to start and what are the necessary things?

[andy0712dik]

Hello @rgaudin by looking at the project idea, it seemed interesting to me. I have done Full-stack and networking projects so I am comfortable with the tech stack.

I have some questions about this project.

1. Since on VPN the speedtest will test speed between client and server but not between server and internet ( here the kiwix mirrors ). So can I use the model in the diagram depicted below ?
2. For making the web-based interface are we allowed to use React/Nextjs/Angular ?
3. Are there any restriction of database ? or can we use any . For example : MySQL, Mongodb, PostgreSQL etc.

@Sarthak-Biswas good idea but do you have any algo?

[rgaudin]

@andy0712dik yes the diagram is correct. There might be some additional information collected. For instance, we want to test the download.kiwix.org load-balancer and record which mirror it is recommending us. Because users don't pick mirrors manually.

2. & 3. Vue.js and PostgreSQL or SQLite see https://github.com/openzim/overview/wiki/openZIM-code-contribution-workflow#frontend

[andy0712dik]

Can you please suggest me what will the technology I should prepare for contributing this project

[Avadhutgiri]

hello @rgaudin I would like to contribute to this project
as per I understood here are some important steps
1.Identify Test Location
2.Install Python and Necessary Libraries
3.Python Script Development
4.Choosing a Database
5.Develop Web-Based Interface

I have already created a python script while will store the avg. download speed if the link of the downloading file is provided
and we can use vpn to change the location

Am I on the right track?
can You please Suggest me what should I do more ?
Thank you

[rgaudin]

@Royalty252 I believe all the information you need is in this ticket already.

[anuragrao04]

@omthakare16 are you working on this?

[andy0712dik]

hello @rgaudin as much i researched to this project here are some steps i think we should follow these steps to complete this project

Automated download speed testing for kwix.org:

1. Test location selection: Optimize for global coverage and network diversity.
2. Python environment setup: Install requisite libraries, leveraging tools like pipenv or virtualenv.
3. Python script creation: Develop a script using speedtest-cli and requests for automated speed testing.
4. Database selection: Choose a scalable solution like MySQL or PostgreSQL for efficient storage and retrieval of test data.
5. Web interface development: Implement a user-friendly interface using frameworks such as Flask or Django for seamless interaction and result visualization.

[rgaudin]

@andy0712dik this ticket is a place to ask for clarifications on the project so that everyone can benefit from answers. Your plan is supposed to go into your proposal

[andy0712dik]

@rgaudin can you please tell me if I'm on right track or not

[Avadhutgiri]

@rgaudin hey if I want to send the proposal where can I send it?

[rgaudin]

Google platform. We can see them all and you can still edit until the deadline. We are not notified though so a mention on Slack or somewhere that you sent your proposal is welcome

[elfkuzco]

@rgaudin , I would like one minor clarification regarding the approach suggested in this image. Does it mean that when we want to check the download speed of file, we should download from all the mirrors for that file in each instance? For example, if we are downloading from Brazil and there are six mirrors when we append .mirrorslist to the name of the file, the instance running with the configuration for Brazil should download from all six mirrors?

Also, while downloading, can we randomly choose any of the files in any of the directories to download?

[rgaudin]

What's pertinent to download, from where, at what time is yet to be defined. We'll have to discuss what information we want to get and what input is required for it to be computed. Feel free to make suggestions in your proposal.

Also, while downloading, can we randomly choose any of the files in any of the directories to download?

Yes and no. Not all mirrors mirror everything so you might pick a file with a single mirror if it's completely random but yeah, that's the idea.

[andy0712dik]

@rgaudin i have submitted my proposal what is the next thing i should work on

[rgaudin]

We'll be reviewing the proposals over the next 3 weeks ; everyone is welcome to contribute to any repo at any time.

[Vishnurr2k01]

Is someone selected for this as part of gsoc? If not, shall I do it ouside of gsoc?

[rgaudin]

Yes, we got a slot and a developer ; it's happening.