-
Notifications
You must be signed in to change notification settings - Fork 0
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Automated Download Speed Testing for download.kiwix.org #1
Comments
Thank you for your work!
|
That's interesting. Maybe we could run speedtest first so we'd have reference, general data to compare with.
No. It will need to be packaged as a docker image.
Let me copy-paste a comment I sent to someone else before opning this ticket
Doesn't matter much at this point. |
Thanks for your reply! It’s very enlightening. |
I have some other questions I'd like to ask. Do we really need to use the VPN technology, like OpenVPN and WireGuard? If the download speed is the only thing we want to collect, I think using proxy, like socks5, is a more reasonable choice. Using proxy makes it easier for the software to change "location". |
Summary Is that the case? |
Hello @rgaudin by looking at the project idea, it seemed interesting to me. I have done Full-stack and networking projects so I am comfortable with the tech stack. I have some questions about this project.
|
For this question, I just check the comment under the another project, I think |
@FledgeXu thanks for the quick reply 😊 |
Hi @rgaudin my question is: |
Hello, @rgaudin Also take a look at this https://geonode.com/free-proxy-list they provide free proxies with different protocols and there are so many filter options for us. |
Hello @rgaudin From my understanding, I believe that I need to follow these steps to implement this project:
On looking at the project idea and the required technical stack, I am confident that I will be able to implement this. And as per above discussion, I have previously worked extensively on Python, Django, Docker and React. |
All, thank you for your interest in the project. Allow me to catch-up with the discussion by responding to all questions in a single comment
Yes, we want to use VPN and more specifically Wireguard because:
Yes
Somewhat. Not sure at this point if it will be multiple instances running in parallel or a single one sequentially switching location but that's the idea.
Flask or FastAPI either serving frontend or complemented by Vue.js
Ideally DB-agnostic (using an ORM). We tend to deploy production on PostgreSQL and develop with SQLite. We're moving away from MongoDB which is difficult to scale (all stored in RAM)
No. While it would provide more realistic data, we just can't support this. It would require a larger infrastructure, more complex code that would not be centralized and require managing the community of reporters. We thus decided on the lesser-value VPN option
See answer above.
Yes but there won't be testing machines in each location. That's the point of the project, a single machine (in the cloud) will connect (using wireguard) to various locations using the VPN provider's list and perfom the tests: speedtest, download from kiwix, and insert this data into the DB
Yes, see above for specific frameworks. |
Hello @rgaudin I want to fix some issues of Kiwix. Since there are many kiwix repositories, can you suggest me repo's on whose issues I can work on ? |
No. it's part of the process. |
So that means I can fix the issues on any repo ? |
@rgaudin ? |
@Sarthak-Biswas this is not an interactive chat and that has already been replied. Read previous comments |
I am an extreme beginner in open source contribution, so are there any issues I can contribute? |
hey there, I want to contribute to this project and would like to know a bit more about this project. |
@abhijeet10052002 this is a GSoC ticket. GSoc rules applies |
Can I get an update on this project? How much of the objectives were completed/what objectives are yet to be completed? |
We did not affect any GSoC student for this project in 2023. |
Why can't we use a server less infrastructure like Aws-lambda? Assuming the speed tests happen a few times a day (once every 30 minutes or so), this option seems much simpler and cheaper compared to using dedicated VPN servers. |
Here's my understanding of the project scope. Please let me know if I'm on the right track or if misunderstood something: We have a central machine that will most likely store: the database and a docker image. This Docker setup will connect to the Kiwix's mirrors via VPN, run tests, and save the data in the database. We'll also create a frontend that connects to the database using either FastAPI or Flask. A couple of tech clarifications: Are we still using PostgreSQL for deployment and SQLite for development? Are the any preferred ORMs in the organization? And is WireGuard still our go-to VPN? |
Feel free to propose something that's simpler and cheaper, as long as the mandatory features are present: download speed tests from different locations around the globe. |
There are multiple ways to orchestrate this but indeed, in the end, there's a container running, connecting to a VPN for a specified location, downloading stuff from Kiwix mirror then inserting this to the central DB (probably via an API so we could easily add additional contributors)
Yes
Yes (PG is fine for both as well), SQLAlchemy, and yes but feel free to propose a better alternative (with arguments!) |
@rgaudin
Correct me if I am wrong. I just tried using wireguard cli and an upvote for choosing that. Few suggestions: for the webinterface part, what options should it contain ? Basically to get the download speed of a particular location. |
You would still need VPN servers. I don't think AWS Lambda functions natively support VPN connections, you would need to set up a VPN connection at the VPC level, or use a third-party service (wireguard for example) that provides an API for establishing VPN connections. |
@Vishnurr2k01 AFAIK speedtest-cli is for testing one's internet connection (using speedtest-partner servers). |
you are right, so we can compute the throughput normally by dividing the file size by the time taken for download. Is that okay ? |
Yes but that sounds like a detail at this stage |
I would like to work on this as part of Gsoc. Is there anything more I should look into before writing a proposal for this. |
That's all we have at the moment |
As far as I understood the issue, we have to create a solution/tool for testing the download speed in the various regions provided by kiwix mirrors using vpn to take a look into the status of the server and improve/fix if there is any issue. Step 1: Initial Setup and Environment Preparation
Step 2: Python Script Development
Step 3: Database Integration
Step 4: Web Interface Development
Step 5: Documentation and Dockerization
Am I missing anything in above steps, I'm open to any specific preferences or modifications to these steps. |
Looks good except we package with docker early to ease testing by reviewers and gather actual data. |
Hey @rgaudin, thanks for the quick response! Right I agree that dockerizing the image early on will centralize everything and streamline the process. |
Hello, @rgaudin I am new here and I want to contribute to Quix's project. Can you tell me how to start and what are the necessary things? |
@Sarthak-Biswas good idea but do you have any algo? |
@andy0712dik yes the diagram is correct. There might be some additional information collected. For instance, we want to test the download.kiwix.org load-balancer and record which mirror it is recommending us. Because users don't pick mirrors manually.
|
Can you please suggest me what will the technology I should prepare for contributing this project |
hello @rgaudin I would like to contribute to this project I have already created a python script while will store the avg. download speed if the link of the downloading file is provided Am I on the right track? |
@Royalty252 I believe all the information you need is in this ticket already. |
@omthakare16 are you working on this? |
hello @rgaudin as much i researched to this project here are some steps i think we should follow these steps to complete this project Automated download speed testing for kwix.org:
|
@andy0712dik this ticket is a place to ask for clarifications on the project so that everyone can benefit from answers. Your plan is supposed to go into your proposal |
@rgaudin can you please tell me if I'm on right track or not |
@rgaudin hey if I want to send the proposal where can I send it? |
Google platform. We can see them all and you can still edit until the deadline. We are not notified though so a mention on Slack or somewhere that you sent your proposal is welcome |
@rgaudin , I would like one minor clarification regarding the approach suggested in this image. Does it mean that when we want to check the download speed of file, we should download from all the mirrors for that file in each instance? For example, if we are downloading from Brazil and there are six mirrors when we append Also, while downloading, can we randomly choose any of the files in any of the directories to download? |
What's pertinent to download, from where, at what time is yet to be defined. We'll have to discuss what information we want to get and what input is required for it to be computed. Feel free to make suggestions in your proposal.
Yes and no. Not all mirrors mirror everything so you might pick a file with a single mirror if it's completely random but yeah, that's the idea. |
@rgaudin i have submitted my proposal what is the next thing i should work on |
We'll be reviewing the proposals over the next 3 weeks ; everyone is welcome to contribute to any repo at any time. |
Is someone selected for this as part of gsoc? If not, shall I do it ouside of gsoc? |
Yes, we got a slot and a developer ; it's happening. |
This ticket tracks the Kiwix GSoC
20232024 Project Automated Download Speed Testing for download.kiwix.org until code contributions and/or specific tickets requires creating its own dedicated repository.Candidates, contributors, this ticket is the preferred location to discuss this project. Ask your questions here and mentors (@rgaudin, @kelson42) shall respond.
Mandatory reads:
Automated Download Speed Testing for download.kiwix.org
Objective: create a solution that automatically tests the download speed of download.kiwix.org from various locations on Earth. This will help Kiwix to ensure fast and reliable downloads for all users and improve the overall user experience, as 80% of our users are located in the Global South. By automating the testing process, Kiwix can save time and resources (we’re a non-profit, after all).
Technologies: Python, VPN, datacenters
Description:
Kiwix provides offline access to Wikipedia and other educational content to millions of users worldwide. Ensuring fast and reliable downloads from download.kiwix.org is critical to delivering a good user experience. However, it is challenging to accurately test download speeds from different locations on earth due to network latency and other factors.
The objective of this project is to create an automated solution to test the download speed of download.kiwix.org from various locations on earth using VPN or datacenters. The solution will use python to run the tests and record the results. The tests will be run at regular intervals and the results will be stored in a database. This will allow Kiwix to monitor download speeds over time and identify any issues that may be affecting performance.
Key Deliverables:
Skills required:
Difficulty: Easy, but probably a longer (350 hours) project
The text was updated successfully, but these errors were encountered: