Skip to content
This repository has been archived by the owner on Jun 14, 2020. It is now read-only.

Be able to absorb 100s of new users at once #143

Open
lourot opened this issue Sep 11, 2018 · 19 comments
Open

Be able to absorb 100s of new users at once #143

lourot opened this issue Sep 11, 2018 · 19 comments
Labels
enhancement New feature or request
Milestone

Comments

@lourot
Copy link
Member

lourot commented Sep 11, 2018

Demand is high: https://news.ycombinator.com/item?id=17951478

@lourot lourot added the enhancement New feature or request label Sep 11, 2018
@lourot lourot self-assigned this Sep 11, 2018
lourot added a commit that referenced this issue Sep 11, 2018
lourot added a commit that referenced this issue Sep 11, 2018
@lourot lourot added this to the scaling issues - round 2 milestone Sep 11, 2018
lourot added a commit that referenced this issue Sep 11, 2018
@sam0x17
Copy link

sam0x17 commented Sep 11, 2018

as per my comment on HN, a good way to do this might be using headless chrome via puppeteer from within a Google Cloud Function or using AWS Lambda. IP rate limiting might not be an issue given how the network of worker nodes is set up. You could even write one cloud function per provider (there are like 4 or 5 that do serverless functions including Google, AWS, CloudFlare, etc) and randomly select between these to further increase your IP uniqueness.

@sam0x17
Copy link

sam0x17 commented Sep 11, 2018

Yet another option might be to use Google's networking module to purchase a large pool of IPs and proxy through them via a cloud function as described above. I'm pretty sure that is possible.

Edit: looks like this might be possible from the Google Cloud control panel if you go to VPC Networking => External IP Addresses

edit again: apparently those can only be assigned to VPS instances. Still, you could use that to your advantage. Assign like 10 IPs to a beefy VPS, then you can do 10 profiles at a time if you can programattically select which IP to use for the current request via Google's API. I don't know amazon's stack as well but I'm sure they have the equivalent.

@styfle
Copy link
Contributor

styfle commented Sep 11, 2018

You might want to reach out to @zeit because they have a system that can build and deploy any Dockerfile and then auto scale as traffic increases. They will typically provide free or subsidized hosting for open source projects.

I am using zeit now to host a somewhat popular project with great success 😄

https://zeit.co/now

@joecohens
Copy link

Maybe this can repo can help https://github.com/zeit/now-examples it's filled with lot of examples using Dockerfile deployments, also you can read more about it here https://zeit.co/blog/serverless-docker if you have questions let me know 😄

@tunnckoCore
Copy link

Yup. Now is the way to go. Or at least if the current plans are don't not enough, contact them, they may support OSS with some customized plan.

Now is just freaking zero pain and zero efforts, 3 characters on a terminal and nothing more, and gives so freaking much in return. It's not just a hype or whatever. This company is really pawning and rocketing the things.

@ocdtrekkie
Copy link

One thing I would've recommended is to have a way to be notified when the service is available again when you removed the ability to join the queue. "We'll email you when your profile is ready" or even "we'll email you when our service can handle more profiles" would've been a better approach, though I am sure you were in a hurry to shut it off.

@lourot
Copy link
Member Author

lourot commented Sep 14, 2018

@ocdtrekkie Good idea, we have now created a twitter account: https://twitter.com/ghuserio . Thanks!

@michaelmior
Copy link

@AurelienLourot In case you weren't aware, Twitter is telling me that the profile is temporarily restricted when I try to visit.

lourot added a commit that referenced this issue Sep 14, 2018
lourot added a commit that referenced this issue Sep 14, 2018
lourot added a commit that referenced this issue Sep 14, 2018
@brillout
Copy link
Collaborator

We know and I just contacted Twitter about it https://twitter.com/brillout/status/1040643164774834176

@DiegoRBaquero
Copy link

Still? D:

@sam0x17
Copy link

sam0x17 commented Sep 21, 2018

Would love to use this if it is ready

@lourot
Copy link
Member Author

lourot commented Sep 21, 2018

@sam0x17 actually your profile is already there 😉 https://ghuser.io/sam0x17

@DiegoRBaquero yes still heavily working on scaling up. When this issue got opened we were serving about 200 profiles. Now there are 700 profiles and you are probably one of the 2000+ users who clicked "Get your profile" at some point, and we won't forget you 🙂 We're just kind of slow I admit. More details:

@aisensiy
Copy link

Waiting for my profile 😃

@reeversedev
Copy link

How soon can I expect that I'll be able to create my profile in ghuser.io? Would love to contribute as well.

@ghuser-io ghuser-io locked and limited conversation to collaborators Oct 3, 2018
@ghuser-io ghuser-io unlocked this conversation Oct 12, 2018
lourot added a commit that referenced this issue Oct 12, 2018
@lourot
Copy link
Member Author

lourot commented Oct 12, 2018

🎉 we have just managed to catch up. Everyone who has ever clicked on 'Get your profile' up to yesterday should now have a profile. A few more facts:

  • We are not overloaded anymore. We serve 3k users and with the current design we can grow up to 7k users. Probably easily 20k users with a few tweaks. The bottleneck is GitHub's API rate limit but we're not yet out of ideas to go beyond, e.g. see Leverage GraphQL db#13
  • About 10 people are joining every day at the moment. We don't have a satisfying automation implemented yet so when someone clicks on 'Get your profile', their username ends up in a queue which we process semi-automatically within 48 hours. Obviously this is one of the most urgent things to properly implement. This is what this issue is now about.
  • The code base got dirtier at many places because we tried to rush at some point and we are now investing some time into cleaning this up.
  • And as you may have noticed: we haven't stopped implementing new features :)

@brillout
Copy link
Collaborator

Update: Onboarding is currently stalled again: We are having an issue with our script that crawls GitHub (see #172). Good news though: We already have found a solution.

@lourot
Copy link
Member Author

lourot commented Oct 16, 2018

Update: #172 is fixed 🎉 Onboarding is reopened.

@8bittts
Copy link

8bittts commented Dec 7, 2018

cool. i shall wait patiently... 📦

@noxstream
Copy link

noxstream commented Dec 7, 2018 via email

@ghuser-io ghuser-io locked and limited conversation to collaborators Dec 7, 2018
@lourot lourot removed their assignment Jul 4, 2019
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests