-
-
Notifications
You must be signed in to change notification settings - Fork 1.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Ants + fasthttp #95
Comments
It's unnecessary to use |
I guess the WorkerPool in fasthttp is for Server use only. So this would not help solving the task.
|
Sorry, I didn't check out the problem carefully before and thought you were talking about the server-side, have you tried not to use If the memory usage is still high, it probably has nothing to do with |
I tried now to use it without Calling So im not sure what and why.. even with |
There is no way that You can add some debug code into your |
I include You could be right with "close any sockets after task"-issue. It could be the reason i have to increase the ulimit of open files (sockets in this case) before execution of the application: I use:
I checked the godoc https://godoc.org/github.com/valyala/fasthttp again, i dont see any other option to close the connection. In my opinion Any idea? |
How are you measuring memory usage? If you want to close keep-alive connections you should set a Or you could set https://godoc.org/github.com/valyala/fasthttp#Client.MaxIdleConnDuration to a second for example if you want to reuse the connection shortly but don't want to keep it around for long. |
I measure using Ok i will add the Regarding |
Ants: I added now the The
Its a lot for 25 workers checking 2650 urls. For my logical understanding 25 workers -> each one takes a URL and request the HTML from it. Then do the parsing stuff. So even when the HTML is 1MB each URL so we are in total at 25MB, some worker is faster, some slower.. lets increase it to the double size -> 50MB. So lets say the GC is very slow and need some time to recognise the current As you see the Application use I checked my code again and the only one point where is a real consumption of memory is happening is the fasthttp request while receiving the requested HTML Code from the URL. Without Ants:
For my understanding the Goroutines should activate the GC after the |
Sounds like more memory than should be required. But keep in mind that fasthttp trades more memory usage for less CPU usage by reusing buffers. What I would do next is use https://golang.org/pkg/net/http/pprof/ to see where the allocations are. Make sure to set Also could you try using |
Ok thank you for the advise, i will check it and we will see. Yes the memory consumption is abnormal huge. Good that you see it the same way. I was already worry in myself. Yes i read already that fasthttp trades memory for cpu. But everything over 500MB for this task is crazy and unreal. |
So i checked no the noAnts_Program with 50 rows only:
Looks like everything is as it should be. Hmm. |
noAnts_NetHttp_Program / 50 rows=Urls:
noAnts_FastHttp_Program / 50 rows=Urls:
Ofc time measuring is difficult because of network latency but i took the same 50 urls.. so in one way its not 100% reliable but on the other side it is telling us something. |
Ants_NetHttp_Program / 50 rows=Urls / 25 Ants Worker:
Ants_FastHttp_Program / 50 rows=Urls / 25 Ants Worker:
Im on the end with my knowledge and really dont know how to continue this task with golang. Maybe i have some mistake in my code, but the code is 200 lines.. and Golang is very simple.. so i dont think there can be one/such a bad mistake. With Net/Http the Memory Usage is lower than with FastHttp but even this memory consumption is high in my opinion. 50 seconds/2 minutes for 50 http requests on 25 workers? If i use the BulkOpener Extension for Chrome and Open the 50 Urls it takes same time including rendering. It is strange, i dont trust all this numbers - i have even better numbers with php+pthreads (except the crazy high cpu usage). |
Would you be willing to share your code in private with me so I can see what can be improved? |
Not just willing i would appreciate it much - so i even can learn. I will send it to you by email in a few minutes. Already now i wanna thank you. |
This issue is marked as stale because it has been open for 30 days with no activity. You should take one of the following actions:
This issue will be automatically closed in 7 days if no further activity occurs. |
This issue was closed because it has been inactive for 7 days since being marked as stale. If you believe this is a false alarm, please leave a comment for it or open a new issue, you can also reopen this issue directly if you have permission. |
Hello,
I am using Ants with fasthttp. Is it possible that this both packages are not compatible?
My Program fetchs 15000 rows from a MysqlDB and request the HTML code for parse 2 strings out of it.
My memory usage is crazy 1,9g with only 434 open socket files = 434 connections (maximum was around 2800):
15921 jonny 20 0 3093,3m 1,9g 6,9m S 7,3 25,1 11:59.63 goCheckResults
root@jdebian:~$ ls -l /proc/15921/fd | wc -l
434
in main():
The function processRow calls
It is crazy how much memory usage this small program has.
The text was updated successfully, but these errors were encountered: