-
-
Notifications
You must be signed in to change notification settings - Fork 2.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Internal panics in CurrentThread::block_on()
under WSLv1
#6620
Comments
CurrentThread::block_on()
under WSLv1
I'm thinking of #6424. Do you have records of which packages were upgraded since it became unstable? Or whether you changed rustc version? |
Thanks for the quick reply! I check in my |
It's very likely to be caused by either the upgrade of rustc, or the upgrade of a dependency. It would be extremely helpful if you could help us narrow it down; as you can see, other people have also reported problems, and I do not have access to any way to actually reproduce it. To start off, would you be able to revert the change to |
Sure, that's doable. I've reverted the changes to
|
Ok, it crashed with SIGSEGV, which is new. I'll try again with ASAN. |
Running under ASAN doesn't report any memory corruption our out-of-bound reads/writes until something (tokio, I think) tries to access an invalid address:
The ASAN stack trace isn't symbolized (though the release profile specifies debug=2), so here's a symbolification of it:
|
In that case it sounds like a compiler bug. If you try the old compiler again, does the problem go away? If so, then you should be able to bisect to the nightly that introduced the problem by checking only 7 compilers. |
Unfortunately easier said than done. It's an interactive application that requires user input and performs non-repeatable work. The crashes generally don't happen right away (though that has occurred); I'll probably need to try and script it so that I can leave it to run automated for some amount of time and see if it segfaults. I don't know if the idle "waiting for user input" state contributes to the bug, in which case it might not reproduce under automation. But we'll see. |
I don't think it's a compiler regression; I've managed to reproduce it with the nightly-2022-09-01 toolchain. It reproduces under ASAN but not under valgrind (so probably timing dependent) but neither complain of any memory corruption. I commented out a huge chunk of the code in the process of automating and speeding up the bisection runs and it still happens. The stack trace from tokio 1.21 is attached. (2023-02-01 tokio 1.21 crash.txt) It's either an invalid argument or bad address tokio error under 1.38 or event store missing under 1.21 (or sigsegv without a tokio message, but still in the tokio stack). Haven't seen any crashes outside the tokio stack trace, but obviously that isn't definitive proof of anything. As mentioned, this is under WSLv1. Is there anything I could try to debug the tokio internals here? |
I was able to compile the same binary on the same machine and run it against the same workload but as a Windows executable rather than as a Linux one (thanks, WSL) and run that for an extended test without any issues, so it seems to be something in the tokio's unix code that is triggering this. (My own code is ~the same between Windows and Linux). Also for what it's worth, this is a machine with ECC ram. |
Are you able to find any configuration on Linux where the crash doesn't happen? |
Version
tokio v1.38.0
Platform
WSL installation of Ubuntu 23.10
Linux 4.4.0-19041-Microsoft #4355-Microsoft Thu Apr 12 17:37:00 PST 2024 x86_64 x86_64 x86_64 GNU/Linux
Description
I have a project using tokio for async process support and it has been rock-solid and stable for years, but has recently begun to crash deep inside calls to
Runtime::block_on()
(using thecurrent_thread
runtime), mostly (see below) with the following error:It's a large project and next to impossible to isolate this issue, but knowing there would be a suggestion this is the application itself at fault, I compiled with ASAN enabled for rust and the C dependencies, but no issues were reported. (Running under miri is impossible and out of the question). I was able to reproduce the crash with a debug ASAN build (previous errors were compiled as release), but this time the error was
I have attached the full stack trace for both exceptions:
The text was updated successfully, but these errors were encountered: