Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Receiver crashes if DNs file is being written to and read from at the same time. #79

Open
gregcorbett opened this issue Sep 10, 2018 · 5 comments · May be fixed by #252
Open

Receiver crashes if DNs file is being written to and read from at the same time. #79

gregcorbett opened this issue Sep 10, 2018 · 5 comments · May be fixed by #252
Assignees
@DanielPerkins7
Labels
bug good first issue Good for newcomers
Milestone
4.1.0

Comments

@gregcorbett
Copy link
Member

As per this ELOG, bin/receiver.py seems to crashes if get_dns is called while the DNs file is being written to by the auth component of APEL.

It would be good to be able to reproduce this bug to determine the exact cause.
@tofu-rocketry
Copy link
Member

It happened again.

@gregcorbett
Copy link
Member Author

I did the tiniest amount of investigation to try and come up with a repeatable test for this and it seems (at least under Python 3) having the dns file open for writing while trying to read from it isn't sufficient to cause this. Maybe the cause is trying to read the exact moment the write is actually flushed to disk...

@tofu-rocketry
Copy link
Member

tofu-rocketry commented Jul 17, 2019 •
edited
Loading

Sounds like a simple retry would be sufficient after ~30 seconds, with a limit of, say, 3 attempts before barfing.

@tofu-rocketry tofu-rocketry added the good first issue Good for newcomers label Jul 30, 2019
@gregcorbett
Copy link
Member Author

It happened again

@tofu-rocketry
Copy link
Member

So it's ending up here:

ssm/ssm/agents.py

Line 377 in 83e33c7

raise Ssm2Exception('No valid DNs found in %s. SSM will not start' % dn_file)

So it's getting a len of 0. So we could just add a couple of retries within that function to see if it can get len>0

DanielPerkins7 added a commit to DanielPerkins7/ssm that referenced this issue Aug 2, 2023
@DanielPerkins7
adds retries to fix len(dns). issue 79
96020a3 
resolve apel#79 . i was unsure which part to loop so i just repeated the whole chunk within the try statement.
@DanielPerkins7 DanielPerkins7 mentioned this issue Aug 2, 2023
Closed
DanielPerkins7 added a commit to DanielPerkins7/ssm that referenced this issue Aug 2, 2023
@DanielPerkins7
adds retries to fix len(dns). issue 79
d034c40 
resolve apel#79 . i was unsure which part to loop so i just repeated the whole chunk within the try statement.
DanielPerkins7 added a commit to DanielPerkins7/ssm that referenced this issue Aug 2, 2023
@DanielPerkins7
adds retries to fix len(dns). issue 79
31dc072 
resolve apel#79 . i was unsure which part to loop so i just repeated the whole chunk within the try statement.
DanielPerkins7 added a commit to DanielPerkins7/ssm that referenced this issue Aug 2, 2023
@DanielPerkins7
adds loop to fix len(dns). issue 79
2cf2a76 
resolve apel#79 . i have just looped everything in the try statement although i am not sure if it is whats needed.
DanielPerkins7 added a commit to DanielPerkins7/ssm that referenced this issue Aug 2, 2023
@DanielPerkins7
adds loop to fix len(dns). issue 79
30f22c0 
resolve apel#79 . i have just looped everything in the try statement although i am not sure if it is whats needed.
DanielPerkins7 added a commit to DanielPerkins7/ssm that referenced this issue Aug 2, 2023
@DanielPerkins7
adds loop to fix len(dns). issue 79
86db120 
resolve apel#79 . i have just looped everything in the try statement although i am not sure if it is whats needed.
@DanielPerkins7 DanielPerkins7 linked a pull request Aug 2, 2023 that will close this issue
Open
DanielPerkins7 added a commit to DanielPerkins7/ssm that referenced this issue Aug 2, 2023
@DanielPerkins7
adds loop to fix len(dns) - fixed. issue 79
46f0d1a 
resolve apel#79
DanielPerkins7 added a commit to DanielPerkins7/ssm that referenced this issue Aug 2, 2023
@DanielPerkins7
adds loop to fix len(dns) - fixed. issue 79
fb427c0 
resolve apel#79
DanielPerkins7 added a commit to DanielPerkins7/ssm that referenced this issue Aug 4, 2023
@DanielPerkins7
adds loop to fix len(dns). issue 79
db0d6af 
resolve apel#79 . i have just looped everything in the try statement although i am not sure if it is whats needed.
@tofu-rocketry tofu-rocketry added this to the 3.5.0 milestone Oct 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@DanielPerkins7 DanielPerkins7

Labels
bug good first issue Good for newcomers
Milestone
4.1.0
3 participants
@gregcorbett @tofu-rocketry @DanielPerkins7