Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TryStopNukeOpsFromConstantlyFailing test is flaky #29821

Open
ShadowCommander opened this issue Jul 8, 2024 · 4 comments
Open

TryStopNukeOpsFromConstantlyFailing test is flaky #29821

ShadowCommander opened this issue Jul 8, 2024 · 4 comments
Labels
Issue: Heisentest An issue that sometimes causes CI tests to fail, seemingly at random

Comments

@ShadowCommander
Copy link
Member

ShadowCommander commented Jul 8, 2024 •
edited
Loading

Description

https://github.com/space-wizards/space-station-14/actions/runs/9837766958/job/27156273176?pr=29819

https://github.com/space-wizards/space-station-14/actions/runs/9837925999/job/27156732569
Oldest one found by @slarticodefast: https://github.com/space-wizards/space-station-14/actions/runs/9805417608/job/27083467045?pr=29731

  Failed TryStopNukeOpsFromConstantlyFailing [12 s]
  Error Message:
   Robust.Shared.Utility.DebugAssertException : Exception of type 'Robust.Shared.Utility.DebugAssertException' was thrown.

  1) Test was dirty-disposed.

  Stack Trace:
     at Content.Server.Atmos.Piping.EntitySystems.AtmosDeviceSystem.Update(Single frameTime) in /home/runner/work/space-station-14/space-station-14/Content.Server/Atmos/Piping/EntitySystems/AtmosDeviceSystem.cs:line 124
   at Robust.Shared.GameObjects.EntityManager.TickUpdate(Single frameTime, Boolean noPredictions, Histogram histogram) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/EntityManager.cs:line 256
   at Robust.Server.GameObjects.ServerEntityManager.TickUpdate(Single frameTime, Boolean noPredictions, Histogram histogram) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Server/GameObjects/ServerEntityManager.cs:line 195
   at Robust.Server.BaseServer.Update(FrameEventArgs frameEventArgs) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Server/BaseServer.cs:line 702
   at Robust.UnitTesting.RobustIntegrationTest.IntegrationGameLoop.SingleThreadRunUntilEmpty() in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.UnitTesting/RobustIntegrationTest.cs:line 1097
   at Robust.UnitTesting.RobustIntegrationTest.IntegrationGameLoop.Run() in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.UnitTesting/RobustIntegrationTest.cs:line 1084
   at Robust.UnitTesting.RobustIntegrationTest.ServerIntegrationInstance._serverMain() in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.UnitTesting/RobustIntegrationTest.cs:line 655

2)    at Content.Server.Atmos.Piping.EntitySystems.AtmosDeviceSystem.Update(Single frameTime) in /home/runner/work/space-station-14/space-station-14/Content.Server/Atmos/Piping/EntitySystems/AtmosDeviceSystem.cs:line 124
   at Robust.Shared.GameObjects.EntityManager.TickUpdate(Single frameTime, Boolean noPredictions, Histogram histogram) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/EntityManager.cs:line 256
   at Robust.Server.GameObjects.ServerEntityManager.TickUpdate(Single frameTime, Boolean noPredictions, Histogram histogram) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Server/GameObjects/ServerEntityManager.cs:line 195
   at Robust.Server.BaseServer.Update(FrameEventArgs frameEventArgs) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Server/BaseServer.cs:line 702
   at Robust.UnitTesting.RobustIntegrationTest.IntegrationGameLoop.SingleThreadRunUntilEmpty() in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.UnitTesting/RobustIntegrationTest.cs:line 1097
   at Robust.UnitTesting.RobustIntegrationTest.IntegrationGameLoop.Run() in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.UnitTesting/RobustIntegrationTest.cs:line 1084
   at Robust.UnitTesting.RobustIntegrationTest.ServerIntegrationInstance._serverMain() in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.UnitTesting/RobustIntegrationTest.cs:line 655


 GetServerClientPair: Pair 1 Test History Start
 - Pair 1 Test #0: Actions.ActionPvsDetachTest.TestActionDetach
 - Pair 1 Test #1: Administration.Logs.AddTests.AddAndGetUnformattedLog
 - Pair 1 Test #2: Administration.Logs.AddTests.PreRoundAddAndGetSingle
 - Pair 1 Test #3: Administration.Logs.QueryTests.QuerySingleLog
 - Pair 1 Test #4: Atmos.AlarmThresholdTest.TestAlarmThreshold
 - Pair 1 Test #5: Atmos.ConstantsTest.TotalGasesTest
 - Pair 1 Test #6: Body.GibTest.TestGib
 - Pair 1 Test #7: Atmos.GasMixtureTest.RemoveRatio(0.5f)
 - Pair 1 Test #8: Atmos.GasMixtureTest.RemoveRatio(0.25f)
 - Pair 1 Test #9: Atmos.GasMixtureTest.RemoveRatio(0.75f)
 - Pair 1 Test #10: Atmos.GasMixtureTest.RemoveRatio(1.0f)
 - Pair 1 Test #11: Atmos.GasMixtureTest.RemoveRatio(0.0f)
 - Pair 1 Test #12: Atmos.GasMixtureTest.RemoveRatio(0.0002f)
 - Pair 1 Test #13: Atmos.GasMixtureTest.TestMerge
 - Pair 1 Test #14: Body.LegTest.RemoveLegsFallTest
 - Pair 1 Test #15: Shuttle.DockTest.TestDockingConfig(<0.5, 0.5>,<0.5, 0.5>,0 rad,0 rad,True)
 - Pair 1 Test #16: Shuttle.DockTest.TestDockingConfig(<0.5, 1.5>,<0.5, 1.5>,0 rad,0 rad,False)
 - Pair 1 Test #17: Shuttle.DockTest.TestPlanetDock
 - Pair 1 Test #18: Buckle.BuckleDragTest.BucklePullTest
 - Pair 1 Test #19: Buckle.BuckleTest.ForceUnbuckleBuckleTest
 - Pair 1 Test #20: CargoTest.NoCargoBountyArbitrageTest
 - Pair 1 Test #21: CargoTest.NoSliceableBountyArbitrageTest
 - Pair 1 Test #22: CargoTest.NoStaticPriceAndStackPrice
 - Pair 1 Test #23: Chemistry.SolutionSystemTests.TestTemperatureCalculations
 - Pair 1 Test #24: Chemistry.SolutionSystemTests.TryAddTooMuchNonReactiveReagent
 - Pair 1 Test #25: Chemistry.SolutionSystemTests.TryAddTwoNonReactiveReagent
 - Pair 1 Test #26: Chemistry.SolutionSystemTests.TryMixAndOverflowTooBigOverflow
 - Pair 1 Test #27: Chemistry.SolutionSystemTests.TryMixAndOverflowTooMuchReagent
 - Pair 1 Test #28: Chemistry.SolutionRoundingTest.Test
 - Pair 1 Test #29: Cleanup.EuiManagerTest.EuiManagerRecycleWithOpenWindowTest
 - Pair 1 Test #30: Cleanup.EuiManagerTest.EuiManagerRecycleWithOpenWindowTest
 - Pair 1 Test #31: Chemistry.TryAllReactionsTest.TryAllTest
 - Pair 1 Test #32: Climbing.ClimbingTest.ClimbTableTest
 - Pair 1 Test #33: ClickableTest.Test("ClickTestRotatingCornerInvisible",-0.25f,-0.25f,3.141592653589793d,1)
 - Pair 1 Test #34: ClickableTest.Test("ClickTestRotatingCornerInvisibleNoRot",-0.25f,-0.25f,0.0d,1)
 - Pair 1 Test #35: ClickableTest.Test("ClickTestRotatingCornerInvisibleNoRot",0.25f,0.25f,3.141592653589793d,1)
 - Pair 1 Test #36: ClickableTest.Test("ClickTestRotatingCornerInvisibleNoRot",-0.25f,-0.25f,3.141592653589793d,1)
 - Pair 1 Test #37: ClickableTest.Test("ClickTestRotatingCornerInvisibleNoRot",0,0.35f,0.6853981633974483d,1)
 - Pair 1 Test #38: ClickableTest.Test("ClickTestRotatingCornerInvisibleNoRot",0.25f,0.25f,0.6853981633974483d,1)
 - Pair 1 Test #39: Commands.RejuvenateTest.RejuvenateDeadTest
 - Pair 1 Test #40: Construction.ConstructionActionValid.ConstructionGraphEdgeValid
 - Pair 1 Test #41: Construction.ConstructionActionValid.ConstructionGraphSpawnPrototypeValid
 - Pair 1 Test #42: Construction.Interaction.ComputerConstruction.ChangeComputer
 - Pair 1 Test #43: Construction.Interaction.ComputerConstruction.DeconstructComputer
 - Pair 1 Test #44: Construction.Interaction.CraftingTests.CraftRods
 - Pair 1 Test #45: Construction.ConstructionPrototypeTest.TestStartReachesValidTarget
 - Pair 1 Test #46: Construction.ConstructionPrototypeTest.TestTargetIsValid
 - Pair 1 Test #47: Construction.Interaction.MachineConstruction.ChangeMachine
 - Pair 1 Test #48: Construction.Interaction.PanelScrewing.WiresPanelScrewing("Airlock")
 - Pair 1 Test #49: Construction.Interaction.PanelScrewing.WiresPanelScrewing("TelecomServerFilled")
 - Pair 1 Test #50: Construction.Interaction.WallConstruction.ConstructWall
 - Pair 1 Test #51: Construction.Interaction.GrilleWindowConstruction.ConstructionBlocker("Grille","Grille")
 - Pair 1 Test #52: Construction.Interaction.GrilleWindowConstruction.ConstructionBlocker("Window","Grille")
 - Pair 1 Test #53: Construction.Interaction.GrilleWindowConstruction.ConstructionBlocker("Window","Window")
 - Pair 1 Test #54: Construction.Interaction.GrilleWindowConstruction.WindowOnGrille
 - Pair 1 Test #55: Construction.Interaction.WindowRepair.RepairReinforcedWindow
 - Pair 1 Test #56: DummyIconTest.Test
 - Pair 1 Test #57: EntityTest.SpawnAndDeleteEntityCountTest

Reproduction

Screenshots

Additional context
@github-actions github-actions bot added the Status: Untriaged Somebody go through and give this appropriate labels label Jul 8, 2024
@metalgearsloth
Copy link
Contributor

What I think is actually happening is 2 things:

  1. vgroid has a small chance of triggering the assert and so this carries across to tests.
  2. Any tests that have grid_fill can probably trigger it due to making vgroid.

@slarticodefast slarticodefast mentioned this issue Jul 8, 2024
Closed
@Tayrtahn Tayrtahn added Issue: Heisentest An issue that sometimes causes CI tests to fail, seemingly at random and removed Status: Untriaged Somebody go through and give this appropriate labels labels Jul 8, 2024
@Tayrtahn
Copy link
Contributor

Tayrtahn commented Jul 8, 2024 •
edited
Loading

I believe @metalgearsloth's assessment is right. To add a bit more info:

  • vgroid is big enough that AutomaticAtmosSystem gives it an atmosphere component.
  • Something that counts as an atmos device (a gas canister?) probably spawns on the vgroid grid at least some of the time.

AtmosDeviceSystem tracks the canister as an off-grid device when it spawns on a grid with no atmosphere. If AutomaticAtmosSystem then adds an atmosphere to the grid, the assert will trip the next time AtmosDeviceSystem updates, as it does a check of each tracked off-grid device to make sure that its grid doesn't have an atmosphere component.

I have a potential fix here: #29752

Edit: the logs seem to support this theory:
Screenshot_20240708-093117~2.png

Screenshot_20240708-094301~2.png

Screenshot_20240708-094329~2.png

@metalgearsloth
Copy link
Contributor

I think grid_fill could be updated to give the components first + an atmos fix but at either rate we could also comment out the vgroid yaml in the interim to make tests not flaky.

@ShadowCommander
Copy link
Member Author

It also seems to be failing the FailAndStartTest.

https://github.com/space-wizards/space-station-14/actions/runs/9839682882/job/27162272383

  Failed FailAndStartTest [5 s]
  Error Message:
   Robust.Shared.Utility.DebugAssertException : Exception of type 'Robust.Shared.Utility.DebugAssertException' was thrown.

  1) Test was dirty-disposed.

  Stack Trace:
     at Content.Server.Atmos.Piping.EntitySystems.AtmosDeviceSystem.Update(Single frameTime) in /home/runner/work/space-station-14/space-station-14/Content.Server/Atmos/Piping/EntitySystems/AtmosDeviceSystem.cs:line 124
   at Robust.Shared.GameObjects.EntityManager.TickUpdate(Single frameTime, Boolean noPredictions, Histogram histogram) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Shared/GameObjects/EntityManager.cs:line 256
   at Robust.Server.GameObjects.ServerEntityManager.TickUpdate(Single frameTime, Boolean noPredictions, Histogram histogram) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Server/GameObjects/ServerEntityManager.cs:line 195
   at Robust.Server.BaseServer.Update(FrameEventArgs frameEventArgs) in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.Server/BaseServer.cs:line 702
   at Robust.UnitTesting.RobustIntegrationTest.IntegrationGameLoop.SingleThreadRunUntilEmpty() in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.UnitTesting/RobustIntegrationTest.cs:line 1097
   at Robust.UnitTesting.RobustIntegrationTest.IntegrationGameLoop.Run() in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.UnitTesting/RobustIntegrationTest.cs:line 1084
   at Robust.UnitTesting.RobustIntegrationTest.ServerIntegrationInstance._serverMain() in /home/runner/work/space-station-14/space-station-14/RobustToolbox/Robust.UnitTesting/RobustIntegrationTest.cs:line 655

Also vgroid getting GridAtmosphereComponent.

 SERVER: 3.669s [INFO] ticker: Started game rule  (40601/n40601, TestRule)
 SERVER: 3.771s [INFO] system.mind: Session integration_18 transferred to entity 42985.
 SERVER: 3.772s [INFO] SERVER: Have a happy Writer's Day!
 SERVER: 3.777s [INFO] system.dungeon: Generating dungeon VGRoid with seed 876740989 on Fasciola-51-T (40608/n40608)
 CLIENT: 4.213s [INFO] player: Attaching local player to Porter Pennington (3055/n42985, MobDwarf, integration_18).
 CLIENT: 4.232s [DEBG] root: Switching to state Content.Client.Gameplay.GameplayState
 SERVER: 4.805s [INFO] loader: No user map found: /Maps/Dungeon/experiment.yml
 SERVER: 4.805s [INFO] loader: Loading Map: /Maps/Dungeon/experiment.yml
 SERVER: 4.861s [WARN] loader: Map contains an obsolete/removed prototype: RPED. This may cause unexpected errors.
 SERVER: 4.861s [WARN] loader: Map contains an obsolete/removed prototype: VehicleJanicartDestroyed. This may cause unexpected errors.
 SERVER: 5.073s [INFO] loader: Loading map file with a root node onto an existing map!
 SERVER: 5.222s [INFO] system.automatic_atmos: Giving grid 40608 GridAtmosphereComponent.
 DisposeAsync: Dirty return of pair 1 started
 DisposeAsync: Test gave back pair 1 in 4349.8446 ms
 DisposeAsync: Disposed pair 1 in 0.1845 ms

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Issue: Heisentest An issue that sometimes causes CI tests to fail, seemingly at random
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@Tayrtahn @ShadowCommander @metalgearsloth