Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AArch64 Mac SC_Softmx_Increase_0 crash vmState=0x00000000 #16029

Open
pshipton opened this issue Oct 3, 2022 · 4 comments
Open

AArch64 Mac SC_Softmx_Increase_0 crash vmState=0x00000000 #16029

pshipton opened this issue Oct 3, 2022 · 4 comments
Labels
arch:aarch64 os:macos test failure

Comments

@pshipton
Copy link
Member

pshipton commented Oct 3, 2022

https://openj9-jenkins.osuosl.org/job/Test_openjdk17_j9_extended.system_aarch64_mac_Nightly_testList_1/150
SC_Softmx_Increase_0 -Xdump:system:events=user

https://openj9-artifactory.osuosl.org/artifactory/ci-openj9/Test/Test_openjdk17_j9_extended.system_aarch64_mac_Nightly_testList_1/150/system_test_output.tar.gz

STF 10:22:56.286 - +------ Step 9 - Run a second workload after the cache size has been increased
...
STF 10:22:58.059 - Found dump at: /Users/jenkins/workspace/Test_openjdk17_j9_extended.system_aarch64_mac_Nightly_testList_1/aqa-tests/TKG/output_16645821062257/SC_Softmx_Increase_0/20221001-102230-SharedClassesWorkloadTest_Softmx_Increase/results/core.20221001.102258.29697.0001.dmp
STF 10:23:03.421 - **FAILED** Process Jvm2 ended with exit code (255) and not the expected exit code/s (0,1)
STF 10:23:03.421 - Monitoring Report Summary:
STF 10:23:03.421 -   o Process Jvm2 has crashed unexpectedly

9.Jvm2.stderr

Type=Segmentation error vmState=0x00000000
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000002
Handler1=0000000103294AA0 Handler2=00000001030C6E18 InaccessibleAddress=0000000000000012
x0=00000001230D5100 x1=00000001230D5100 x2=0000000118593B00 x3=000000016DBE6920
x4=000000016DBE68E8 x5=000000016DBE68D8 x6=000000016DBE68D0 x7=000000016DBE68C8
x8=0000000123018020 x9=000000000000000A x10=000000016DBE68D0 x11=000000016DBE68D8
x12=000000016DBE68E8 x13=000000016DBE68C0 x14=00000001032DD1B0 x15=0000000000000010
x16=00000001032DCC4C x17=00000001F77A1648 x18=00000001297CD480 x19=0000000040000000
x20=0000000000000002 x21=000000028009A290 x22=000000016DBE68D0 x23=000000013015C28C
x24=000000016DBE68E8 x25=000000016DBE68D8 x26=000000016DBE68D0 x27=000000016DBE68C8
x28=00000001033B2C64 x29(FP)=000000016DBE68B0 x30(LR)=00000001032DCC40 x31(SP)=000000016DBE6620
PC=00000001032F1A20 SP=000000016DBE6620
v0 0000000123974809 (f: 597116928.000000, d: 2.417011e-314)
v1 0000000122020b78 (f: 570559360.000000, d: 2.403890e-314)
v2 41cdcd6500000000 (f: 0.000000, d: 1.000000e+09)
v3 0000000000000001 (f: 1.000000, d: 4.940656e-324)
v4 0000000000000003 (f: 3.000000, d: 1.482197e-323)
v5 0000000000000002 (f: 2.000000, d: 9.881313e-324)
v6 0000080000000800 (f: 2048.000000, d: 4.345847e-311)
v7 000000000000000c (f: 12.000000, d: 5.928788e-323)
v8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v16 bfd0000000000000 (f: 0.000000, d: -2.500000e-01)
v17 3fd5455f10377955 (f: 272071008.000000, d: 3.323591e-01)
v18 3f6fdca83ae5a053 (f: 988127296.000000, d: 3.889397e-03)
v19 3fe62e42fefa39ef (f: 4277811712.000000, d: 6.931472e-01)
v20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v21 ffffffffffffffff (f: 4294967296.000000, d: nan)
v22 ffffffffffffffff (f: 4294967296.000000, d: nan)
v23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
Module=/Users/jenkins/workspace/Test_openjdk17_j9_extended.system_aarch64_mac_Nightly_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9vm29.dylib
Module_base_address=0000000103270000 Symbol=_ZN26VM_BytecodeInterpreterFull3runEP10J9VMThread
Symbol_address=00000001032DCC4C
Target=2_90_20221001_158 (Mac OS X 11.5.2)
CPU=aarch64 (8 logical CPUs) (0x400000000 RAM)
----------- Stack Backtrace -----------
---------------------------------------
JVMDUMP039I Processing dump event "gpf", detail "" at 2022/10/01 10:22:58 - please wait.
JVMDUMP032I JVM requested System dump using '/Users/jenkins/workspace/Test_openjdk17_j9_extended.system_aarch64_mac_Nightly_testList_1/aqa-tests/TKG/output_16645821062257/SC_Softmx_Increase_0/20221001-102230-SharedClassesWorkloadTest_Softmx_Increase/results/core.20221001.102258.29697.0001.dmp' in response to an event
Unhandled exception
Type=Segmentation error vmState=0x00000000
J9Generic_Signal_Number=00000018 Signal_Number=0000000b Error_Value=00000000 Signal_Code=00000002
Handler1=0000000103294AA0 Handler2=00000001030C6E18 InaccessibleAddress=0000000000000010
x0=00000001231A4F00 x1=00000001231A4F00 x2=0000000118593B00 x3=000000016DD62410
x4=000000016DD623D8 x5=000000016DD623C8 x6=000000016DD623C0 x7=000000016DD623B8
x8=0000000000000000 x9=0000000000000010 x10=000000016DD623C0 x11=000000016DD623C8
x12=000000016DD623D8 x13=000000016DD623B0 x14=00000001032DD1B0 x15=0000000000000010
x16=00000001032DCC4C x17=008B0301008B0300 x18=0000000000000000 x19=0000000040000000
x20=000000037F2E3850 x21=000000016DD623B0 x22=000000016DD623C0 x23=000000013015C28C
x24=000000016DD623D8 x25=000000016DD623C8 x26=000000016DD623C0 x27=000000016DD623B8
x28=00000001033B2C64 x29(FP)=000000016DD623A0 x30(LR)=00000001032DCC40 x31(SP)=000000016DD62110
PC=00000001032F1AB8 SP=000000016DD62110
v0 00000001231c1309 (f: 589042432.000000, d: 2.413021e-314)
v1 0000000122020b78 (f: 570559360.000000, d: 2.403890e-314)
v2 00000000490f45c0 (f: 1225737728.000000, d: 6.055949e-315)
v3 bfd07139884d55b6 (f: 2286769664.000000, d: -2.569107e-01)
v4 4034000000000000 (f: 0.000000, d: 2.000000e+01)
v5 3ff0000000000000 (f: 0.000000, d: 1.000000e+00)
v6 0000080000000800 (f: 2048.000000, d: 4.345847e-311)
v7 0000000000000006 (f: 6.000000, d: 2.964394e-323)
v8 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v9 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v10 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v11 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v12 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v13 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v14 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v15 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v16 bfd0000000000000 (f: 0.000000, d: -2.500000e-01)
v17 3fd544a8c2a0c955 (f: 3265317120.000000, d: 3.323156e-01)
v18 3f70a3e883bb79e8 (f: 2210101760.000000, d: 4.062565e-03)
v19 3fe62e42fefa39ef (f: 4277811712.000000, d: 6.931472e-01)
v20 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v21 ffffffffffffffff (f: 4294967296.000000, d: nan)
v22 ffffffffffffffff (f: 4294967296.000000, d: nan)
v23 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v24 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v25 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v26 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v27 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v28 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v29 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v30 0000000000000000 (f: 0.000000, d: 0.000000e+00)
v31 0000000000000000 (f: 0.000000, d: 0.000000e+00)
Module=/Users/jenkins/workspace/Test_openjdk17_j9_extended.system_aarch64_mac_Nightly_testList_1/openjdkbinary/j2sdk-image/lib/default/libj9vm29.dylib
Module_base_address=0000000103270000 Symbol=_ZN26VM_BytecodeInterpreterFull3runEP10J9VMThread
Symbol_address=00000001032DCC4C
Target=2_90_20221001_158 (Mac OS X 11.5.2)
CPU=aarch64 (8 logical CPUs) (0x400000000 RAM)
----------- Stack Backtrace -----------
---------------------------------------
JVMDUMP039I Processing dump event "gpf", detail "" at 2022/10/01 10:22:58 - please wait.
core file generated - /Users/jenkins/workspace/Test_openjdk17_j9_extended.system_aarch64_mac_Nightly_testList_1/aqa-tests/TKG/output_16645821062257/SC_Softmx_Increase_0/20221001-102230-SharedClassesWorkloadTest_Softmx_Increase/results/core.20221001.102258.29697.0001.dmp
JVMDUMP010I System dump written to /Users/jenkins/workspace/Test_openjdk17_j9_extended.system_aarch64_mac_Nightly_testList_1/aqa-tests/TKG/output_16645821062257/SC_Softmx_Increase_0/20221001-102230-SharedClassesWorkloadTest_Softmx_Increase/results/core.20221001.102258.29697.0001.dmp
00000001230D5100: Object neither in heap nor stack-allocated in thread load-0
00000001230D5100:	O-Slot=00000001239745B0
00000001230D5100:	O-Slot value=00000000000001D0
00000001230D5100:	PC=0000000122FB52EB
00000001230D5100:	framesWalked=0
00000001230D5100:	arg0EA=0000000123974688
00000001230D5100:	walkSP=0000000123974570
00000001230D5100:	literals=00000001239C9748
00000001230D5100:	jitInfo=0000000000000000
00000001230D5100:	method=00000001239C9748 (java/lang/invoke/LambdaForm$NFI/0x0000000022fb51f0.invoke_LL_L(Ljava/lang/invoke/MethodHandle;[Ljava/lang/Object;)Ljava/lang/Object;) (Interpreted)
00000001230D5100:	stack=000000012396F240-0000000123974A80

@knn-k
@knn-k
Copy link
Contributor

knn-k commented Oct 4, 2022

I ran a 40x Grinder job, and got one failure. There is a core file, but other diagnostic files are not available.
https://openj9-jenkins.osuosl.org/job/Grinder/1319/

[2022-10-04T00:46:52.807Z] STF 11:46:51.973 - Monitoring processes: Jvm1
[2022-10-04T00:46:54.258Z] STF 11:46:53.828 - Found dump at: /Users/jenkins/workspace/Grinder/aqa-tests/TKG/output_16648444098847/SC_Softmx_Increase_0/20221004-114650-SharedClassesWorkloadTest_Softmx_Increase/results/core.20221004.114653.98385.0001.dmp
[2022-10-04T00:46:58.374Z] STF 11:46:57.350 - **FAILED** Process Jvm1 ended with exit code (255) and not the expected exit code/s (0,1)
[2022-10-04T00:46:58.374Z] STF 11:46:57.350 - Monitoring Report Summary:
[2022-10-04T00:46:58.374Z] STF 11:46:57.350 -   o Process Jvm1 has crashed unexpectedly

@knn-k
Copy link
Contributor

knn-k commented Oct 4, 2022

The crash in Test_openjdk17_j9_extended.system_aarch64_mac_Nightly_testList_1/150:

It is not a crash at the VM termination.
jdmpview shows there are 201 threads running in the VM, and it was the thread load-4 that failed.

> !gpinfo
Failing Thread: !j9vmthread 0x1231a4f00
Failing Thread ID: 0xf0e4277 (252592759)

> !threads
        !stack 0x1231a4f00      !j9vmthread 0x1231a4f00 !j9thread 0x12318cf58   tid 0xf0e4277 (252592759) // (load-4)

> !stack 0x1231a4f00
<1231a4f00>     !j9method 0x00000001239C9748   java/lang/invoke/LambdaForm$NFI/0x0000000022fb51f0.invoke_LL_L(Ljava/lang/invoke/MethodHandle;[Ljava/lang/Object;)Ljava/lang/Object;
<1231a4f00>     !j9method 0x00000001238737C8   java/lang/invoke/LambdaForm$DMH/0x0000000023872e20.invokeStatic(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
<1231a4f00>     !j9method 0x0000000118597210   java/lang/invoke/LambdaForm$NamedFunction.invokeWithArguments([Ljava/lang/Object;)Ljava/lang/Object;
<1231a4f00>     !j9method 0x00000001180C24D8   java/lang/invoke/LambdaForm.interpretName(Ljava/lang/invoke/LambdaForm$Name;[Ljava/lang/Object;)Ljava/lang/Object;
<1231a4f00>     !j9method 0x00000001180C24B8   java/lang/invoke/LambdaForm.interpretWithArguments([Ljava/lang/Object;)Ljava/lang/Object;
<1231a4f00>     !j9method 0x0000000122020B78   java/lang/invoke/LambdaForm$LFI/0x0000000020606200.interpret_L(Ljava/lang/invoke/MethodHandle;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
<1231a4f00>     !j9method 0x0000000123110D98   java/lang/invoke/LambdaForm$MH/0x0000000023110420.linkToTargetMethod(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;
<1231a4f00>     !j9method 0x00000001297CF7A8   net/openj9/test/decimals/TestDecimalData2.<init>()V
<1231a4f00>                             JNI call-in frame
<1231a4f00>     !j9method 0x00000001183FA838   jdk/internal/reflect/NativeConstructorAccessorImpl.newInstance0(Ljava/lang/reflect/Constructor;[Ljava/lang/Object;)Ljava/lang/Object;
<1231a4f00>     !j9method 0x00000001183FA838   jdk/internal/reflect/NativeConstructorAccessorImpl.newInstance0(Ljava/lang/reflect/Constructor;[Ljava/lang/Object;)Ljava/lang/Object;
<1231a4f00>     !j9method 0x00000001183FA7F8   jdk/internal/reflect/NativeConstructorAccessorImpl.newInstance([Ljava/lang/Object;)Ljava/lang/Object;
<1231a4f00>     !j9method 0x00000001183FAD58   jdk/internal/reflect/DelegatingConstructorAccessorImpl.newInstance([Ljava/lang/Object;)Ljava/lang/Object;
<1231a4f00>     !j9method 0x00000001180642D0   java/lang/reflect/Constructor.newInstanceWithCaller([Ljava/lang/Object;ZLjava/lang/Class;)Ljava/lang/Object;
<1231a4f00>     !j9method 0x00000001180642B0   java/lang/reflect/Constructor.newInstance([Ljava/lang/Object;)Ljava/lang/Object;
<1231a4f00>     !j9method 0x0000000158009FA8   org/junit/runners/BlockJUnit4ClassRunner.createTest()Ljava/lang/Object;
(... snip ...)
<1231a4f00>     !j9method 0x0000000128BE1C68   net/adoptopenjdk/loadTest/LoadTestRunner$2.run()V
<1231a4f00>     !j9method 0x00000001291B9290   java/util/concurrent/ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V
<1231a4f00>     !j9method 0x00000001480E4FC0   java/util/concurrent/ThreadPoolExecutor$Worker.run()V
<1231a4f00>     !j9method 0x0000000118077110   java/lang/Thread.run()V
<1231a4f00>                             JNI call-in frame
<1231a4f00>                             Native method frame

@knn-k
Copy link
Contributor

knn-k commented Oct 6, 2022

I mentioned a different failure (HeapHogLoadTest_5m_0) in my previous comment.
I deleted it because it was confusing.

@pshipton pshipton mentioned this issue Mar 6, 2024
Open
@pshipton
Copy link
Member Author

pshipton commented Mar 6, 2024

A possible dup #19081

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
arch:aarch64 os:macos test failure
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@pshipton @knn-k