Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

SPARSE_seq* tests fail on PowerPC #83

Open
barracuda156 opened this issue Jan 20, 2023 · 17 comments
Open

SPARSE_seq* tests fail on PowerPC #83

barracuda156 opened this issue Jan 20, 2023 · 17 comments

Comments

@barracuda156
Copy link

Everything else passes, SPARSE_seq* all fail:

--->  Testing strumpack
Executing:  cd "/opt/local/var/macports/build/_opt_PPCRosettaPorts_math_strumpack/strumpack/work/build" && ctest test 
Test project /opt/local/var/macports/build/_opt_PPCRosettaPorts_math_strumpack/strumpack/work/build
        Start   1: user_test_HSS_seq
  1/126 Test   #1: user_test_HSS_seq ................   Passed    0.25 sec
        Start   2: user_test_sparse_seq
  2/126 Test   #2: user_test_sparse_seq .............***Exception: SegFault  0.80 sec
        Start   3: user_matrix_IO
  3/126 Test   #3: user_matrix_IO ...................   Passed    1.34 sec
        Start   4: user_test_BLR_seq
  4/126 Test   #4: user_test_BLR_seq ................   Passed    0.19 sec
        Start   5: user_test_HSS_mpi
  5/126 Test   #5: user_test_HSS_mpi ................   Passed    0.15 sec
        Start   6: user_test_sparse_mpi
  6/126 Test   #6: user_test_sparse_mpi .............   Passed    0.05 sec
        Start   7: user_structure_reuse_mpi
  7/126 Test   #7: user_structure_reuse_mpi .........   Passed    0.05 sec
        Start   8: HSS_seq_1
  8/126 Test   #8: HSS_seq_1 ........................   Passed    0.18 sec
        Start   9: HSS_seq_2
  9/126 Test   #9: HSS_seq_2 ........................   Passed    0.30 sec
        Start  10: HSS_seq_3
 10/126 Test  #10: HSS_seq_3 ........................   Passed    0.22 sec
        Start  11: HSS_seq_4
 11/126 Test  #11: HSS_seq_4 ........................   Passed    0.45 sec
        Start  12: HSS_seq_5
 12/126 Test  #12: HSS_seq_5 ........................   Passed    0.53 sec
        Start  13: HSS_seq_6
 13/126 Test  #13: HSS_seq_6 ........................   Passed    0.15 sec
        Start  14: HSS_seq_7
 14/126 Test  #14: HSS_seq_7 ........................   Passed    0.81 sec
        Start  15: HSS_seq_8
 15/126 Test  #15: HSS_seq_8 ........................   Passed    0.28 sec
        Start  16: HSS_seq_9
 16/126 Test  #16: HSS_seq_9 ........................   Passed    0.15 sec
        Start  17: HSS_seq_10
 17/126 Test  #17: HSS_seq_10 .......................   Passed    0.15 sec
        Start  18: HSS_seq_11
 18/126 Test  #18: HSS_seq_11 .......................   Passed    0.33 sec
        Start  19: HSS_seq_12
 19/126 Test  #19: HSS_seq_12 .......................   Passed    1.56 sec
        Start  20: HSS_seq_13
 20/126 Test  #20: HSS_seq_13 .......................   Passed    0.45 sec
        Start  21: HSS_seq_14
 21/126 Test  #21: HSS_seq_14 .......................   Passed    0.29 sec
        Start  22: HSS_seq_15
 22/126 Test  #22: HSS_seq_15 .......................   Passed    0.15 sec
        Start  23: HSS_seq_16
 23/126 Test  #23: HSS_seq_16 .......................   Passed    0.45 sec
        Start  24: HSS_seq_17
 24/126 Test  #24: HSS_seq_17 .......................   Passed    0.16 sec
        Start  25: HSS_seq_18
 25/126 Test  #25: HSS_seq_18 .......................   Passed    0.43 sec
        Start  26: HSS_seq_19
 26/126 Test  #26: HSS_seq_19 .......................   Passed    0.62 sec
        Start  27: HSS_seq_20
 27/126 Test  #27: HSS_seq_20 .......................   Passed    0.24 sec
        Start  28: HSS_seq_21
 28/126 Test  #28: HSS_seq_21 .......................   Passed    0.65 sec
        Start  29: BLR_seq_1
 29/126 Test  #29: BLR_seq_1 ........................   Passed    0.19 sec
        Start  30: BLR_seq_2
 30/126 Test  #30: BLR_seq_2 ........................   Passed    0.19 sec
        Start  31: BLR_seq_3
 31/126 Test  #31: BLR_seq_3 ........................   Passed    0.18 sec
        Start  32: BLR_seq_4
 32/126 Test  #32: BLR_seq_4 ........................   Passed    0.19 sec
        Start  33: BLR_seq_5
 33/126 Test  #33: BLR_seq_5 ........................   Passed    0.19 sec
        Start  34: BLR_seq_6
 34/126 Test  #34: BLR_seq_6 ........................   Passed    0.19 sec
        Start  35: HSS_mpi_1
 35/126 Test  #35: HSS_mpi_1 ........................   Passed    0.05 sec
        Start  36: HSS_mpi_2
 36/126 Test  #36: HSS_mpi_2 ........................   Passed    0.05 sec
        Start  37: HSS_mpi_3
 37/126 Test  #37: HSS_mpi_3 ........................   Passed    0.05 sec
        Start  38: HSS_mpi_4
 38/126 Test  #38: HSS_mpi_4 ........................   Passed    0.05 sec
        Start  39: HSS_mpi_6
 39/126 Test  #39: HSS_mpi_6 ........................   Passed    0.05 sec
        Start  40: HSS_mpi_7
 40/126 Test  #40: HSS_mpi_7 ........................   Passed    0.05 sec
        Start  41: HSS_mpi_8
 41/126 Test  #41: HSS_mpi_8 ........................   Passed    0.05 sec
        Start  42: HSS_mpi_9
 42/126 Test  #42: HSS_mpi_9 ........................   Passed    0.05 sec
        Start  43: HSS_mpi_10
 43/126 Test  #43: HSS_mpi_10 .......................   Passed    0.05 sec
        Start  44: HSS_mpi_11
 44/126 Test  #44: HSS_mpi_11 .......................   Passed    0.06 sec
        Start  45: HSS_mpi_12
 45/126 Test  #45: HSS_mpi_12 .......................   Passed    0.05 sec
        Start  46: HSS_mpi_13
 46/126 Test  #46: HSS_mpi_13 .......................   Passed    0.05 sec
        Start  47: HSS_mpi_14
 47/126 Test  #47: HSS_mpi_14 .......................   Passed    0.05 sec
        Start  48: HSS_mpi_15
 48/126 Test  #48: HSS_mpi_15 .......................   Passed    0.06 sec
        Start  49: HSS_mpi_16
 49/126 Test  #49: HSS_mpi_16 .......................   Passed    0.05 sec
        Start  50: HSS_mpi_17
 50/126 Test  #50: HSS_mpi_17 .......................   Passed    0.05 sec
        Start  51: HSS_mpi_18
 51/126 Test  #51: HSS_mpi_18 .......................   Passed    0.05 sec
        Start  52: HSS_mpi_19
 52/126 Test  #52: HSS_mpi_19 .......................   Passed    0.05 sec
        Start  53: HSS_mpi_20
 53/126 Test  #53: HSS_mpi_20 .......................   Passed    0.05 sec
        Start  54: HSS_mpi_21
 54/126 Test  #54: HSS_mpi_21 .......................   Passed    0.05 sec
        Start  55: HSS_mpi_22
 55/126 Test  #55: HSS_mpi_22 .......................   Passed    0.05 sec
        Start  56: HSS_mpi_23
 56/126 Test  #56: HSS_mpi_23 .......................   Passed    0.05 sec
        Start  57: HSS_mpi_24
 57/126 Test  #57: HSS_mpi_24 .......................   Passed    0.05 sec
        Start  58: HSS_mpi_25
 58/126 Test  #58: HSS_mpi_25 .......................   Passed    0.05 sec
        Start  59: HSS_mpi_26
 59/126 Test  #59: HSS_mpi_26 .......................   Passed    0.05 sec
        Start  60: HSS_mpi_27
 60/126 Test  #60: HSS_mpi_27 .......................   Passed    0.05 sec
        Start  61: HSS_mpi_28
 61/126 Test  #61: HSS_mpi_28 .......................   Passed    0.05 sec
        Start  62: HSS_mpi_29
 62/126 Test  #62: HSS_mpi_29 .......................   Passed    0.05 sec
        Start  63: HSS_mpi_30
 63/126 Test  #63: HSS_mpi_30 .......................   Passed    0.05 sec
        Start  64: HSS_mpi_31
 64/126 Test  #64: HSS_mpi_31 .......................   Passed    0.05 sec
        Start  65: HSS_mpi_32
 65/126 Test  #65: HSS_mpi_32 .......................   Passed    0.05 sec
        Start  66: HSS_mpi_33
 66/126 Test  #66: HSS_mpi_33 .......................   Passed    0.05 sec
        Start  67: HSS_mpi_34
 67/126 Test  #67: HSS_mpi_34 .......................   Passed    0.05 sec
        Start  68: HSS_mpi_35
 68/126 Test  #68: HSS_mpi_35 .......................   Passed    0.05 sec
        Start  69: HSS_mpi_36
 69/126 Test  #69: HSS_mpi_36 .......................   Passed    0.05 sec
        Start  70: HSS_mpi_38
 70/126 Test  #70: HSS_mpi_38 .......................   Passed    0.05 sec
        Start  71: HSS_mpi_39
 71/126 Test  #71: HSS_mpi_39 .......................   Passed    0.05 sec
        Start  72: HSS_mpi_40
 72/126 Test  #72: HSS_mpi_40 .......................   Passed    0.05 sec
        Start  73: HSS_mpi_41
 73/126 Test  #73: HSS_mpi_41 .......................   Passed    0.05 sec
        Start  74: HSS_mpi_42
 74/126 Test  #74: HSS_mpi_42 .......................   Passed    0.05 sec
        Start  75: HSS_mpi_43
 75/126 Test  #75: HSS_mpi_43 .......................   Passed    0.05 sec
        Start  76: HSS_mpi_44
 76/126 Test  #76: HSS_mpi_44 .......................   Passed    0.05 sec
        Start  77: HSS_mpi_45
 77/126 Test  #77: HSS_mpi_45 .......................   Passed    0.05 sec
        Start  78: HSS_mpi_46
 78/126 Test  #78: HSS_mpi_46 .......................   Passed    0.05 sec
        Start  79: HSS_mpi_47
 79/126 Test  #79: HSS_mpi_47 .......................   Passed    0.05 sec
        Start  80: HSS_mpi_48
 80/126 Test  #80: HSS_mpi_48 .......................   Passed    0.05 sec
        Start  81: HSS_mpi_49
 81/126 Test  #81: HSS_mpi_49 .......................   Passed    0.05 sec
        Start  82: HSS_mpi_50
 82/126 Test  #82: HSS_mpi_50 .......................   Passed    0.05 sec
        Start  83: HSS_mpi_51
 83/126 Test  #83: HSS_mpi_51 .......................   Passed    0.05 sec
        Start  84: HSS_mpi_52
 84/126 Test  #84: HSS_mpi_52 .......................   Passed    0.05 sec
        Start  85: HSS_mpi_53
 85/126 Test  #85: HSS_mpi_53 .......................   Passed    0.05 sec
        Start  86: HSS_mpi_54
 86/126 Test  #86: HSS_mpi_54 .......................   Passed    0.05 sec
        Start  87: HSS_mpi_55
 87/126 Test  #87: HSS_mpi_55 .......................   Passed    0.05 sec
        Start  88: Download_sparse_test_matrices
 88/126 Test  #88: Download_sparse_test_matrices ....   Passed   34.51 sec
        Start  89: SPARSE_seq_3
 89/126 Test  #89: SPARSE_seq_3 .....................***Exception: SegFault  1.02 sec
        Start  90: SPARSE_seq_9
 90/126 Test  #90: SPARSE_seq_9 .....................***Exception: SegFault  0.63 sec
        Start  91: SPARSE_seq_15
 91/126 Test  #91: SPARSE_seq_15 ....................***Exception: SegFault  1.64 sec
        Start  92: SPARSE_seq_21
 92/126 Test  #92: SPARSE_seq_21 ....................***Exception: SegFault  4.82 sec
        Start  93: SPARSE_seq_27
 93/126 Test  #93: SPARSE_seq_27 ....................***Exception: SegFault  6.60 sec
        Start  94: SPARSE_seq_30
 94/126 Test  #94: SPARSE_seq_30 ....................***Exception: SegFault  6.77 sec
        Start  95: SPARSE_seq_33
 95/126 Test  #95: SPARSE_seq_33 ....................***Exception: SegFault  0.85 sec
        Start  96: SPARSE_seq_39
 96/126 Test  #96: SPARSE_seq_39 ....................***Exception: SegFault  0.84 sec
        Start  97: SPARSE_seq_44
 97/126 Test  #97: SPARSE_seq_44 ....................***Exception: SegFault  1.00 sec
        Start  98: SPARSE_seq_50
 98/126 Test  #98: SPARSE_seq_50 ....................***Exception: SegFault  0.95 sec
        Start  99: SPARSE_seq_56
 99/126 Test  #99: SPARSE_seq_56 ....................***Exception: SegFault  0.66 sec
        Start 100: SPARSE_seq_66
100/126 Test #100: SPARSE_seq_66 ....................***Failed    2.26 sec
        Start 101: SPARSE_seq_74
101/126 Test #101: SPARSE_seq_74 ....................***Exception: SegFault 16.75 sec
        Start 102: SPARSE_seq_77
102/126 Test #102: SPARSE_seq_77 ....................***Exception: SegFault 18.53 sec
        Start 103: SPARSE_seq_80
103/126 Test #103: SPARSE_seq_80 ....................***Exception: SegFault  0.85 sec
        Start 104: SPARSE_seq_86
104/126 Test #104: SPARSE_seq_86 ....................***Exception: SegFault  0.87 sec
        Start 105: SPARSE_HSS_mpi_1
105/126 Test #105: SPARSE_HSS_mpi_1 .................   Passed    0.05 sec
        Start 106: SPARSE_HSS_mpi_5
106/126 Test #106: SPARSE_HSS_mpi_5 .................   Passed    0.05 sec
        Start 107: SPARSE_HSS_mpi_7
107/126 Test #107: SPARSE_HSS_mpi_7 .................   Passed    0.05 sec
        Start 108: SPARSE_HSS_mpi_8
108/126 Test #108: SPARSE_HSS_mpi_8 .................   Passed    0.05 sec
        Start 109: SPARSE_HSS_mpi_10
109/126 Test #109: SPARSE_HSS_mpi_10 ................   Passed    0.05 sec
        Start 110: SPARSE_HSS_mpi_14
110/126 Test #110: SPARSE_HSS_mpi_14 ................   Passed    0.05 sec
        Start 111: SPARSE_HSS_mpi_19
111/126 Test #111: SPARSE_HSS_mpi_19 ................   Passed    0.05 sec
        Start 112: SPARSE_HSS_mpi_21
112/126 Test #112: SPARSE_HSS_mpi_21 ................   Passed    0.05 sec
        Start 113: structure_reuse_1
113/126 Test #113: structure_reuse_1 ................   Passed    0.05 sec
        Start 114: structure_reuse_2
114/126 Test #114: structure_reuse_2 ................   Passed    0.05 sec
        Start 115: structure_reuse_3
115/126 Test #115: structure_reuse_3 ................   Passed    0.05 sec
        Start 116: structure_reuse_4
116/126 Test #116: structure_reuse_4 ................   Passed    0.05 sec
        Start 117: structure_reuse_5
117/126 Test #117: structure_reuse_5 ................   Passed    0.05 sec
        Start 118: structure_reuse_HSS_1
118/126 Test #118: structure_reuse_HSS_1 ............   Passed    0.05 sec
        Start 119: structure_reuse_HSS_2
119/126 Test #119: structure_reuse_HSS_2 ............   Passed    0.05 sec
        Start 120: structure_reuse_HSS_3
120/126 Test #120: structure_reuse_HSS_3 ............   Passed    0.05 sec
        Start 121: structure_reuse_HSS_4
121/126 Test #121: structure_reuse_HSS_4 ............   Passed    0.05 sec
        Start 122: structure_reuse_HSS_5
122/126 Test #122: structure_reuse_HSS_5 ............   Passed    0.05 sec
        Start 123: SPARSE_mpi_matching_2
123/126 Test #123: SPARSE_mpi_matching_2 ............   Passed    0.05 sec
        Start 124: SPARSE_mpi_matching_3
124/126 Test #124: SPARSE_mpi_matching_3 ............   Passed    0.05 sec
        Start 125: SPARSE_mpi_matching_4
125/126 Test #125: SPARSE_mpi_matching_4 ............   Passed    0.05 sec
        Start 126: SPARSE_mpi_matching_5
126/126 Test #126: SPARSE_mpi_matching_5 ............   Passed    0.05 sec

87% tests passed, 17 tests failed out of 126

Total Test time (real) = 116.28 sec

The following tests FAILED:
	  2 - user_test_sparse_seq (SEGFAULT)
	 89 - SPARSE_seq_3 (SEGFAULT)
	 90 - SPARSE_seq_9 (SEGFAULT)
	 91 - SPARSE_seq_15 (SEGFAULT)
	 92 - SPARSE_seq_21 (SEGFAULT)
	 93 - SPARSE_seq_27 (SEGFAULT)
	 94 - SPARSE_seq_30 (SEGFAULT)
	 95 - SPARSE_seq_33 (SEGFAULT)
	 96 - SPARSE_seq_39 (SEGFAULT)
	 97 - SPARSE_seq_44 (SEGFAULT)
	 98 - SPARSE_seq_50 (SEGFAULT)
	 99 - SPARSE_seq_56 (SEGFAULT)
	100 - SPARSE_seq_66 (Failed)
	101 - SPARSE_seq_74 (SEGFAULT)
	102 - SPARSE_seq_77 (SEGFAULT)
	103 - SPARSE_seq_80 (SEGFAULT)
	104 - SPARSE_seq_86 (SEGFAULT)
Errors while running CTest

10.6.8, ppc32, gcc 12.2.0, OpenBLAS 0.3.21 (linking to vecLibFort seems not to work, therefore had to link to OpenBLAS).

Build logs:
build_log.txt
Tests log:
tests_log.txt
@barracuda156
Copy link
Author

Subsequently, the same happens with HiOp, when it uses STRUMPACK back-end:

      Start 18: NlpSparse1_1
18/32 Test #18: NlpSparse1_1 .....................***Failed    0.68 sec
      Start 19: NlpSparse1_2
19/32 Test #19: NlpSparse1_2 .....................***Failed    0.66 sec
      Start 20: NlpSparse2_1
20/32 Test #20: NlpSparse2_1 .....................***Failed    0.40 sec
      Start 21: NlpSparse2_2
21/32 Test #21: NlpSparse2_2 .....................***Failed    0.38 sec
      Start 22: NlpSparse3_1
22/32 Test #22: NlpSparse3_1 .....................***Failed    0.35 sec
      Start 23: NlpSparseCinterface
23/32 Test #23: NlpSparseCinterface ..............***Failed    0.36 sec

@barracuda156 barracuda156 mentioned this issue Jan 20, 2023
Closed
@pghysels
Copy link
Owner

This is concerning, but I cannot reproduce these segfaults on my machine.
I will try on summit once I get back home.

@barracuda156
Copy link
Author 

--->  Testing strumpack
Executing:  cd "/opt/local/var/macports/build/_opt_PPCRosettaPorts_math_strumpack/strumpack/work/build" && ctest test 
Test project /opt/local/var/macports/build/_opt_PPCRosettaPorts_math_strumpack/strumpack/work/build
        Start   1: user_test_HSS_seq
  1/126 Test   #1: user_test_HSS_seq ................   Passed    0.73 sec
        Start   2: user_test_sparse_seq
  2/126 Test   #2: user_test_sparse_seq .............***Exception: SegFault  2.42 sec
        Start   3: user_matrix_IO
  3/126 Test   #3: user_matrix_IO ...................   Passed    3.18 sec
        Start   4: user_test_BLR_seq
  4/126 Test   #4: user_test_BLR_seq ................   Passed    0.75 sec
        Start   5: user_test_HSS_mpi
  5/126 Test   #5: user_test_HSS_mpi ................   Passed    0.42 sec
        Start   6: user_test_sparse_mpi
  6/126 Test   #6: user_test_sparse_mpi .............   Passed    0.05 sec
        Start   7: user_structure_reuse_mpi
  7/126 Test   #7: user_structure_reuse_mpi .........   Passed    0.06 sec
        Start   8: HSS_seq_1
  8/126 Test   #8: HSS_seq_1 ........................   Passed    0.19 sec
        Start   9: HSS_seq_2
  9/126 Test   #9: HSS_seq_2 ........................   Passed    0.50 sec
        Start  10: HSS_seq_3
 10/126 Test  #10: HSS_seq_3 ........................   Passed    0.21 sec
        Start  11: HSS_seq_4
 11/126 Test  #11: HSS_seq_4 ........................   Passed    0.66 sec
        Start  12: HSS_seq_5
 12/126 Test  #12: HSS_seq_5 ........................   Passed    1.00 sec
        Start  13: HSS_seq_6
 13/126 Test  #13: HSS_seq_6 ........................   Passed    0.19 sec
        Start  14: HSS_seq_7
 14/126 Test  #14: HSS_seq_7 ........................   Passed    0.98 sec
        Start  15: HSS_seq_8
 15/126 Test  #15: HSS_seq_8 ........................   Passed    0.37 sec
        Start  16: HSS_seq_9
 16/126 Test  #16: HSS_seq_9 ........................   Passed    0.16 sec
        Start  17: HSS_seq_10
 17/126 Test  #17: HSS_seq_10 .......................   Passed    0.15 sec
        Start  18: HSS_seq_11
 18/126 Test  #18: HSS_seq_11 .......................   Passed    0.59 sec
        Start  19: HSS_seq_12
 19/126 Test  #19: HSS_seq_12 .......................   Passed    2.95 sec
        Start  20: HSS_seq_13
 20/126 Test  #20: HSS_seq_13 .......................   Passed    1.05 sec
        Start  21: HSS_seq_14
 21/126 Test  #21: HSS_seq_14 .......................   Passed    0.68 sec
        Start  22: HSS_seq_15
 22/126 Test  #22: HSS_seq_15 .......................   Passed    0.16 sec
        Start  23: HSS_seq_16
 23/126 Test  #23: HSS_seq_16 .......................   Passed    0.80 sec
        Start  24: HSS_seq_17
 24/126 Test  #24: HSS_seq_17 .......................   Passed    0.43 sec
        Start  25: HSS_seq_18
 25/126 Test  #25: HSS_seq_18 .......................   Passed    1.39 sec
        Start  26: HSS_seq_19
 26/126 Test  #26: HSS_seq_19 .......................   Passed    1.02 sec
        Start  27: HSS_seq_20
 27/126 Test  #27: HSS_seq_20 .......................   Passed    0.50 sec
        Start  28: HSS_seq_21
 28/126 Test  #28: HSS_seq_21 .......................   Passed    0.98 sec
        Start  29: BLR_seq_1
 29/126 Test  #29: BLR_seq_1 ........................   Passed    0.49 sec
        Start  30: BLR_seq_2
 30/126 Test  #30: BLR_seq_2 ........................   Passed    0.58 sec
        Start  31: BLR_seq_3
 31/126 Test  #31: BLR_seq_3 ........................   Passed    0.73 sec
        Start  32: BLR_seq_4
 32/126 Test  #32: BLR_seq_4 ........................   Passed    0.66 sec
        Start  33: BLR_seq_5
 33/126 Test  #33: BLR_seq_5 ........................   Passed    0.60 sec
        Start  34: BLR_seq_6
 34/126 Test  #34: BLR_seq_6 ........................   Passed    0.51 sec
        Start  35: HSS_mpi_1
 35/126 Test  #35: HSS_mpi_1 ........................   Passed    0.05 sec
        Start  36: HSS_mpi_2
 36/126 Test  #36: HSS_mpi_2 ........................   Passed    0.06 sec
        Start  37: HSS_mpi_3
 37/126 Test  #37: HSS_mpi_3 ........................   Passed    0.05 sec
        Start  38: HSS_mpi_4
 38/126 Test  #38: HSS_mpi_4 ........................   Passed    0.06 sec
        Start  39: HSS_mpi_6
 39/126 Test  #39: HSS_mpi_6 ........................   Passed    0.07 sec
        Start  40: HSS_mpi_7
 40/126 Test  #40: HSS_mpi_7 ........................   Passed    0.06 sec
        Start  41: HSS_mpi_8
 41/126 Test  #41: HSS_mpi_8 ........................   Passed    0.05 sec
        Start  42: HSS_mpi_9
 42/126 Test  #42: HSS_mpi_9 ........................   Passed    0.06 sec
        Start  43: HSS_mpi_10
 43/126 Test  #43: HSS_mpi_10 .......................   Passed    0.05 sec
        Start  44: HSS_mpi_11
 44/126 Test  #44: HSS_mpi_11 .......................   Passed    0.07 sec
        Start  45: HSS_mpi_12
 45/126 Test  #45: HSS_mpi_12 .......................   Passed    0.05 sec
        Start  46: HSS_mpi_13
 46/126 Test  #46: HSS_mpi_13 .......................   Passed    0.06 sec
        Start  47: HSS_mpi_14
 47/126 Test  #47: HSS_mpi_14 .......................   Passed    0.06 sec
        Start  48: HSS_mpi_15
 48/126 Test  #48: HSS_mpi_15 .......................   Passed    0.06 sec
        Start  49: HSS_mpi_16
 49/126 Test  #49: HSS_mpi_16 .......................   Passed    0.06 sec
        Start  50: HSS_mpi_17
 50/126 Test  #50: HSS_mpi_17 .......................   Passed    0.05 sec
        Start  51: HSS_mpi_18
 51/126 Test  #51: HSS_mpi_18 .......................   Passed    0.06 sec
        Start  52: HSS_mpi_19
 52/126 Test  #52: HSS_mpi_19 .......................   Passed    0.05 sec
        Start  53: HSS_mpi_20
 53/126 Test  #53: HSS_mpi_20 .......................   Passed    0.06 sec
        Start  54: HSS_mpi_21
 54/126 Test  #54: HSS_mpi_21 .......................   Passed    0.06 sec
        Start  55: HSS_mpi_22
 55/126 Test  #55: HSS_mpi_22 .......................   Passed    0.05 sec
        Start  56: HSS_mpi_23
 56/126 Test  #56: HSS_mpi_23 .......................   Passed    0.05 sec
        Start  57: HSS_mpi_24
 57/126 Test  #57: HSS_mpi_24 .......................   Passed    0.07 sec
        Start  58: HSS_mpi_25
 58/126 Test  #58: HSS_mpi_25 .......................   Passed    0.07 sec
        Start  59: HSS_mpi_26
 59/126 Test  #59: HSS_mpi_26 .......................   Passed    0.06 sec
        Start  60: HSS_mpi_27
 60/126 Test  #60: HSS_mpi_27 .......................   Passed    0.05 sec
        Start  61: HSS_mpi_28
 61/126 Test  #61: HSS_mpi_28 .......................   Passed    0.07 sec
        Start  62: HSS_mpi_29
 62/126 Test  #62: HSS_mpi_29 .......................   Passed    0.07 sec
        Start  63: HSS_mpi_30
 63/126 Test  #63: HSS_mpi_30 .......................   Passed    0.07 sec
        Start  64: HSS_mpi_31
 64/126 Test  #64: HSS_mpi_31 .......................   Passed    0.06 sec
        Start  65: HSS_mpi_32
 65/126 Test  #65: HSS_mpi_32 .......................   Passed    0.06 sec
        Start  66: HSS_mpi_33
 66/126 Test  #66: HSS_mpi_33 .......................   Passed    0.06 sec
        Start  67: HSS_mpi_34
 67/126 Test  #67: HSS_mpi_34 .......................   Passed    0.05 sec
        Start  68: HSS_mpi_35
 68/126 Test  #68: HSS_mpi_35 .......................   Passed    0.05 sec
        Start  69: HSS_mpi_36
 69/126 Test  #69: HSS_mpi_36 .......................   Passed    0.06 sec
        Start  70: HSS_mpi_38
 70/126 Test  #70: HSS_mpi_38 .......................   Passed    0.06 sec
        Start  71: HSS_mpi_39
 71/126 Test  #71: HSS_mpi_39 .......................   Passed    0.06 sec
        Start  72: HSS_mpi_40
 72/126 Test  #72: HSS_mpi_40 .......................   Passed    0.06 sec
        Start  73: HSS_mpi_41
 73/126 Test  #73: HSS_mpi_41 .......................   Passed    0.06 sec
        Start  74: HSS_mpi_42
 74/126 Test  #74: HSS_mpi_42 .......................   Passed    0.07 sec
        Start  75: HSS_mpi_43
 75/126 Test  #75: HSS_mpi_43 .......................   Passed    0.07 sec
        Start  76: HSS_mpi_44
 76/126 Test  #76: HSS_mpi_44 .......................   Passed    0.06 sec
        Start  77: HSS_mpi_45
 77/126 Test  #77: HSS_mpi_45 .......................   Passed    0.08 sec
        Start  78: HSS_mpi_46
 78/126 Test  #78: HSS_mpi_46 .......................   Passed    0.07 sec
        Start  79: HSS_mpi_47
 79/126 Test  #79: HSS_mpi_47 .......................   Passed    0.06 sec
        Start  80: HSS_mpi_48
 80/126 Test  #80: HSS_mpi_48 .......................   Passed    0.05 sec
        Start  81: HSS_mpi_49
 81/126 Test  #81: HSS_mpi_49 .......................   Passed    0.05 sec
        Start  82: HSS_mpi_50
 82/126 Test  #82: HSS_mpi_50 .......................   Passed    0.05 sec
        Start  83: HSS_mpi_51
 83/126 Test  #83: HSS_mpi_51 .......................   Passed    0.05 sec
        Start  84: HSS_mpi_52
 84/126 Test  #84: HSS_mpi_52 .......................   Passed    0.06 sec
        Start  85: HSS_mpi_53
 85/126 Test  #85: HSS_mpi_53 .......................   Passed    0.05 sec
        Start  86: HSS_mpi_54
 86/126 Test  #86: HSS_mpi_54 .......................   Passed    0.05 sec
        Start  87: HSS_mpi_55
 87/126 Test  #87: HSS_mpi_55 .......................   Passed    0.06 sec
        Start  88: Download_sparse_test_matrices
 88/126 Test  #88: Download_sparse_test_matrices ....   Passed   48.22 sec
        Start  89: SPARSE_seq_3
 89/126 Test  #89: SPARSE_seq_3 .....................***Exception: SegFault  1.64 sec
        Start  90: SPARSE_seq_9
 90/126 Test  #90: SPARSE_seq_9 .....................***Exception: SegFault  1.11 sec
        Start  91: SPARSE_seq_15
 91/126 Test  #91: SPARSE_seq_15 ....................***Exception: SegFault  3.98 sec
        Start  92: SPARSE_seq_21
 92/126 Test  #92: SPARSE_seq_21 ....................***Exception: SegFault  9.51 sec
        Start  93: SPARSE_seq_27
 93/126 Test  #93: SPARSE_seq_27 ....................***Exception: SegFault  7.61 sec
        Start  94: SPARSE_seq_30
 94/126 Test  #94: SPARSE_seq_30 ....................***Exception: SegFault 10.35 sec
        Start  95: SPARSE_seq_33
 95/126 Test  #95: SPARSE_seq_33 ....................***Exception: SegFault  3.26 sec
        Start  96: SPARSE_seq_39
 96/126 Test  #96: SPARSE_seq_39 ....................***Exception: SegFault  1.13 sec
        Start  97: SPARSE_seq_44
 97/126 Test  #97: SPARSE_seq_44 ....................***Exception: SegFault  2.41 sec
        Start  98: SPARSE_seq_50
 98/126 Test  #98: SPARSE_seq_50 ....................***Exception: SegFault  3.47 sec
        Start  99: SPARSE_seq_56
 99/126 Test  #99: SPARSE_seq_56 ....................***Exception: SegFault  2.64 sec
        Start 100: SPARSE_seq_66
100/126 Test #100: SPARSE_seq_66 ....................***Failed    5.92 sec
        Start 101: SPARSE_seq_74
101/126 Test #101: SPARSE_seq_74 ....................***Exception: SegFault 23.82 sec
        Start 102: SPARSE_seq_77
102/126 Test #102: SPARSE_seq_77 ....................***Exception: SegFault 21.30 sec
        Start 103: SPARSE_seq_80
103/126 Test #103: SPARSE_seq_80 ....................***Exception: SegFault  1.07 sec
        Start 104: SPARSE_seq_86
104/126 Test #104: SPARSE_seq_86 ....................***Exception: SegFault  1.21 sec
        Start 105: SPARSE_HSS_mpi_1
105/126 Test #105: SPARSE_HSS_mpi_1 .................   Passed    0.26 sec
        Start 106: SPARSE_HSS_mpi_5
106/126 Test #106: SPARSE_HSS_mpi_5 .................   Passed    0.23 sec
        Start 107: SPARSE_HSS_mpi_7
107/126 Test #107: SPARSE_HSS_mpi_7 .................   Passed    0.13 sec
        Start 108: SPARSE_HSS_mpi_8
108/126 Test #108: SPARSE_HSS_mpi_8 .................   Passed    0.11 sec
        Start 109: SPARSE_HSS_mpi_10
109/126 Test #109: SPARSE_HSS_mpi_10 ................   Passed    0.11 sec
        Start 110: SPARSE_HSS_mpi_14
110/126 Test #110: SPARSE_HSS_mpi_14 ................   Passed    0.17 sec
        Start 111: SPARSE_HSS_mpi_19
111/126 Test #111: SPARSE_HSS_mpi_19 ................   Passed    0.13 sec
        Start 112: SPARSE_HSS_mpi_21
112/126 Test #112: SPARSE_HSS_mpi_21 ................   Passed    0.28 sec
        Start 113: structure_reuse_1
113/126 Test #113: structure_reuse_1 ................   Passed    0.13 sec
        Start 114: structure_reuse_2
114/126 Test #114: structure_reuse_2 ................   Passed    0.12 sec
        Start 115: structure_reuse_3
115/126 Test #115: structure_reuse_3 ................   Passed    0.12 sec
        Start 116: structure_reuse_4
116/126 Test #116: structure_reuse_4 ................   Passed    0.11 sec
        Start 117: structure_reuse_5
117/126 Test #117: structure_reuse_5 ................   Passed    0.14 sec
        Start 118: structure_reuse_HSS_1
118/126 Test #118: structure_reuse_HSS_1 ............   Passed    0.22 sec
        Start 119: structure_reuse_HSS_2
119/126 Test #119: structure_reuse_HSS_2 ............   Passed    0.12 sec
        Start 120: structure_reuse_HSS_3
120/126 Test #120: structure_reuse_HSS_3 ............   Passed    0.11 sec
        Start 121: structure_reuse_HSS_4
121/126 Test #121: structure_reuse_HSS_4 ............   Passed    0.12 sec
        Start 122: structure_reuse_HSS_5
122/126 Test #122: structure_reuse_HSS_5 ............   Passed    0.16 sec
        Start 123: SPARSE_mpi_matching_2
123/126 Test #123: SPARSE_mpi_matching_2 ............   Passed    0.08 sec
        Start 124: SPARSE_mpi_matching_3
124/126 Test #124: SPARSE_mpi_matching_3 ............   Passed    0.10 sec
        Start 125: SPARSE_mpi_matching_4
125/126 Test #125: SPARSE_mpi_matching_4 ............   Passed    0.10 sec
        Start 126: SPARSE_mpi_matching_5
126/126 Test #126: SPARSE_mpi_matching_5 ............   Passed    0.09 sec

87% tests passed, 17 tests failed out of 126

Total Test time (real) = 181.55 sec

The following tests FAILED:
	  2 - user_test_sparse_seq (SEGFAULT)
	 89 - SPARSE_seq_3 (SEGFAULT)
	 90 - SPARSE_seq_9 (SEGFAULT)
	 91 - SPARSE_seq_15 (SEGFAULT)
	 92 - SPARSE_seq_21 (SEGFAULT)
	 93 - SPARSE_seq_27 (SEGFAULT)
	 94 - SPARSE_seq_30 (SEGFAULT)
	 95 - SPARSE_seq_33 (SEGFAULT)
	 96 - SPARSE_seq_39 (SEGFAULT)
	 97 - SPARSE_seq_44 (SEGFAULT)
	 98 - SPARSE_seq_50 (SEGFAULT)
	 99 - SPARSE_seq_56 (SEGFAULT)
	100 - SPARSE_seq_66 (Failed)
	101 - SPARSE_seq_74 (SEGFAULT)
	102 - SPARSE_seq_77 (SEGFAULT)
	103 - SPARSE_seq_80 (SEGFAULT)
	104 - SPARSE_seq_86 (SEGFAULT)
Errors while running CTest

Now we know that OpenBLAS is not to blame here – same failures with vecLibFort (Accelerate).

@barracuda156
Copy link
Author

I have built and enabled scotch, and all scotch-related tests pass. Something goes wrong exclusively with SPARSE_seq.

--->  Testing strumpack
Executing:  cd "/opt/local/var/macports/build/_opt_PPCRosettaPorts_math_strumpack/strumpack/work/build" && ctest test 
Test project /opt/local/var/macports/build/_opt_PPCRosettaPorts_math_strumpack/strumpack/work/build
        Start   1: user_test_HSS_seq
  1/143 Test   #1: user_test_HSS_seq ................   Passed    0.26 sec
        Start   2: user_test_sparse_seq
  2/143 Test   #2: user_test_sparse_seq .............***Exception: SegFault  0.88 sec
        Start   3: user_matrix_IO
  3/143 Test   #3: user_matrix_IO ...................   Passed    2.51 sec
        Start   4: user_test_BLR_seq
  4/143 Test   #4: user_test_BLR_seq ................   Passed    0.47 sec
        Start   5: user_test_HSS_mpi
  5/143 Test   #5: user_test_HSS_mpi ................   Passed    0.14 sec
        Start   6: user_test_sparse_mpi
  6/143 Test   #6: user_test_sparse_mpi .............   Passed    0.06 sec
        Start   7: user_structure_reuse_mpi
  7/143 Test   #7: user_structure_reuse_mpi .........   Passed    0.06 sec
        Start   8: HSS_seq_1
  8/143 Test   #8: HSS_seq_1 ........................   Passed    0.20 sec
        Start   9: HSS_seq_2
  9/143 Test   #9: HSS_seq_2 ........................   Passed    0.45 sec
        Start  10: HSS_seq_3
 10/143 Test  #10: HSS_seq_3 ........................   Passed    0.16 sec
        Start  11: HSS_seq_4
 11/143 Test  #11: HSS_seq_4 ........................   Passed    0.57 sec
        Start  12: HSS_seq_5
 12/143 Test  #12: HSS_seq_5 ........................   Passed    1.04 sec
        Start  13: HSS_seq_6
 13/143 Test  #13: HSS_seq_6 ........................   Passed    0.15 sec
        Start  14: HSS_seq_7
 14/143 Test  #14: HSS_seq_7 ........................   Passed    0.96 sec
        Start  15: HSS_seq_8
 15/143 Test  #15: HSS_seq_8 ........................   Passed    0.37 sec
        Start  16: HSS_seq_9
 16/143 Test  #16: HSS_seq_9 ........................   Passed    0.16 sec
        Start  17: HSS_seq_10
 17/143 Test  #17: HSS_seq_10 .......................   Passed    0.15 sec
        Start  18: HSS_seq_11
 18/143 Test  #18: HSS_seq_11 .......................   Passed    0.40 sec
        Start  19: HSS_seq_12
 19/143 Test  #19: HSS_seq_12 .......................   Passed    2.50 sec
        Start  20: HSS_seq_13
 20/143 Test  #20: HSS_seq_13 .......................   Passed    0.75 sec
        Start  21: HSS_seq_14
 21/143 Test  #21: HSS_seq_14 .......................   Passed    0.74 sec
        Start  22: HSS_seq_15
 22/143 Test  #22: HSS_seq_15 .......................   Passed    0.19 sec
        Start  23: HSS_seq_16
 23/143 Test  #23: HSS_seq_16 .......................   Passed    0.91 sec
        Start  24: HSS_seq_17
 24/143 Test  #24: HSS_seq_17 .......................   Passed    0.16 sec
        Start  25: HSS_seq_18
 25/143 Test  #25: HSS_seq_18 .......................   Passed    0.93 sec
        Start  26: HSS_seq_19
 26/143 Test  #26: HSS_seq_19 .......................   Passed    0.90 sec
        Start  27: HSS_seq_20
 27/143 Test  #27: HSS_seq_20 .......................   Passed    0.56 sec
        Start  28: HSS_seq_21
 28/143 Test  #28: HSS_seq_21 .......................   Passed    1.53 sec
        Start  29: BLR_seq_1
 29/143 Test  #29: BLR_seq_1 ........................   Passed    0.64 sec
        Start  30: BLR_seq_2
 30/143 Test  #30: BLR_seq_2 ........................   Passed    0.43 sec
        Start  31: BLR_seq_3
 31/143 Test  #31: BLR_seq_3 ........................   Passed    0.54 sec
        Start  32: BLR_seq_4
 32/143 Test  #32: BLR_seq_4 ........................   Passed    0.53 sec
        Start  33: BLR_seq_5
 33/143 Test  #33: BLR_seq_5 ........................   Passed    0.49 sec
        Start  34: BLR_seq_6
 34/143 Test  #34: BLR_seq_6 ........................   Passed    0.43 sec
        Start  35: HSS_mpi_1
 35/143 Test  #35: HSS_mpi_1 ........................   Passed    0.05 sec
        Start  36: HSS_mpi_2
 36/143 Test  #36: HSS_mpi_2 ........................   Passed    0.06 sec
        Start  37: HSS_mpi_3
 37/143 Test  #37: HSS_mpi_3 ........................   Passed    0.06 sec
        Start  38: HSS_mpi_4
 38/143 Test  #38: HSS_mpi_4 ........................   Passed    0.05 sec
        Start  39: HSS_mpi_6
 39/143 Test  #39: HSS_mpi_6 ........................   Passed    0.05 sec
        Start  40: HSS_mpi_7
 40/143 Test  #40: HSS_mpi_7 ........................   Passed    0.05 sec
        Start  41: HSS_mpi_8
 41/143 Test  #41: HSS_mpi_8 ........................   Passed    0.06 sec
        Start  42: HSS_mpi_9
 42/143 Test  #42: HSS_mpi_9 ........................   Passed    0.05 sec
        Start  43: HSS_mpi_10
 43/143 Test  #43: HSS_mpi_10 .......................   Passed    0.05 sec
        Start  44: HSS_mpi_11
 44/143 Test  #44: HSS_mpi_11 .......................   Passed    0.05 sec
        Start  45: HSS_mpi_12
 45/143 Test  #45: HSS_mpi_12 .......................   Passed    0.05 sec
        Start  46: HSS_mpi_13
 46/143 Test  #46: HSS_mpi_13 .......................   Passed    0.05 sec
        Start  47: HSS_mpi_14
 47/143 Test  #47: HSS_mpi_14 .......................   Passed    0.06 sec
        Start  48: HSS_mpi_15
 48/143 Test  #48: HSS_mpi_15 .......................   Passed    0.05 sec
        Start  49: HSS_mpi_16
 49/143 Test  #49: HSS_mpi_16 .......................   Passed    0.05 sec
        Start  50: HSS_mpi_17
 50/143 Test  #50: HSS_mpi_17 .......................   Passed    0.05 sec
        Start  51: HSS_mpi_18
 51/143 Test  #51: HSS_mpi_18 .......................   Passed    0.05 sec
        Start  52: HSS_mpi_19
 52/143 Test  #52: HSS_mpi_19 .......................   Passed    0.05 sec
        Start  53: HSS_mpi_20
 53/143 Test  #53: HSS_mpi_20 .......................   Passed    0.05 sec
        Start  54: HSS_mpi_21
 54/143 Test  #54: HSS_mpi_21 .......................   Passed    0.05 sec
        Start  55: HSS_mpi_22
 55/143 Test  #55: HSS_mpi_22 .......................   Passed    0.05 sec
        Start  56: HSS_mpi_23
 56/143 Test  #56: HSS_mpi_23 .......................   Passed    0.05 sec
        Start  57: HSS_mpi_24
 57/143 Test  #57: HSS_mpi_24 .......................   Passed    0.05 sec
        Start  58: HSS_mpi_25
 58/143 Test  #58: HSS_mpi_25 .......................   Passed    0.05 sec
        Start  59: HSS_mpi_26
 59/143 Test  #59: HSS_mpi_26 .......................   Passed    0.05 sec
        Start  60: HSS_mpi_27
 60/143 Test  #60: HSS_mpi_27 .......................   Passed    0.05 sec
        Start  61: HSS_mpi_28
 61/143 Test  #61: HSS_mpi_28 .......................   Passed    0.06 sec
        Start  62: HSS_mpi_29
 62/143 Test  #62: HSS_mpi_29 .......................   Passed    0.07 sec
        Start  63: HSS_mpi_30
 63/143 Test  #63: HSS_mpi_30 .......................   Passed    0.07 sec
        Start  64: HSS_mpi_31
 64/143 Test  #64: HSS_mpi_31 .......................   Passed    0.05 sec
        Start  65: HSS_mpi_32
 65/143 Test  #65: HSS_mpi_32 .......................   Passed    0.05 sec
        Start  66: HSS_mpi_33
 66/143 Test  #66: HSS_mpi_33 .......................   Passed    0.05 sec
        Start  67: HSS_mpi_34
 67/143 Test  #67: HSS_mpi_34 .......................   Passed    0.05 sec
        Start  68: HSS_mpi_35
 68/143 Test  #68: HSS_mpi_35 .......................   Passed    0.05 sec
        Start  69: HSS_mpi_36
 69/143 Test  #69: HSS_mpi_36 .......................   Passed    0.05 sec
        Start  70: HSS_mpi_38
 70/143 Test  #70: HSS_mpi_38 .......................   Passed    0.05 sec
        Start  71: HSS_mpi_39
 71/143 Test  #71: HSS_mpi_39 .......................   Passed    0.05 sec
        Start  72: HSS_mpi_40
 72/143 Test  #72: HSS_mpi_40 .......................   Passed    0.05 sec
        Start  73: HSS_mpi_41
 73/143 Test  #73: HSS_mpi_41 .......................   Passed    0.05 sec
        Start  74: HSS_mpi_42
 74/143 Test  #74: HSS_mpi_42 .......................   Passed    0.05 sec
        Start  75: HSS_mpi_43
 75/143 Test  #75: HSS_mpi_43 .......................   Passed    0.05 sec
        Start  76: HSS_mpi_44
 76/143 Test  #76: HSS_mpi_44 .......................   Passed    0.05 sec
        Start  77: HSS_mpi_45
 77/143 Test  #77: HSS_mpi_45 .......................   Passed    0.05 sec
        Start  78: HSS_mpi_46
 78/143 Test  #78: HSS_mpi_46 .......................   Passed    0.05 sec
        Start  79: HSS_mpi_47
 79/143 Test  #79: HSS_mpi_47 .......................   Passed    0.05 sec
        Start  80: HSS_mpi_48
 80/143 Test  #80: HSS_mpi_48 .......................   Passed    0.05 sec
        Start  81: HSS_mpi_49
 81/143 Test  #81: HSS_mpi_49 .......................   Passed    0.05 sec
        Start  82: HSS_mpi_50
 82/143 Test  #82: HSS_mpi_50 .......................   Passed    0.05 sec
        Start  83: HSS_mpi_51
 83/143 Test  #83: HSS_mpi_51 .......................   Passed    0.05 sec
        Start  84: HSS_mpi_52
 84/143 Test  #84: HSS_mpi_52 .......................   Passed    0.05 sec
        Start  85: HSS_mpi_53
 85/143 Test  #85: HSS_mpi_53 .......................   Passed    0.05 sec
        Start  86: HSS_mpi_54
 86/143 Test  #86: HSS_mpi_54 .......................   Passed    0.05 sec
        Start  87: HSS_mpi_55
 87/143 Test  #87: HSS_mpi_55 .......................   Passed    0.05 sec
        Start  88: Download_sparse_test_matrices
 88/143 Test  #88: Download_sparse_test_matrices ....   Passed   38.14 sec
        Start  89: SPARSE_seq_3
 89/143 Test  #89: SPARSE_seq_3 .....................***Exception: SegFault  0.92 sec
        Start  90: SPARSE_seq_9
 90/143 Test  #90: SPARSE_seq_9 .....................***Exception: SegFault  0.61 sec
        Start  91: SPARSE_seq_15
 91/143 Test  #91: SPARSE_seq_15 ....................***Exception: SegFault  1.58 sec
        Start  92: SPARSE_seq_21
 92/143 Test  #92: SPARSE_seq_21 ....................***Exception: SegFault  5.40 sec
        Start  93: SPARSE_seq_27
 93/143 Test  #93: SPARSE_seq_27 ....................***Exception: SegFault  6.64 sec
        Start  94: SPARSE_seq_30
 94/143 Test  #94: SPARSE_seq_30 ....................***Exception: SegFault  7.21 sec
        Start  95: SPARSE_seq_33
 95/143 Test  #95: SPARSE_seq_33 ....................***Exception: SegFault  0.87 sec
        Start  96: SPARSE_seq_39
 96/143 Test  #96: SPARSE_seq_39 ....................***Exception: SegFault  0.81 sec
        Start  97: SPARSE_seq_44
 97/143 Test  #97: SPARSE_seq_44 ....................***Exception: SegFault  0.96 sec
        Start  98: SPARSE_seq_50
 98/143 Test  #98: SPARSE_seq_50 ....................***Exception: SegFault  0.99 sec
        Start  99: SPARSE_seq_56
 99/143 Test  #99: SPARSE_seq_56 ....................***Exception: SegFault  0.64 sec
        Start 100: SPARSE_seq_66
100/143 Test #100: SPARSE_seq_66 ....................***Failed    2.51 sec
        Start 101: SPARSE_seq_74
101/143 Test #101: SPARSE_seq_74 ....................***Exception: SegFault 19.94 sec
        Start 102: SPARSE_seq_77
102/143 Test #102: SPARSE_seq_77 ....................***Exception: SegFault 18.91 sec
        Start 103: SPARSE_seq_80
103/143 Test #103: SPARSE_seq_80 ....................***Exception: SegFault  0.83 sec
        Start 104: SPARSE_seq_86
104/143 Test #104: SPARSE_seq_86 ....................***Exception: SegFault  0.81 sec
        Start 105: SPARSE_seq_scotch_1
105/143 Test #105: SPARSE_seq_scotch_1 ..............   Passed    1.39 sec
        Start 106: SPARSE_seq_scotch_2
106/143 Test #106: SPARSE_seq_scotch_2 ..............   Passed    0.83 sec
        Start 107: SPARSE_seq_scotch_3
107/143 Test #107: SPARSE_seq_scotch_3 ..............   Passed    3.08 sec
        Start 108: SPARSE_seq_scotch_4
108/143 Test #108: SPARSE_seq_scotch_4 ..............   Passed   11.20 sec
        Start 109: SPARSE_seq_scotch_5
109/143 Test #109: SPARSE_seq_scotch_5 ..............   Passed    1.14 sec
        Start 110: SPARSE_seq_scotch_6
110/143 Test #110: SPARSE_seq_scotch_6 ..............   Passed    1.21 sec
        Start 111: SPARSE_seq_scotch_7
111/143 Test #111: SPARSE_seq_scotch_7 ..............   Passed    1.61 sec
        Start 112: SPARSE_seq_scotch_9
112/143 Test #112: SPARSE_seq_scotch_9 ..............   Passed    0.82 sec
        Start 113: SPARSE_seq_scotch_10
113/143 Test #113: SPARSE_seq_scotch_10 .............   Passed    5.90 sec
        Start 114: SPARSE_seq_scotch_11
114/143 Test #114: SPARSE_seq_scotch_11 .............   Passed    0.99 sec
        Start 115: SPARSE_seq_scotch_12
115/143 Test #115: SPARSE_seq_scotch_12 .............   Passed    1.15 sec
        Start 116: SPARSE_seq_scotch_13
116/143 Test #116: SPARSE_seq_scotch_13 .............   Passed    1.93 sec
        Start 117: SPARSE_HSS_mpi_1
117/143 Test #117: SPARSE_HSS_mpi_1 .................   Passed    0.05 sec
        Start 118: SPARSE_HSS_mpi_5
118/143 Test #118: SPARSE_HSS_mpi_5 .................   Passed    0.05 sec
        Start 119: SPARSE_HSS_mpi_7
119/143 Test #119: SPARSE_HSS_mpi_7 .................   Passed    0.05 sec
        Start 120: SPARSE_HSS_mpi_8
120/143 Test #120: SPARSE_HSS_mpi_8 .................   Passed    0.05 sec
        Start 121: SPARSE_HSS_mpi_10
121/143 Test #121: SPARSE_HSS_mpi_10 ................   Passed    0.05 sec
        Start 122: SPARSE_HSS_mpi_14
122/143 Test #122: SPARSE_HSS_mpi_14 ................   Passed    0.05 sec
        Start 123: SPARSE_HSS_mpi_19
123/143 Test #123: SPARSE_HSS_mpi_19 ................   Passed    0.05 sec
        Start 124: SPARSE_HSS_mpi_21
124/143 Test #124: SPARSE_HSS_mpi_21 ................   Passed    0.05 sec
        Start 125: SPARSE_mpi_13
125/143 Test #125: SPARSE_mpi_13 ....................   Passed    0.05 sec
        Start 126: SPARSE_mpi_16
126/143 Test #126: SPARSE_mpi_16 ....................   Passed    0.05 sec
        Start 127: SPARSE_mpi_18
127/143 Test #127: SPARSE_mpi_18 ....................   Passed    0.05 sec
        Start 128: SPARSE_mpi_22
128/143 Test #128: SPARSE_mpi_22 ....................   Passed    0.05 sec
        Start 129: SPARSE_mpi_24
129/143 Test #129: SPARSE_mpi_24 ....................   Passed    0.05 sec
        Start 130: structure_reuse_1
130/143 Test #130: structure_reuse_1 ................   Passed    0.05 sec
        Start 131: structure_reuse_2
131/143 Test #131: structure_reuse_2 ................   Passed    0.05 sec
        Start 132: structure_reuse_3
132/143 Test #132: structure_reuse_3 ................   Passed    0.05 sec
        Start 133: structure_reuse_4
133/143 Test #133: structure_reuse_4 ................   Passed    0.05 sec
        Start 134: structure_reuse_5
134/143 Test #134: structure_reuse_5 ................   Passed    0.05 sec
        Start 135: structure_reuse_HSS_1
135/143 Test #135: structure_reuse_HSS_1 ............   Passed    0.05 sec
        Start 136: structure_reuse_HSS_2
136/143 Test #136: structure_reuse_HSS_2 ............   Passed    0.05 sec
        Start 137: structure_reuse_HSS_3
137/143 Test #137: structure_reuse_HSS_3 ............   Passed    0.05 sec
        Start 138: structure_reuse_HSS_4
138/143 Test #138: structure_reuse_HSS_4 ............   Passed    0.05 sec
        Start 139: structure_reuse_HSS_5
139/143 Test #139: structure_reuse_HSS_5 ............   Passed    0.05 sec
        Start 140: SPARSE_mpi_matching_2
140/143 Test #140: SPARSE_mpi_matching_2 ............   Passed    0.05 sec
        Start 141: SPARSE_mpi_matching_3
141/143 Test #141: SPARSE_mpi_matching_3 ............   Passed    0.05 sec
        Start 142: SPARSE_mpi_matching_4
142/143 Test #142: SPARSE_mpi_matching_4 ............   Passed    0.05 sec
        Start 143: SPARSE_mpi_matching_5
143/143 Test #143: SPARSE_mpi_matching_5 ............   Passed    0.05 sec

88% tests passed, 17 tests failed out of 143

Total Test time (real) = 164.77 sec

@barracuda156
Copy link
Author

--oversubscribe part is doing some nonsense out there:

45/155 Testing: HSS_mpi_12
45/155 Test: HSS_mpi_12
Command: "/usr/bin/mpiexec" "-n" "17" "--oversubscribe" "/opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_math_strumpack/strumpack/work/build/test/test_HSS_mpi" "T" "1" "--hss_leaf_size" "3" "--hss_rel_tol" "1e-10" "--hss_abs_tol" "1e-10" "--hss_enable_sync" "--hss_compression_algorithm" "stable" "--hss_d0" "64" "--hss_dd" "4"
Directory: /opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_math_strumpack/strumpack/work/build/test
"HSS_mpi_12" start time: Jan 22 17:10 CST
Output:
----------------------------------------------------------
--------------------------------------------------------------------------
Failed to find the following executable:

Host:       31-47.wireless-city.scu.edu.tw
Executable: --oversubscribe

Cannot continue.
--------------------------------------------------------------------------
<end of output>
Test time =   0.19 sec
----------------------------------------------------------
Test Passed.
"HSS_mpi_12" end time: Jan 22 17:10 CST
"HSS_mpi_12" time elapsed: 00:00:00

@barracuda156
Copy link
Author

Apparently that --oversubscribe arg does not work with new MPICH too:

5/155 Testing: user_test_HSS_mpi
5/155 Test: user_test_HSS_mpi
Command: "/opt/local/bin/mpiexec-mpich-gcc12" "-n" "2" "--oversubscribe" "/opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_math_strumpack/strumpack/work/build/test/test_HSS_mpi" "T" "100"
Directory: /opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_math_strumpack/strumpack/work/build/test
"user_test_HSS_mpi" start time: Jan 22 23:16 CST
Output:
----------------------------------------------------------
[[email protected]] match_arg (utils/args/args.c:166): unrecognized argument oversubscribe
[[email protected]] HYDU_parse_array (utils/args/args.c:181): argument matching returned error
[[email protected]] parse_args (ui/mpich/utils.c:1639): error parsing input array
[[email protected]] HYD_uii_mpx_get_parameters (ui/mpich/utils.c:1691): unable to parse user arguments
[[email protected]] main (ui/mpich/mpiexec.c:127): error parsing parameters
<end of output>
Test time =   0.05 sec
----------------------------------------------------------
Test Failed.
"user_test_HSS_mpi" end time: Jan 22 23:16 CST
"user_test_HSS_mpi" time elapsed: 00:00:00

@barracuda156
Copy link
Author

I suspect we were getting false test passes, all those with 0.05 sec pass time.
I tried to rebuild with a number of options, no luck.

--->  Testing strumpack
Executing:  cd "/opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_math_strumpack/strumpack/work/build" && ctest test 
Test project /opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_math_strumpack/strumpack/work/build
        Start   1: user_test_HSS_seq
  1/155 Test   #1: user_test_HSS_seq ................   Passed    0.09 sec
        Start   2: user_test_sparse_seq
  2/155 Test   #2: user_test_sparse_seq .............***Exception: SegFault  5.12 sec
        Start   3: user_matrix_IO
  3/155 Test   #3: user_matrix_IO ...................   Passed    0.86 sec
        Start   4: user_test_BLR_seq
  4/155 Test   #4: user_test_BLR_seq ................   Passed    0.11 sec
        Start   5: user_test_HSS_mpi
  5/155 Test   #5: user_test_HSS_mpi ................***Failed    0.54 sec
        Start   6: user_test_sparse_mpi
  6/155 Test   #6: user_test_sparse_mpi .............***Failed    8.68 sec
        Start   7: user_structure_reuse_mpi
  7/155 Test   #7: user_structure_reuse_mpi .........***Failed    7.00 sec
        Start   8: HSS_seq_1
  8/155 Test   #8: HSS_seq_1 ........................   Passed    0.11 sec
        Start   9: HSS_seq_2
  9/155 Test   #9: HSS_seq_2 ........................   Passed    0.14 sec
        Start  10: HSS_seq_3
 10/155 Test  #10: HSS_seq_3 ........................   Passed    0.08 sec
        Start  11: HSS_seq_4
 11/155 Test  #11: HSS_seq_4 ........................   Passed    0.20 sec
        Start  12: HSS_seq_5
 12/155 Test  #12: HSS_seq_5 ........................   Passed    0.29 sec
        Start  13: HSS_seq_6
 13/155 Test  #13: HSS_seq_6 ........................   Passed    0.08 sec
        Start  14: HSS_seq_7
 14/155 Test  #14: HSS_seq_7 ........................   Passed    0.43 sec
        Start  15: HSS_seq_8
 15/155 Test  #15: HSS_seq_8 ........................   Passed    0.13 sec
        Start  16: HSS_seq_9
 16/155 Test  #16: HSS_seq_9 ........................   Passed    0.08 sec
        Start  17: HSS_seq_10
 17/155 Test  #17: HSS_seq_10 .......................   Passed    0.08 sec
        Start  18: HSS_seq_11
 18/155 Test  #18: HSS_seq_11 .......................   Passed    0.13 sec
        Start  19: HSS_seq_12
 19/155 Test  #19: HSS_seq_12 .......................   Passed    0.98 sec
        Start  20: HSS_seq_13
 20/155 Test  #20: HSS_seq_13 .......................   Passed    0.28 sec
        Start  21: HSS_seq_14
 21/155 Test  #21: HSS_seq_14 .......................   Passed    0.15 sec
        Start  22: HSS_seq_15
 22/155 Test  #22: HSS_seq_15 .......................   Passed    0.08 sec
        Start  23: HSS_seq_16
 23/155 Test  #23: HSS_seq_16 .......................   Passed    0.27 sec
        Start  24: HSS_seq_17
 24/155 Test  #24: HSS_seq_17 .......................   Passed    0.08 sec
        Start  25: HSS_seq_18
 25/155 Test  #25: HSS_seq_18 .......................   Passed    0.24 sec
        Start  26: HSS_seq_19
 26/155 Test  #26: HSS_seq_19 .......................   Passed    0.43 sec
        Start  27: HSS_seq_20
 27/155 Test  #27: HSS_seq_20 .......................   Passed    0.12 sec
        Start  28: HSS_seq_21
 28/155 Test  #28: HSS_seq_21 .......................   Passed    0.44 sec
        Start  29: BLR_seq_1
 29/155 Test  #29: BLR_seq_1 ........................   Passed    0.11 sec
        Start  30: BLR_seq_2
 30/155 Test  #30: BLR_seq_2 ........................   Passed    0.11 sec
        Start  31: BLR_seq_3
 31/155 Test  #31: BLR_seq_3 ........................   Passed    0.11 sec
        Start  32: BLR_seq_4
 32/155 Test  #32: BLR_seq_4 ........................   Passed    0.11 sec
        Start  33: BLR_seq_5
 33/155 Test  #33: BLR_seq_5 ........................   Passed    0.11 sec
        Start  34: BLR_seq_6
 34/155 Test  #34: BLR_seq_6 ........................   Passed    0.11 sec
        Start  35: HSS_mpi_1
 35/155 Test  #35: HSS_mpi_1 ........................***Failed    1.50 sec
        Start  36: HSS_mpi_2
 36/155 Test  #36: HSS_mpi_2 ........................***Failed    1.52 sec
        Start  37: HSS_mpi_3
 37/155 Test  #37: HSS_mpi_3 ........................***Failed    1.40 sec
        Start  38: HSS_mpi_4
 38/155 Test  #38: HSS_mpi_4 ........................***Failed    1.42 sec
        Start  39: HSS_mpi_6
 39/155 Test  #39: HSS_mpi_6 ........................***Failed    0.80 sec
        Start  40: HSS_mpi_7
 40/155 Test  #40: HSS_mpi_7 ........................***Failed    0.31 sec
        Start  41: HSS_mpi_8
 41/155 Test  #41: HSS_mpi_8 ........................***Failed    1.06 sec
        Start  42: HSS_mpi_9
 42/155 Test  #42: HSS_mpi_9 ........................***Failed    0.85 sec
        Start  43: HSS_mpi_10
 43/155 Test  #43: HSS_mpi_10 .......................***Failed    0.31 sec
        Start  44: HSS_mpi_11
 44/155 Test  #44: HSS_mpi_11 .......................***Failed    0.66 sec
        Start  45: HSS_mpi_12
 45/155 Test  #45: HSS_mpi_12 .......................***Failed    0.87 sec
        Start  46: HSS_mpi_13
 46/155 Test  #46: HSS_mpi_13 .......................***Failed    0.99 sec
        Start  47: HSS_mpi_14
 47/155 Test  #47: HSS_mpi_14 .......................***Failed    0.53 sec
        Start  48: HSS_mpi_15
 48/155 Test  #48: HSS_mpi_15 .......................***Failed    0.70 sec
        Start  49: HSS_mpi_16
 49/155 Test  #49: HSS_mpi_16 .......................***Failed    0.67 sec
        Start  50: HSS_mpi_17
 50/155 Test  #50: HSS_mpi_17 .......................***Failed    0.65 sec
        Start  51: HSS_mpi_18
 51/155 Test  #51: HSS_mpi_18 .......................***Failed    0.67 sec
        Start  52: HSS_mpi_19
 52/155 Test  #52: HSS_mpi_19 .......................***Failed    0.76 sec
        Start  53: HSS_mpi_20
 53/155 Test  #53: HSS_mpi_20 .......................***Failed    0.31 sec
        Start  54: HSS_mpi_21
 54/155 Test  #54: HSS_mpi_21 .......................***Failed    0.51 sec
        Start  55: HSS_mpi_22
 55/155 Test  #55: HSS_mpi_22 .......................***Failed    1.71 sec
        Start  56: HSS_mpi_23
 56/155 Test  #56: HSS_mpi_23 .......................***Failed    0.57 sec
        Start  57: HSS_mpi_24
 57/155 Test  #57: HSS_mpi_24 .......................***Failed    0.76 sec
        Start  58: HSS_mpi_25
 58/155 Test  #58: HSS_mpi_25 .......................***Failed    0.69 sec
        Start  59: HSS_mpi_26
 59/155 Test  #59: HSS_mpi_26 .......................***Failed    0.34 sec
        Start  60: HSS_mpi_27
 60/155 Test  #60: HSS_mpi_27 .......................***Failed    0.36 sec
        Start  61: HSS_mpi_28
 61/155 Test  #61: HSS_mpi_28 .......................***Failed    0.85 sec
        Start  62: HSS_mpi_29
 62/155 Test  #62: HSS_mpi_29 .......................***Failed    0.52 sec
        Start  63: HSS_mpi_30
 63/155 Test  #63: HSS_mpi_30 .......................***Failed    0.80 sec
        Start  64: HSS_mpi_31
 64/155 Test  #64: HSS_mpi_31 .......................***Failed    0.65 sec
        Start  65: HSS_mpi_32
 65/155 Test  #65: HSS_mpi_32 .......................***Failed    0.56 sec
        Start  66: HSS_mpi_33
 66/155 Test  #66: HSS_mpi_33 .......................***Failed    0.39 sec
        Start  67: HSS_mpi_34
 67/155 Test  #67: HSS_mpi_34 .......................***Failed    0.32 sec
        Start  68: HSS_mpi_35
 68/155 Test  #68: HSS_mpi_35 .......................***Failed    1.01 sec
        Start  69: HSS_mpi_36
 69/155 Test  #69: HSS_mpi_36 .......................***Failed    0.61 sec
        Start  70: HSS_mpi_38
 70/155 Test  #70: HSS_mpi_38 .......................***Failed    0.31 sec
        Start  71: HSS_mpi_39
 71/155 Test  #71: HSS_mpi_39 .......................***Failed    0.56 sec
        Start  72: HSS_mpi_40
 72/155 Test  #72: HSS_mpi_40 .......................***Failed    0.88 sec
        Start  73: HSS_mpi_41
 73/155 Test  #73: HSS_mpi_41 .......................***Failed    0.49 sec
        Start  74: HSS_mpi_42
 74/155 Test  #74: HSS_mpi_42 .......................***Failed    0.79 sec
        Start  75: HSS_mpi_43
 75/155 Test  #75: HSS_mpi_43 .......................***Failed    0.35 sec
        Start  76: HSS_mpi_44
 76/155 Test  #76: HSS_mpi_44 .......................***Failed    0.37 sec
        Start  77: HSS_mpi_45
 77/155 Test  #77: HSS_mpi_45 .......................***Failed    0.91 sec
        Start  78: HSS_mpi_46
 78/155 Test  #78: HSS_mpi_46 .......................***Failed    0.82 sec
        Start  79: HSS_mpi_47
 79/155 Test  #79: HSS_mpi_47 .......................***Failed    0.86 sec
        Start  80: HSS_mpi_48
 80/155 Test  #80: HSS_mpi_48 .......................***Failed    0.64 sec
        Start  81: HSS_mpi_49
 81/155 Test  #81: HSS_mpi_49 .......................***Failed    1.08 sec
        Start  82: HSS_mpi_50
 82/155 Test  #82: HSS_mpi_50 .......................***Failed    0.81 sec
        Start  83: HSS_mpi_51
 83/155 Test  #83: HSS_mpi_51 .......................***Failed    0.55 sec
        Start  84: HSS_mpi_52
 84/155 Test  #84: HSS_mpi_52 .......................***Failed    0.84 sec
        Start  85: HSS_mpi_53
 85/155 Test  #85: HSS_mpi_53 .......................***Failed    0.60 sec
        Start  86: HSS_mpi_54
 86/155 Test  #86: HSS_mpi_54 .......................***Failed    0.32 sec
        Start  87: HSS_mpi_55
 87/155 Test  #87: HSS_mpi_55 .......................***Failed    0.31 sec
        Start  88: Download_sparse_test_matrices
 88/155 Test  #88: Download_sparse_test_matrices ....   Passed   33.52 sec
        Start  89: SPARSE_seq_3
 89/155 Test  #89: SPARSE_seq_3 .....................***Exception: SegFault  4.63 sec
        Start  90: SPARSE_seq_9
 90/155 Test  #90: SPARSE_seq_9 .....................***Exception: SegFault  4.31 sec
        Start  91: SPARSE_seq_15
 91/155 Test  #91: SPARSE_seq_15 ....................***Exception: SegFault  4.69 sec
        Start  92: SPARSE_seq_21
 92/155 Test  #92: SPARSE_seq_21 ....................***Exception: SegFault  6.20 sec
        Start  93: SPARSE_seq_27
 93/155 Test  #93: SPARSE_seq_27 ....................***Exception: SegFault  6.44 sec
        Start  94: SPARSE_seq_30
 94/155 Test  #94: SPARSE_seq_30 ....................***Exception: SegFault  6.45 sec
        Start  95: SPARSE_seq_33
 95/155 Test  #95: SPARSE_seq_33 ....................***Exception: SegFault  4.33 sec
        Start  96: SPARSE_seq_39
 96/155 Test  #96: SPARSE_seq_39 ....................***Exception: SegFault  4.32 sec
        Start  97: SPARSE_seq_44
 97/155 Test  #97: SPARSE_seq_44 ....................***Exception: SegFault  4.36 sec
        Start  98: SPARSE_seq_50
 98/155 Test  #98: SPARSE_seq_50 ....................***Exception: SegFault  4.35 sec
        Start  99: SPARSE_seq_56
 99/155 Test  #99: SPARSE_seq_56 ....................***Exception: SegFault  4.26 sec
        Start 100: SPARSE_seq_66
100/155 Test #100: SPARSE_seq_66 ....................***Failed    2.23 sec
        Start 101: SPARSE_seq_74
101/155 Test #101: SPARSE_seq_74 ....................***Exception: SegFault  8.95 sec
        Start 102: SPARSE_seq_77
102/155 Test #102: SPARSE_seq_77 ....................***Exception: SegFault  8.83 sec
        Start 103: SPARSE_seq_80
103/155 Test #103: SPARSE_seq_80 ....................***Exception: SegFault  4.32 sec
        Start 104: SPARSE_seq_86
104/155 Test #104: SPARSE_seq_86 ....................***Exception: SegFault  4.32 sec
        Start 105: SPARSE_seq_scotch_1
105/155 Test #105: SPARSE_seq_scotch_1 ..............   Passed    0.32 sec
        Start 106: SPARSE_seq_scotch_2
106/155 Test #106: SPARSE_seq_scotch_2 ..............   Passed    0.18 sec
        Start 107: SPARSE_seq_scotch_3
107/155 Test #107: SPARSE_seq_scotch_3 ..............   Passed    0.68 sec
        Start 108: SPARSE_seq_scotch_4
108/155 Test #108: SPARSE_seq_scotch_4 ..............   Passed    3.40 sec
        Start 109: SPARSE_seq_scotch_5
109/155 Test #109: SPARSE_seq_scotch_5 ..............   Passed    0.23 sec
        Start 110: SPARSE_seq_scotch_6
110/155 Test #110: SPARSE_seq_scotch_6 ..............   Passed    0.26 sec
        Start 111: SPARSE_seq_scotch_7
111/155 Test #111: SPARSE_seq_scotch_7 ..............   Passed    0.32 sec
        Start 112: SPARSE_seq_scotch_9
112/155 Test #112: SPARSE_seq_scotch_9 ..............   Passed    0.18 sec
        Start 113: SPARSE_seq_scotch_10
113/155 Test #113: SPARSE_seq_scotch_10 .............   Passed    1.19 sec
        Start 114: SPARSE_seq_scotch_11
114/155 Test #114: SPARSE_seq_scotch_11 .............   Passed    0.22 sec
        Start 115: SPARSE_seq_scotch_12
115/155 Test #115: SPARSE_seq_scotch_12 .............   Passed    0.25 sec
        Start 116: SPARSE_seq_scotch_13
116/155 Test #116: SPARSE_seq_scotch_13 .............   Passed    0.40 sec
        Start 117: SPARSE_HSS_mpi_1
117/155 Test #117: SPARSE_HSS_mpi_1 .................***Failed   31.39 sec
        Start 118: SPARSE_HSS_mpi_5
118/155 Test #118: SPARSE_HSS_mpi_5 .................***Failed    6.89 sec
        Start 119: SPARSE_HSS_mpi_7
119/155 Test #119: SPARSE_HSS_mpi_7 .................***Failed    8.33 sec
        Start 120: SPARSE_HSS_mpi_8
120/155 Test #120: SPARSE_HSS_mpi_8 .................***Failed   11.70 sec
        Start 121: SPARSE_HSS_mpi_10
121/155 Test #121: SPARSE_HSS_mpi_10 ................***Failed   22.11 sec
        Start 122: SPARSE_HSS_mpi_14
122/155 Test #122: SPARSE_HSS_mpi_14 ................***Failed   23.39 sec
        Start 123: SPARSE_HSS_mpi_19
123/155 Test #123: SPARSE_HSS_mpi_19 ................***Failed   31.43 sec
        Start 124: SPARSE_HSS_mpi_21
124/155 Test #124: SPARSE_HSS_mpi_21 ................***Failed    0.41 sec
        Start 125: SPARSE_mpi_13
125/155 Test #125: SPARSE_mpi_13 ....................   Passed    0.44 sec
        Start 126: SPARSE_mpi_16
126/155 Test #126: SPARSE_mpi_16 ....................***Failed    4.20 sec
        Start 127: SPARSE_mpi_18
127/155 Test #127: SPARSE_mpi_18 ....................   Passed    0.94 sec
        Start 128: SPARSE_mpi_22
128/155 Test #128: SPARSE_mpi_22 ....................***Failed    1.34 sec
        Start 129: SPARSE_mpi_24
129/155 Test #129: SPARSE_mpi_24 ....................   Passed    3.96 sec
        Start 130: SPARSE_mpi_9
130/155 Test #130: SPARSE_mpi_9 .....................***Failed   22.36 sec
        Start 131: SPARSE_mpi_6
131/155 Test #131: SPARSE_mpi_6 .....................***Failed   30.16 sec
        Start 132: SPARSE_mpi_11
132/155 Test #132: SPARSE_mpi_11 ....................***Failed   22.33 sec
        Start 133: SPARSE_mpi_12
133/155 Test #133: SPARSE_mpi_12 ....................***Failed    0.51 sec
        Start 134: SPARSE_mpi_15
134/155 Test #134: SPARSE_mpi_15 ....................***Failed   29.61 sec
        Start 135: SPARSE_mpi_17
135/155 Test #135: SPARSE_mpi_17 ....................***Failed   22.19 sec
        Start 136: SPARSE_mpi_20
136/155 Test #136: SPARSE_mpi_20 ....................***Failed   21.46 sec
        Start 137: SPARSE_mpi_23
137/155 Test #137: SPARSE_mpi_23 ....................***Failed   31.98 sec
        Start 138: SPARSE_mpi_25
138/155 Test #138: SPARSE_mpi_25 ....................***Failed   22.60 sec
        Start 139: SPARSE_mpi_2
139/155 Test #139: SPARSE_mpi_2 .....................***Failed   32.51 sec
        Start 140: SPARSE_mpi_3
140/155 Test #140: SPARSE_mpi_3 .....................***Failed   21.97 sec
        Start 141: SPARSE_mpi_4
141/155 Test #141: SPARSE_mpi_4 .....................***Failed   30.96 sec
        Start 142: structure_reuse_1
142/155 Test #142: structure_reuse_1 ................***Failed   21.11 sec
        Start 143: structure_reuse_2
143/155 Test #143: structure_reuse_2 ................***Failed   21.23 sec
        Start 144: structure_reuse_3
144/155 Test #144: structure_reuse_3 ................***Failed   21.16 sec
        Start 145: structure_reuse_4
145/155 Test #145: structure_reuse_4 ................***Failed   21.21 sec
        Start 146: structure_reuse_5
146/155 Test #146: structure_reuse_5 ................***Failed   10.18 sec
        Start 147: structure_reuse_HSS_1
147/155 Test #147: structure_reuse_HSS_1 ............***Failed   21.56 sec
        Start 148: structure_reuse_HSS_2
148/155 Test #148: structure_reuse_HSS_2 ............***Failed   21.35 sec
        Start 149: structure_reuse_HSS_3
149/155 Test #149: structure_reuse_HSS_3 ............***Failed   21.31 sec
        Start 150: structure_reuse_HSS_4
150/155 Test #150: structure_reuse_HSS_4 ............***Failed   21.35 sec
        Start 151: structure_reuse_HSS_5
151/155 Test #151: structure_reuse_HSS_5 ............***Failed   21.41 sec
        Start 152: SPARSE_mpi_matching_2
152/155 Test #152: SPARSE_mpi_matching_2 ............***Failed    8.94 sec
        Start 153: SPARSE_mpi_matching_3
153/155 Test #153: SPARSE_mpi_matching_3 ............***Failed   16.82 sec
        Start 154: SPARSE_mpi_matching_4
154/155 Test #154: SPARSE_mpi_matching_4 ............***Failed    7.37 sec
        Start 155: SPARSE_mpi_matching_5
155/155 Test #155: SPARSE_mpi_matching_5 ............***Failed   10.87 sec

30% tests passed, 109 tests failed out of 155

Total Test time (real) = 871.49 sec

The following tests FAILED:
	  2 - user_test_sparse_seq (SEGFAULT)
	  5 - user_test_HSS_mpi (Failed)
	  6 - user_test_sparse_mpi (Failed)
	  7 - user_structure_reuse_mpi (Failed)
	 35 - HSS_mpi_1 (Failed)
	 36 - HSS_mpi_2 (Failed)
	 37 - HSS_mpi_3 (Failed)
	 38 - HSS_mpi_4 (Failed)
	 39 - HSS_mpi_6 (Failed)
	 40 - HSS_mpi_7 (Failed)
	 41 - HSS_mpi_8 (Failed)
	 42 - HSS_mpi_9 (Failed)
	 43 - HSS_mpi_10 (Failed)
	 44 - HSS_mpi_11 (Failed)
	 45 - HSS_mpi_12 (Failed)
	 46 - HSS_mpi_13 (Failed)
	 47 - HSS_mpi_14 (Failed)
	 48 - HSS_mpi_15 (Failed)
	 49 - HSS_mpi_16 (Failed)
	 50 - HSS_mpi_17 (Failed)
	 51 - HSS_mpi_18 (Failed)
	 52 - HSS_mpi_19 (Failed)
	 53 - HSS_mpi_20 (Failed)
	 54 - HSS_mpi_21 (Failed)
	 55 - HSS_mpi_22 (Failed)
	 56 - HSS_mpi_23 (Failed)
	 57 - HSS_mpi_24 (Failed)
	 58 - HSS_mpi_25 (Failed)
	 59 - HSS_mpi_26 (Failed)
	 60 - HSS_mpi_27 (Failed)
	 61 - HSS_mpi_28 (Failed)
	 62 - HSS_mpi_29 (Failed)
	 63 - HSS_mpi_30 (Failed)
	 64 - HSS_mpi_31 (Failed)
	 65 - HSS_mpi_32 (Failed)
	 66 - HSS_mpi_33 (Failed)
	 67 - HSS_mpi_34 (Failed)
	 68 - HSS_mpi_35 (Failed)
	 69 - HSS_mpi_36 (Failed)
	 70 - HSS_mpi_38 (Failed)
	 71 - HSS_mpi_39 (Failed)
	 72 - HSS_mpi_40 (Failed)
	 73 - HSS_mpi_41 (Failed)
	 74 - HSS_mpi_42 (Failed)
	 75 - HSS_mpi_43 (Failed)
	 76 - HSS_mpi_44 (Failed)
	 77 - HSS_mpi_45 (Failed)
	 78 - HSS_mpi_46 (Failed)
	 79 - HSS_mpi_47 (Failed)
	 80 - HSS_mpi_48 (Failed)
	 81 - HSS_mpi_49 (Failed)
	 82 - HSS_mpi_50 (Failed)
	 83 - HSS_mpi_51 (Failed)
	 84 - HSS_mpi_52 (Failed)
	 85 - HSS_mpi_53 (Failed)
	 86 - HSS_mpi_54 (Failed)
	 87 - HSS_mpi_55 (Failed)
	 89 - SPARSE_seq_3 (SEGFAULT)
	 90 - SPARSE_seq_9 (SEGFAULT)
	 91 - SPARSE_seq_15 (SEGFAULT)
	 92 - SPARSE_seq_21 (SEGFAULT)
	 93 - SPARSE_seq_27 (SEGFAULT)
	 94 - SPARSE_seq_30 (SEGFAULT)
	 95 - SPARSE_seq_33 (SEGFAULT)
	 96 - SPARSE_seq_39 (SEGFAULT)
	 97 - SPARSE_seq_44 (SEGFAULT)
	 98 - SPARSE_seq_50 (SEGFAULT)
	 99 - SPARSE_seq_56 (SEGFAULT)
	100 - SPARSE_seq_66 (Failed)
	101 - SPARSE_seq_74 (SEGFAULT)
	102 - SPARSE_seq_77 (SEGFAULT)
	103 - SPARSE_seq_80 (SEGFAULT)
	104 - SPARSE_seq_86 (SEGFAULT)
	117 - SPARSE_HSS_mpi_1 (Failed)
	118 - SPARSE_HSS_mpi_5 (Failed)
	119 - SPARSE_HSS_mpi_7 (Failed)
	120 - SPARSE_HSS_mpi_8 (Failed)
	121 - SPARSE_HSS_mpi_10 (Failed)
	122 - SPARSE_HSS_mpi_14 (Failed)
	123 - SPARSE_HSS_mpi_19 (Failed)
	124 - SPARSE_HSS_mpi_21 (Failed)
	126 - SPARSE_mpi_16 (Failed)
	128 - SPARSE_mpi_22 (Failed)
	130 - SPARSE_mpi_9 (Failed)
	131 - SPARSE_mpi_6 (Failed)
	132 - SPARSE_mpi_11 (Failed)
	133 - SPARSE_mpi_12 (Failed)
	134 - SPARSE_mpi_15 (Failed)
	135 - SPARSE_mpi_17 (Failed)
	136 - SPARSE_mpi_20 (Failed)
	137 - SPARSE_mpi_23 (Failed)
	138 - SPARSE_mpi_25 (Failed)
	139 - SPARSE_mpi_2 (Failed)
	140 - SPARSE_mpi_3 (Failed)
	141 - SPARSE_mpi_4 (Failed)
	142 - structure_reuse_1 (Failed)
	143 - structure_reuse_2 (Failed)
	144 - structure_reuse_3 (Failed)
	145 - structure_reuse_4 (Failed)
	146 - structure_reuse_5 (Failed)
	147 - structure_reuse_HSS_1 (Failed)
	148 - structure_reuse_HSS_2 (Failed)
	149 - structure_reuse_HSS_3 (Failed)
	150 - structure_reuse_HSS_4 (Failed)
	151 - structure_reuse_HSS_5 (Failed)
	152 - SPARSE_mpi_matching_2 (Failed)
	153 - SPARSE_mpi_matching_3 (Failed)
	154 - SPARSE_mpi_matching_4 (Failed)
	155 - SPARSE_mpi_matching_5 (Failed)
Errors while running CTest

MPI errors look the same:

Abort(603559686) on node 0 (rank 0 in comm 0): Fatal error in internal_Group_incl: Invalid rank, error stack:
internal_Group_incl(98)..........: MPI_Group_incl(group=0x88000000, n=1, ranks=0xbfffe300, newgroup=0xbfffe2b0) failed
MPIR_Group_check_valid_ranks(238): Invalid rank in rank array at index 0; value is 1 but must be in the range 0 to 0
# tol = 0.0001

# Running with:
# OMP_NUM_THREADS=4 mpirun -n 1 /opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_math_strumpack/strumpack/work/build/test/test_HSS_mpi T 10 --hss_leaf_size 128 --hss_rel_tol 1e-1 --hss_abs_tol 1e-10 --hss_enable_sync --hss_compression_algorithm original --hss_d0 64 --hss_dd 8 
# tol = 0.1
# created H matrix of dimension 10 x 10 with 1 levels
# compression succeeded!
# rank(H) = 0
# memory(H) = 0.001776 MB, 222% of dense
PBLAS ERROR 'DESCB[CTXT_] = -1 does not match other operand's context = 0'
from {0,0}, pnum=0, Contxt=0, in routine 'PDGEMM'.

PBLAS ERROR 'DESCC[CTXT_] = -1 does not match other operand's context = 0'
from {0,0}, pnum=0, Contxt=0, in routine 'PDGEMM'.

PBLAS ERROR 'Parameter number 1402 had an illegal value'
from {0,0}, pnum=0, Contxt=0, in routine 'PDGEMM'.

{0,0}, pnum=0, Contxt=0, killed other procs, exiting with error #-1402.

Abort(-1402) on node 0 (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, -1402) - process 0
<end of output>
Test time =   0.99 sec

This is when running under old MPI, but under new everything looks as bad.

@pghysels
Copy link
Owner

Strange.
Can you just run a simple example?
In the build folder run

make examples
cd examples/sparse
OMP_NUM_THREADS=2 ./testPoisson3d 20

if this fails, can you then run valgrind on it?

@barracuda156
Copy link
Author

Strange.

Can you just run a simple example?

In the build folder run


make examples

cd examples/sparse

OMP_NUM_THREADS=2 ./testPoisson3d 20

if this fails, can you then run valgrind on it?

Been away from my PPC hardware. I will update on this in a couple of days.

Valgrind is not within options though, it is broken on PPC. Apple GDB works.

@barracuda156
Copy link
Author

@pghysels Here it is:

[100%] Building CXX object examples/dense/CMakeFiles/testStructuredMPI.dir/testStructuredMPI.cpp.o
cd /opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_math_strumpack/strumpack/work/build/examples/dense && /opt/local/bin/mpicxx-mpich-gcc12  -I/opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_math_strumpack/strumpack/work/STRUMPACK-5ff29f762c43b7e4475aad26a455e02ae917f7ea/src -I/opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_math_strumpack/strumpack/work/build -isystem /opt/local/include -pipe -Os -DNDEBUG -I/opt/local/include -D_GLIBCXX_USE_CXX11_ABI=0 -arch ppc -mmacosx-version-min=10.6 -Wno-overloaded-virtual -fopenmp -MD -MT examples/dense/CMakeFiles/testStructuredMPI.dir/testStructuredMPI.cpp.o -MF CMakeFiles/testStructuredMPI.dir/testStructuredMPI.cpp.o.d -o CMakeFiles/testStructuredMPI.dir/testStructuredMPI.cpp.o -c /opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_math_strumpack/strumpack/work/STRUMPACK-5ff29f762c43b7e4475aad26a455e02ae917f7ea/examples/dense/testStructuredMPI.cpp
[100%] Linking CXX executable testStructuredMPI
cd /opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_math_strumpack/strumpack/work/build/examples/dense && /opt/local/bin/cmake -E cmake_link_script CMakeFiles/testStructuredMPI.dir/link.txt --verbose=1
/opt/local/bin/mpicxx-mpich-gcc12 -pipe -Os -DNDEBUG -I/opt/local/include -D_GLIBCXX_USE_CXX11_ABI=0 -arch ppc -mmacosx-version-min=10.6 -Wl,-search_paths_first -Wl,-headerpad_max_install_names -L/opt/local/lib -Wl,-headerpad_max_install_names -Wl,-rpath,/opt/local/lib/libgcc CMakeFiles/testStructuredMPI.dir/testStructuredMPI.cpp.o -o testStructuredMPI  -Wl,-rpath,/opt/local/lib ../../libstrumpack.a -latomic /opt/local/lib/gcc12/libgomp.dylib /opt/local/lib/libscalapack.dylib -L/opt/local/lib -lvecLibFort -lvecLibFort /opt/local/lib/libparmetis.dylib /opt/local/lib/libmetis.dylib /opt/local/lib/libscotch.dylib /opt/local/lib/libscotcherr.dylib /opt/local/lib/libscotcherrexit.dylib -lmpifort -lgfortran -lemutls_w 
[100%] Built target testStructuredMPI
make  -f examples/CMakeFiles/examples.dir/build.make examples/CMakeFiles/examples.dir/depend
cd /opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_math_strumpack/strumpack/work/build && /opt/local/bin/cmake -E cmake_depends "Unix Makefiles" /opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_math_strumpack/strumpack/work/STRUMPACK-5ff29f762c43b7e4475aad26a455e02ae917f7ea /opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_math_strumpack/strumpack/work/STRUMPACK-5ff29f762c43b7e4475aad26a455e02ae917f7ea/examples /opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_math_strumpack/strumpack/work/build /opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_math_strumpack/strumpack/work/build/examples /opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_math_strumpack/strumpack/work/build/examples/CMakeFiles/examples.dir/DependInfo.cmake --color=
make  -f examples/CMakeFiles/examples.dir/build.make examples/CMakeFiles/examples.dir/build
make[3]: Nothing to be done for `examples/CMakeFiles/examples.dir/build'.
[100%] Built target examples
/opt/local/bin/cmake -E cmake_progress_start /opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_math_strumpack/strumpack/work/build/CMakeFiles 0
36-167% cd /opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_math_strumpack/strumpack/work/build/examples/sparse 
36-167% OMP_NUM_THREADS=2 ./testPoisson3d 20
testPoisson3d(54153) malloc: *** error for object 0x6174034: Non-aligned pointer being freed
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153) malloc: *** error for object 0x61740a4: Non-aligned pointer being freed
*** set a breakpoint in malloc_error_break to debug
# Solving 3d 20^3 Poisson problem, with 1 right hand sides
# Initializing STRUMPACK
# using 2 OpenMP thread(s)
# number of tasking levels = 4 = log_2(#threads) + 3
# matrix equilibration, r_cond = 1 , c_cond = 1 , type = N
# initial matrix:
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134e2c0: Non-aligned pointer being freed (2)
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d928: Non-aligned pointer being freed
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d890: Non-aligned pointer being freed (2)
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d8dc: Non-aligned pointer being freed
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d7c8: Non-aligned pointer being freed
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d6a0: Non-aligned pointer being freed (2)
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d5c0: Non-aligned pointer being freed (2)
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d630: Non-aligned pointer being freed (2)
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d4f8: Non-aligned pointer being freed
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134e200: non-page-aligned, non-allocated pointer being freed
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134e380: Non-aligned pointer being freed (2)
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134e38c: Non-aligned pointer being freed
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134e3a8: Non-aligned pointer being freed
*** set a breakpoint in malloc_error_break to debug
#   - number of unknowns = 8,000
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d928: Non-aligned pointer being freed
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d890: Non-aligned pointer being freed (2)
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d8dc: Non-aligned pointer being freed
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d7c8: Non-aligned pointer being freed
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d6a0: Non-aligned pointer being freed (2)
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d5c0: Non-aligned pointer being freed (2)
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d630: Non-aligned pointer being freed (2)
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d4f8: Non-aligned pointer being freed
*** set a breakpoint in malloc_error_break to debug
#   - number of nonzeros = 53,600
# nested dissection reordering:
#   - Geometric reordering
#   - strategy parameter = 8
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d928: Non-aligned pointer being freed
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d890: Non-aligned pointer being freed (2)
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d8dc: Non-aligned pointer being freed
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d7c8: Non-aligned pointer being freed
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d6a0: Non-aligned pointer being freed (2)
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d5c0: Non-aligned pointer being freed (2)
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d630: Non-aligned pointer being freed (2)
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d4f8: Non-aligned pointer being freed
*** set a breakpoint in malloc_error_break to debug
#   - number of separators = 853
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d928: Non-aligned pointer being freed
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d890: Non-aligned pointer being freed (2)
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d8dc: Non-aligned pointer being freed
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d7c8: Non-aligned pointer being freed
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d6a0: Non-aligned pointer being freed (2)
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d5c0: Non-aligned pointer being freed (2)
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d630: Non-aligned pointer being freed (2)
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d4f8: Non-aligned pointer being freed
*** set a breakpoint in malloc_error_break to debug
#   - number of levels = 13
#   - nd time = 0.0148621
#   - symmetrization time = 1.90735e-06
# symbolic factorization:
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d928: Non-aligned pointer being freed
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d890: Non-aligned pointer being freed (2)
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d8dc: Non-aligned pointer being freed
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d7c8: Non-aligned pointer being freed
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d6a0: Non-aligned pointer being freed (2)
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d5c0: Non-aligned pointer being freed (2)
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d630: Non-aligned pointer being freed (2)
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d4f8: Non-aligned pointer being freed
*** set a breakpoint in malloc_error_break to debug
#   - nr of dense Frontal matrices = 853
#   - symb-factor time = 0.00635004
# multifrontal factorization:
#   - estimated memory usage (exact solver) = 17.7011 MB
#   - minimum pivot, sqrt(eps)*|A|_1 = 1.26441e-07
#   - replacing of small pivots is not enabled
#   - factor time = 0.487673
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d928: Non-aligned pointer being freed
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d890: Non-aligned pointer being freed (2)
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d8dc: Non-aligned pointer being freed
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d7c8: Non-aligned pointer being freed
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d6a0: Non-aligned pointer being freed (2)
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d5c0: Non-aligned pointer being freed (2)
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d630: Non-aligned pointer being freed (2)
*** set a breakpoint in malloc_error_break to debug
testPoisson3d(54153,0x15e0408) malloc: *** error for object 0x134d4f8: Non-aligned pointer being freed
*** set a breakpoint in malloc_error_break to debug
#   - factor nonzeros = 2,212,642
#   - factor memory = 17.7011 MB
REFINEMENT it. 0	res =      563.706	rel.res =            1	bw.error =            1
REFINEMENT it. 1	res =  2.29502e-13	rel.res =   4.0713e-16	bw.error =  1.49376e-15
# DIRECT/GMRES solve:
#   - abs_tol = 1e-10, rel_tol = 1e-06, restart = 30, maxit = 5000
#   - number of Krylov iterations = 1
#   - solve time = 0.018925
# COMPONENTWISE SCALED RESIDUAL = 1.42468e-15

@barracuda156
Copy link
Author

@pghysels It looks like all MPI tests fail simply because wrong arguments are passed:

42/160 Testing: HSS_mpi_3
42/160 Test: HSS_mpi_3
Command: "/opt/local/bin/mpiexec-mpich-gcc12" "-n" "13" "--oversubscribe" "/opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_math_strumpack/strumpack/work/build/test/test_HSS_mpi" "L" "500" "--hss_leaf_size" "8" "--hss_rel_tol" "1e-4" "--hss_abs_tol" "1e-10" "--hss_enable_sync" "--hss_compression_algorithm" "stable" "--hss_d0" "16" "--hss_dd" "8"
Directory: /opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_math_strumpack/strumpack/work/build/test
"HSS_mpi_3" start time: Apr 18 08:04 CST
Output:
----------------------------------------------------------
[[email protected]] match_arg (lib/utils/args.c:166): unrecognized argument oversubscribe
[[email protected]] HYDU_parse_array (lib/utils/args.c:181): argument matching returned error
[[email protected]] parse_args (mpiexec/get_parameters.c:315): error parsing input array
[[email protected]] HYD_uii_mpx_get_parameters (mpiexec/get_parameters.c:47): unable to parse user arguments
[[email protected]] main (mpiexec/mpiexec.c:49): error parsing parameters
<end of output>
Test time =   0.05 sec
----------------------------------------------------------
Test Failed.
"HSS_mpi_3" end time: Apr 18 08:04 CST
"HSS_mpi_3" time elapsed: 00:00:00

This should be fixable, I guess.

@barracuda156
Copy link
Author

barracuda156 commented May 28, 2023 •
edited
Loading

@pghysels Without MPICH, aren’t there just tolerance threshold issues? On 32-bit precision is expected to be lower, also Darwin uses IBM long double (if that is relevant).

52/67 Testing: SPARSE_seq_80
52/67 Test: SPARSE_seq_80
Command: "/opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_math_strumpack/strumpack/work/build/test/test_sparse_seq" "bcsstm08/bcsstm08.mtx" "--sp_compression" "BLR" "--blr_leaf_size" "4" "--blr_rel_tol" "1e-3" "--blr_abs_tol" "1e-10" "--sp_reordering_method" "metis" "--sp_compression_min_sep_size" "25"
Directory: /opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_math_strumpack/strumpack/work/build/test
"SPARSE_seq_80" start time: May 29 03:07 CST
Output:
----------------------------------------------------------
# Running with:
# OMP_NUM_THREADS=8 /opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_math_strumpack/strumpack/work/build/test/test_sparse_seq bcsstm08/bcsstm08.mtx --sp_compression BLR --blr_leaf_size 4 --blr_rel_tol 1e-3 --blr_abs_tol 1e-10 --sp_reordering_method metis --sp_compression_min_sep_size 25 
# opening file 'bcsstm08/bcsstm08.mtx'
# %%MatrixMarket matrix coordinate real symmetric
# reading 1,074 by 1,074 matrix with 1,074 nnz's from bcsstm08/bcsstm08.mtx
# Initializing STRUMPACK
# using 8 OpenMP thread(s)
# matching job: maximum matching with row and column scaling
# matrix equilibration, r_cond = 1 , c_cond = 1 , type = N
# WARNING: matrix seems to be diagonal!
# initial matrix:
#   - number of unknowns = 1,074
#   - number of nonzeros = 1,074
# nested dissection reordering:
#   - Metis reordering
#      - used METIS_NodeND (iso METIS_NodeNDP)
#      - supernodal tree was built from etree
#   - strategy parameter = 8
#   - number of separators = 2,147
#   - number of levels = 12
#   - nd time = 0.00341606
#   - matching time = 0.00154686
#   - symmetrization time = 0.000610828
# symbolic factorization:
#   - nr of dense Frontal matrices = 2,147
#   - nr of BLR Frontal matrices = 0
#   - symb-factor time = 0.00604701
#   - sep-reorder time = 0.00264883
# multifrontal factorization:
#   - estimated memory usage (exact solver) = 0.008592 MB
#   - minimum pivot, sqrt(eps)*|A|_1 = 1.05367e-08
#   - replacing of small pivots is not enabled
#   - factor time = 0.0204289
#   - factor nonzeros = 1,074
#   - factor memory = 0.008592 MB
#   - compression = blr
#   - factor memory/nonzeros = 100 % of multifrontal
#   - BLR relative compression tolerance = 0.001
#   - BLR absolute compression tolerance = 1e-10
GMRES it. 0	res =  1.77112e+06	rel.res =            1	 restart!
GMRES it. 1	res =  3.93268e-10	rel.res =  2.22045e-16
# DIRECT/GMRES solve:
#   - abs_tol = 1e-10, rel_tol = 1e-06, restart = 30, maxit = 5000
#   - number of Krylov iterations = 1
#   - solve time = 0.0124159
# COMPONENTWISE SCALED RESIDUAL = 2.26379e-16
# RELATIVE ERROR = 2.22038e-16
# opening file 'bcsstm08/bcsstm08.mtx'
# %%MatrixMarket matrix coordinate real symmetric
# reading 1,074 by 1,074 matrix with 1,074 nnz's from bcsstm08/bcsstm08.mtx
# Initializing STRUMPACK
# using 8 OpenMP thread(s)
# matching job: maximum matching with row and column scaling
# matrix equilibration, r_cond = 1 , c_cond = 1 , type = N
# WARNING: matrix seems to be diagonal!
<end of output>
Test time =   4.21 sec
----------------------------------------------------------
Test Failed.
"SPARSE_seq_80" end time: May 29 03:07 CST
"SPARSE_seq_80" time elapsed: 00:00:04
----------------------------------------------------------

53/67 Testing: SPARSE_seq_86
53/67 Test: SPARSE_seq_86
Command: "/opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_math_strumpack/strumpack/work/build/test/test_sparse_seq" "sherman4/sherman4.mtx" "--sp_compression" "BLR" "--blr_leaf_size" "4" "--blr_rel_tol" "1e-3" "--blr_abs_tol" "1e-10" "--sp_reordering_method" "metis" "--sp_compression_min_sep_size" "25"
Directory: /opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_math_strumpack/strumpack/work/build/test
"SPARSE_seq_86" start time: May 29 03:07 CST
Output:
----------------------------------------------------------
# Running with:
# OMP_NUM_THREADS=8 /opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_math_strumpack/strumpack/work/build/test/test_sparse_seq sherman4/sherman4.mtx --sp_compression BLR --blr_leaf_size 4 --blr_rel_tol 1e-3 --blr_abs_tol 1e-10 --sp_reordering_method metis --sp_compression_min_sep_size 25 
# opening file 'sherman4/sherman4.mtx'
# %%MatrixMarket matrix coordinate real general
# reading 1,104 by 1,104 matrix with 3,786 nnz's from sherman4/sherman4.mtx
# Initializing STRUMPACK
# using 8 OpenMP thread(s)
# matching job: maximum matching with row and column scaling
# matrix equilibration, r_cond = 1 , c_cond = 1 , type = N
# initial matrix:
#   - number of unknowns = 1,104
#   - number of nonzeros = 3,786
# nested dissection reordering:
#   - Metis reordering
#      - used METIS_NodeND (iso METIS_NodeNDP)
#      - supernodal tree was built from etree
#   - strategy parameter = 8
#   - number of separators = 1,479
#   - number of levels = 18
#   - nd time = 0.00597405
#   - matching time = 0.00187612
#   - symmetrization time = 0.000459909
# symbolic factorization:
#   - nr of dense Frontal matrices = 1,479
#   - nr of BLR Frontal matrices = 0
#   - symb-factor time = 0.00369096
#   - sep-reorder time = 0.005934
# multifrontal factorization:
#   - estimated memory usage (exact solver) = 0.14808 MB
#   - minimum pivot, sqrt(eps)*|A|_1 = 5.30958e-08
#   - replacing of small pivots is not enabled
#   - factor time = 0.025573
#   - factor nonzeros = 18,510
#   - factor memory = 0.14808 MB
#   - compression = blr
#   - factor memory/nonzeros = 100 % of multifrontal
#   - BLR relative compression tolerance = 0.001
#   - BLR absolute compression tolerance = 1e-10
GMRES it. 0	res =        447.2	rel.res =            1	 restart!
GMRES it. 1	res =  2.58018e-12	rel.res =  5.76963e-15
# DIRECT/GMRES solve:
#   - abs_tol = 1e-10, rel_tol = 1e-06, restart = 30, maxit = 5000
#   - number of Krylov iterations = 1
#   - solve time = 0.016633
# COMPONENTWISE SCALED RESIDUAL = 2.71348e-16
# RELATIVE ERROR = 4.69284e-15
# opening file 'sherman4/sherman4.mtx'
# %%MatrixMarket matrix coordinate real general
# reading 1,104 by 1,104 matrix with 3,786 nnz's from sherman4/sherman4.mtx
# Initializing STRUMPACK
# using 8 OpenMP thread(s)
# matching job: maximum matching with row and column scaling
# matrix equilibration, r_cond = 1 , c_cond = 1 , type = N
<end of output>
Test time =   4.07 sec
----------------------------------------------------------
Test Failed.
"SPARSE_seq_86" end time: May 29 03:08 CST
"SPARSE_seq_86" time elapsed: 00:00:04

Without MPICH results are okayish

72% tests passed, 19 tests failed out of 67

Total Test time (real) = 147.08 sec

The following tests FAILED:
	  2 - user_test_sparse_seq (SEGFAULT)
	 38 - SPARSE_seq_3 (SEGFAULT)
	 39 - SPARSE_seq_9 (SEGFAULT)
	 40 - SPARSE_seq_15 (SEGFAULT)
	 41 - SPARSE_seq_21 (SEGFAULT)
	 42 - SPARSE_seq_27 (SEGFAULT)
	 43 - SPARSE_seq_30 (SEGFAULT)
	 44 - SPARSE_seq_33 (SEGFAULT)
	 45 - SPARSE_seq_39 (SEGFAULT)
	 46 - SPARSE_seq_44 (SEGFAULT)
	 47 - SPARSE_seq_50 (SEGFAULT)
	 48 - SPARSE_seq_56 (SEGFAULT)
	 49 - SPARSE_seq_66 (Failed)
	 50 - SPARSE_seq_74 (SEGFAULT)
	 51 - SPARSE_seq_77 (SEGFAULT)
	 52 - SPARSE_seq_80 (SEGFAULT)
	 53 - SPARSE_seq_86 (SEGFAULT)
	 66 - SPARSE_seq_lossy (SEGFAULT)
	 67 - SPARSE_seq_lossless (SEGFAULT)
Errors while running CTest

UPD. I see it was better initially, I added an unneeded flag now to modify alignments.

@pghysels
Copy link
Owner

I'm sorry for the delay.
I know the oversubscribe option can cause issues, but I didn't see a simple way to check when it needs to be included.
and I didn't spent too much time trying to figure this out since these CMake tests are mostly just for the developers.

Since all the failed tests are for SPARSE_seq*, not for SPARSE_seq_scotch*, that makes me think there is something wrong with the METIS ordering.

You can try your code/test that fails and switch the ordering from METIS to Scotch.
You can run with --sp_reordering_method scotch instead of --sp_reordering_method metis (this is the default).

@barracuda156
Copy link
Author

@pghysels Thank you, I will try.

@catap Since you deal with metis and scotch, may you some idea what may be going wrong here?

@pghysels
Copy link
Owner

METIS can be configured with 32 bit or 64 bit integer support. But that shouldn't make a difference, both should work.

One common source of problems is that scotch installs the metis.h header (and parmetis header) for their compatibility layer. This can mes things up with METIS, but then usually the problems occur during compilation. I typically just remove the metis.h file in the scotch install directory.

@barracuda156
Copy link
Author

@pghysels I have suddenly found crash logs. Does this suggest something?

Process:         test_sparse_seq [27608]
Path:            /opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_math_strumpack/strumpack/work/build/test/test_sparse_seq
Identifier:      test_sparse_seq
Version:         ??? (???)
Code Type:       PPC (Native)
Parent Process:  ctest-orig [27469]

Date/Time:       2023-05-29 03:35:10.068 +0800
OS Version:      Mac OS X 10.6 (10A190)
Report Version:  6

Exception Type:  EXC_BAD_ACCESS (SIGSEGV)
Exception Codes: KERN_PROTECTION_FAILURE at 0x00000000bc000fc0
Crashed Thread:  0

Thread 0 Crashed:
0   test_sparse_seq               	0x000cf880 strumpack::SeparatorTree<long long>::level(long long) const + 36
1   test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
2   test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
3   test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
4   test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
5   test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
6   test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
7   test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
8   test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
9   test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
10  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
11  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
12  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
13  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
14  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
15  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
16  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
17  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
18  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
19  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
20  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
21  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
22  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
23  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
24  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
25  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
26  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
27  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
28  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
29  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
30  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
31  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
32  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
33  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
34  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
35  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
36  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
37  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
38  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
39  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
40  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
41  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
42  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
43  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
44  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
45  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
46  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
47  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
48  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
49  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
50  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
51  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
52  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
53  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
54  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
55  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
56  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
57  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
58  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
59  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
60  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
61  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
62  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
63  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
64  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
65  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
66  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
67  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
68  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
69  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
70  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
71  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
72  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
73  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
74  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
75  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
76  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
77  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
78  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
79  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
80  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
81  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
82  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
83  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
84  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
85  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
86  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
87  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
88  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
89  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
90  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
91  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
92  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
93  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
94  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
95  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
96  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
97  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
98  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
99  test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
100 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
101 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
102 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
103 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
104 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
105 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
106 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
107 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
108 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
109 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
110 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
111 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
112 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
113 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
114 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
115 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
116 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
117 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
118 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
119 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
120 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
121 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
122 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
123 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
124 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
125 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
126 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
127 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
128 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
129 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
130 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
131 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
132 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
133 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
134 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
135 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
136 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
137 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
138 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
139 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
140 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
141 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
142 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
143 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
144 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
145 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
146 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
147 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
148 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
149 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
150 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
151 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
152 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
153 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
154 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
155 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
156 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
157 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
158 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
159 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
160 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
161 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
162 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
163 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
164 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
165 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
166 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
167 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
168 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
169 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
170 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
171 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
172 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
173 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
174 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
175 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
176 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
177 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
178 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
179 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
180 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
181 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
182 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
183 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
184 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
185 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
186 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
187 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
188 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
189 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
190 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
191 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
192 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
193 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
194 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
195 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
196 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
197 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
198 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
199 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
200 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
201 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
202 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
203 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
204 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
205 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
206 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
207 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
208 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
209 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
210 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
211 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
212 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
213 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
214 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
215 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
216 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
217 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
218 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
219 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
220 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
221 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
222 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
223 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
224 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
225 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
226 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
227 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
228 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
229 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
230 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
231 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
232 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
233 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
234 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
235 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
236 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
237 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
238 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
239 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
240 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
241 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
242 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
243 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
244 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
245 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
246 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
247 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
248 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
249 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
250 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
251 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
252 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
253 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
254 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
255 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
256 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
257 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
258 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
259 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
260 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
261 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
262 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
263 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
264 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
265 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
266 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
267 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
268 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
269 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
270 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
271 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
272 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
273 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
274 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
275 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
276 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
277 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
278 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
279 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
280 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
281 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
282 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
283 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
284 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
285 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
286 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
287 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
288 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
289 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
290 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
291 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
292 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
293 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
294 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
295 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
296 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
297 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
298 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
299 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
300 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
301 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
302 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
303 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
304 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
305 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
306 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
307 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
308 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
309 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
310 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
311 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
312 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
313 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
314 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
315 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
316 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
317 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
318 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
319 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
320 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
321 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
322 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
323 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
324 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
325 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
326 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
327 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
328 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
329 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
330 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
331 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
332 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
333 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
334 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
335 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
336 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
337 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
338 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
339 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
340 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
341 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
342 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
343 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
344 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
345 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
346 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
347 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
348 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
349 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
350 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
351 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
352 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
353 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
354 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
355 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
356 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
357 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
358 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
359 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
360 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
361 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
362 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
363 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
364 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
365 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
366 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
367 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
368 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
369 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
370 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
371 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
372 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
373 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
374 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
375 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
376 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
377 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
378 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
379 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
380 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
381 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
382 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
383 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
384 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
385 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
386 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
387 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
388 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
389 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
390 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
391 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
392 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
393 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
394 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
395 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
396 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
397 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
398 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
399 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
400 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
401 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
402 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
403 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
404 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
405 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
406 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
407 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
408 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
409 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
410 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
411 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
412 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
413 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
414 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
415 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
416 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
417 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
418 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
419 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
420 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
421 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
422 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
423 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
424 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
425 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
426 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
427 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
428 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
429 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
430 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
431 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
432 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
433 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
434 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
435 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
436 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
437 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
438 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
439 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
440 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
441 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
442 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
443 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
444 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
445 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
446 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
447 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
448 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
449 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
450 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
451 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
452 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
453 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
454 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
455 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
456 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
457 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
458 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
459 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
460 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
461 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
462 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
463 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
464 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
465 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
466 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
467 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
468 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
469 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
470 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
471 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
472 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
473 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
474 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
475 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
476 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
477 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
478 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
479 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
480 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
481 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
482 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
483 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
484 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
485 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
486 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
487 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
488 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
489 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
490 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
491 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
492 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
493 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
494 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
495 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
496 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
497 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
498 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
499 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
500 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
501 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
502 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
503 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
504 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
505 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
506 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
507 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68
508 test_sparse_seq               	0x000cf8a0 strumpack::SeparatorTree<long long>::level(long long) const + 68

Thread 1:
0   libSystem.B.dylib             	0x008bfe10 __semwait_signal + 12
1   libSystem.B.dylib             	0x008fd3b0 _pthread_cond_wait + 1524
2   libgomp.1.dylib               	0x0050a7bc omp_get_num_procs + 260
3   libgomp.1.dylib               	0x0050aa44 omp_get_num_procs + 908
4   libgomp.1.dylib               	0x00508be0 omp_fulfill_event + 1232
5   libSystem.B.dylib             	0x008fc060 _pthread_start + 320

Thread 2:
0   libSystem.B.dylib             	0x008bfe10 __semwait_signal + 12
1   libSystem.B.dylib             	0x008fd3b0 _pthread_cond_wait + 1524
2   libgomp.1.dylib               	0x0050a7bc omp_get_num_procs + 260
3   libgomp.1.dylib               	0x0050aa44 omp_get_num_procs + 908
4   libgomp.1.dylib               	0x00508be0 omp_fulfill_event + 1232
5   libSystem.B.dylib             	0x008fc060 _pthread_start + 320

Thread 3:
0   libSystem.B.dylib             	0x008bfe10 __semwait_signal + 12
1   libSystem.B.dylib             	0x008fd3b0 _pthread_cond_wait + 1524
2   libgomp.1.dylib               	0x0050a7bc omp_get_num_procs + 260
3   libgomp.1.dylib               	0x0050aa44 omp_get_num_procs + 908
4   libgomp.1.dylib               	0x00508be0 omp_fulfill_event + 1232
5   libSystem.B.dylib             	0x008fc060 _pthread_start + 320

Thread 4:
0   libSystem.B.dylib             	0x008bfe10 __semwait_signal + 12
1   libSystem.B.dylib             	0x008fd3b0 _pthread_cond_wait + 1524
2   libgomp.1.dylib               	0x0050a7bc omp_get_num_procs + 260
3   libgomp.1.dylib               	0x0050aa44 omp_get_num_procs + 908
4   libgomp.1.dylib               	0x00508be0 omp_fulfill_event + 1232
5   libSystem.B.dylib             	0x008fc060 _pthread_start + 320

Thread 5:
0   libSystem.B.dylib             	0x008bfe10 __semwait_signal + 12
1   libSystem.B.dylib             	0x008fd3b0 _pthread_cond_wait + 1524
2   libgomp.1.dylib               	0x0050a7bc omp_get_num_procs + 260
3   libgomp.1.dylib               	0x0050aa44 omp_get_num_procs + 908
4   libgomp.1.dylib               	0x00508be0 omp_fulfill_event + 1232
5   libSystem.B.dylib             	0x008fc060 _pthread_start + 320

Thread 6:
0   libSystem.B.dylib             	0x008bfe10 __semwait_signal + 12
1   libSystem.B.dylib             	0x008fd3b0 _pthread_cond_wait + 1524
2   libgomp.1.dylib               	0x0050a7bc omp_get_num_procs + 260
3   libgomp.1.dylib               	0x0050aa44 omp_get_num_procs + 908
4   libgomp.1.dylib               	0x00508be0 omp_fulfill_event + 1232
5   libSystem.B.dylib             	0x008fc060 _pthread_start + 320

Thread 7:
0   libSystem.B.dylib             	0x008bfe10 __semwait_signal + 12
1   libSystem.B.dylib             	0x008fd3b0 _pthread_cond_wait + 1524
2   libgomp.1.dylib               	0x0050a7bc omp_get_num_procs + 260
3   libgomp.1.dylib               	0x0050aa44 omp_get_num_procs + 908
4   libgomp.1.dylib               	0x00508be0 omp_fulfill_event + 1232
5   libSystem.B.dylib             	0x008fc060 _pthread_start + 320

Thread 0 crashed with PPC Thread State 32:
  srr0: 0x000cf880  srr1: 0x0000d030   dar: 0xbc000fc0 dsisr: 0x42000000
    r0: 0x000cf8a4    r1: 0xbc001020    r2: 0x098e7d18    r3: 0x0940050c
    r4: 0xffffffff    r5: 0x00000000    r6: 0x00026dd2    r7: 0x00000000
    r8: 0xffffffff    r9: 0x098e7d18   r10: 0x098e7d18   r11: 0x00361a84
   r12: 0x000cf85c   r13: 0x00000000   r14: 0x0083cb7c   r15: 0x00021bcb
   r16: 0x00766f10   r17: 0x00000001   r18: 0x00000001   r19: 0x00000001
   r20: 0x00000001   r21: 0x0083cb7c   r22: 0x00000001   r23: 0xbffff078
   r24: 0xbfffeef8   r25: 0x00000000   r26: 0x00000001   r27: 0x00000000
   r28: 0x0940050c   r29: 0x00000000   r30: 0x00000000   r31: 0x00180874
    cr: 0x24000242   xer: 0x00000000    lr: 0x000cf8a4   ctr: 0x000cf85c
vrsave: 0x00000000

Binary Images:
    0x1000 -   0x35dffb +test_sparse_seq ??? (???) <1c202e4776b5bf76d793607488cae851> /opt/local/var/macports/build/_opt_PPCSnowLeopardPorts_math_strumpack/strumpack/work/build/test/test_sparse_seq
  0x4f5000 -   0x4f7fff +libatomic.1.dylib ??? (???) <e027bd33ead85ee6229c3a8c013f1d0c> /opt/local/lib/libgcc/libatomic.1.dylib
  0x4fb000 -   0x523ffb +libgomp.1.dylib ??? (???) <ff1d6b3cc96963b1397be079f45968fb> /opt/local/lib/libgcc/libgomp.1.dylib
  0x52a000 -   0x52effd +libvecLibFort.dylib ??? (???) <a6be71026ff5778cb67a4f1b010d4536> /opt/local/lib/libvecLibFort.dylib
  0x531000 -   0x570ff3 +libmetis.dylib ??? (???) <6831a8e7e6407ca379ad1d6339698257> /opt/local/lib/libmetis.dylib
  0x576000 -   0x5edfff +libscotch.dylib ??? (???) <a371e8ef8f4837f33daa5d7371204117> /opt/local/lib/libscotch.dylib
  0x600000 -   0x602ffd +libscotcherr.dylib ??? (???) <5ed310a873ae5375183c25dee7baec04> /opt/local/lib/libscotcherr.dylib
  0x605000 -   0x607ffd +libscotcherrexit.dylib ??? (???) <e0ba2a109eb9f2df24ac26f48234a562> /opt/local/lib/libscotcherrexit.dylib
  0x60a000 -   0x6ebffb +libgfortran.5.dylib ??? (???) <1e9914b4b48543c116f0041e231ae239> /opt/local/lib/libgcc/libgfortran.5.dylib
  0x6fb000 -   0x825ffb +libstdc++.6.dylib ??? (???) <3f2ee93ca894f1a15aa04aa6389e3f43> /opt/local/lib/libgcc/libstdc++.6.dylib
  0x8ad000 -   0x8b2ff7 +libgcc_s.1.1.dylib ??? (???) <bb94c84e3458931279bac16912cebb07> /opt/local/lib/libgcc/libgcc_s.1.1.dylib
  0x8b5000 -   0x8b5fff +libgcc_s.1.dylib ??? (???) <52f8e6b93cfbb9fa40f44e2651069377> /opt/local/lib/libgcc/libgcc_s.1.dylib
  0x8b8000 -   0xa82fe7  libSystem.B.dylib ??? (???) <420b38fe0fa0944f6f5bc4840bf29a4e> /usr/lib/libSystem.B.dylib
  0xb0a000 -   0xb29fff  com.apple.vecLib 3.4.2 (vecLib 3.4.2) <9e27768a4224789f73942ddb2e03e729> /System/Library/Frameworks/vecLib.framework/Versions/A/vecLib
  0xb86000 -   0xb8bff6  libmathCommon.A.dylib ??? (???) <e166c8e59a80a82b4504b1dd43d1f50a> /usr/lib/system/libmathCommon.A.dylib
  0xb8f000 -   0xb8ffff  com.apple.Accelerate 1.4.2 (Accelerate 1.4.2) <e8c929e9ed651c210e5238b66b947e6e> /System/Library/Frameworks/Accelerate.framework/Versions/A/Accelerate
  0xb93000 -   0xcb3ffa  com.apple.vImage 4.0 (4.0) <40c9ff1a236e54afc0af0dbe01d59ae0> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vImage.framework/Versions/A/vImage
  0xcc2000 -   0xce1fff  com.apple.Accelerate.vecLib 3.4.2 (vecLib 3.4.2) <c56e0d707296e54bceac01b6944e99e3> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/vecLib
  0xd3e000 -   0xd43fff +libgcc_ehs.1.1.dylib ??? (???) <f0044cab1e11d85042f936426a994ecd> /opt/local/lib/libgcc/libgcc_ehs.1.1.dylib
  0xd46000 -   0xdc0fff  libvMisc.dylib ??? (???) <2607b58a270aca106281054e078194d1> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libvMisc.dylib
  0xdca000 -   0xe67fe3  libvDSP.dylib ??? (???) <18d61ac588914626b88c2c2babb2ac25> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libvDSP.dylib
  0xe81000 -  0x14eeff7  libBLAS.dylib ??? (???) <b09754bb15f771807655be04f5200efd> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libBLAS.dylib
 0x152c000 -  0x18c4ff7  libLAPACK.dylib ??? (???) <7f1ccd1f986607cb24ada410c3548482> /System/Library/Frameworks/Accelerate.framework/Versions/A/Frameworks/vecLib.framework/Versions/A/libLAPACK.dylib
 0x1a3d000 -  0x1a85fff +libGKlib.dylib ??? (???) <634ca7213d232cd86d903e72a003c3a3> /opt/local/lib/libGKlib.dylib
 0x1a8e000 -  0x1aa1ffc +libz.1.dylib ??? (???) <c697fd0e0e4f9423b001dfffce0f46c3> /opt/local/lib/libz.1.dylib
 0x1aa5000 -  0x1ab3ff5 +libbz2.1.0.dylib ??? (???) <561eabf373907c3899487650867150dc> /opt/local/lib/libbz2.1.0.dylib
 0x1ab7000 -  0x1addffc +liblzma.5.dylib ??? (???) <bd86be9fe9377771f443b62e4fab140e> /opt/local/lib/liblzma.5.dylib
 0x1ae3000 -  0x1decffb +libmpi.12.dylib ??? (???) <71e097b52ecefccabaa192983e49e866> /opt/local/lib/mpich-gcc12/libmpi.12.dylib
 0x1e12000 -  0x1e4bfff +libhwloc.15.dylib ??? (???) <fb2a807dfa53cf1d3a23949675c17022> /opt/local/lib/libhwloc.15.dylib
 0x1e54000 -  0x1e59fff +libMacportsLegacySupport.dylib ??? (???) <866c1cae762ea8789473e05e97b26a5b> /opt/local/lib/libMacportsLegacySupport.dylib
 0x1e5d000 -  0x20b5fff  com.apple.Foundation 6.6 (711.1) <42ee053ddba592040925af9eb9542462> /System/Library/Frameworks/Foundation.framework/Versions/C/Foundation
 0x21ff000 -  0x2255ff3  com.apple.framework.IOKit 1.5.1 (???) <21f9b751b7578a33306804af2c6ded8d> /System/Library/Frameworks/IOKit.framework/Versions/A/IOKit
 0x226e000 -  0x2381ffb +libxml2.2.dylib ??? (???) <969d6acadb675c1c104754f64f383a92> /opt/local/lib/libxml2.2.dylib
 0x23a8000 -  0x23dbff1  libauto.dylib ??? (???) <2848423526fb9c0f381c6ceacc4f2c27> /usr/lib/libauto.dylib
 0x23e7000 -  0x2468fe7  libobjc.A.dylib ??? (???) <bb09ce82013fa9c4a4010c58fe6c1e10> /usr/lib/libobjc.A.dylib
 0x247a000 -  0x2488fff  libz.1.dylib ??? (???) <c296d00fd65aa7271f3d5592c25c0f96> /usr/lib/libz.1.dylib
 0x248d000 -  0x24c8fff  com.apple.SystemConfiguration 1.10 (1.10) <2daf422a1a7ca90ba32720000e56b3aa> /System/Library/Frameworks/SystemConfiguration.framework/Versions/A/SystemConfiguration
 0x24e4000 -  0x24e4ffa  com.apple.CoreServices 41 (41) <97bf3903384c706ba0f7d19577f0ef8c> /System/Library/Frameworks/CoreServices.framework/Versions/A/CoreServices
 0x27f7000 -  0x4cfcd07 +libpmpi.12.dylib ??? (???) <a3f4074e0a4c6c65d0c606783a3ac28d> /opt/local/lib/mpich-gcc12/libpmpi.12.dylib
 0x4fee000 -  0x5146ff7  com.apple.CoreFoundation 6.6 (511.1) <b82d52070a5949f92c5001e06d67044b> /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation
 0x5250000 -  0x53d9fff  libicucore.A.dylib ??? (???) <28f339176d9e26b663f19c1cad91784e> /usr/lib/libicucore.A.dylib
 0x5429000 -  0x5544fff  libxml2.2.dylib ??? (???) <2bdb2ca8d4aabba550ba9f2111ebf46a> /usr/lib/libxml2.2.dylib
 0x5567000 -  0x5627ffb  com.apple.CFNetwork 417.1 (417.1) <911e0fa87591b77e8bb8ec4759941216> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CFNetwork.framework/Versions/A/CFNetwork
 0x568f000 -  0x58bdff7  com.apple.security 6.0 (34772) <df4f0710dfee6e04e7fdba557064977b> /System/Library/Frameworks/Security.framework/Versions/A/Security
 0x59d4000 -  0x5ad9fff +libiconv.2.dylib ??? (???) <71dfab8f77eb71baebdb38550a0ed408> /opt/local/lib/libiconv.2.dylib
 0x5ae8000 -  0x5b99ff7  libsqlite3.0.dylib ??? (???) <cc6135a5433e43988bf825e149e47483> /usr/lib/libsqlite3.0.dylib
 0x5ba5000 -  0x5ecbff7  com.apple.CoreServices.CarbonCore 818 (818) <3e631022aef7f9d15373e1f56e74ff00> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore
 0x5f54000 -  0x5f94fff  com.apple.Metadata 10.6.0 (429.1) <d8b50a8ca605800f8f36caf448daed42> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/Metadata.framework/Versions/A/Metadata
 0x5fad000 -  0x6085fff  com.apple.CoreServices.OSServices 310 (310) <633ea4081155a516c80bf948f004d235> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/OSServices.framework/Versions/A/OSServices
 0x60f0000 -  0x616dffb  com.apple.SearchKit 1.3.0 (1.3.0) <4c7ffa79c1cd48d28eb165768fbfc2f4> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/SearchKit.framework/Versions/A/SearchKit
 0x61ac000 -  0x61e4fff  com.apple.AE 464 (464) <492b7cdb81e64fca5af300b0c231ebec> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/AE.framework/Versions/A/AE
 0x61fc000 -  0x6293ff7  com.apple.LaunchServices 318.1 (318.1) <5ee43f629f636d1863e73dc312c736c2> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/LaunchServices.framework/Versions/A/LaunchServices
 0x62d9000 -  0x62fcfff  com.apple.DictionaryServices 1.0.0 (1.0.0) <f2f1f85469f3036ec9890bfa888edbe2> /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/DictionaryServices.framework/Versions/A/DictionaryServices
 0x6312000 -  0x631bfff  com.apple.DiskArbitration 2.2.1 (2.2.1) <daa422fc5a934366c8f0a12204b46cb1> /System/Library/Frameworks/DiskArbitration.framework/Versions/A/DiskArbitration
 0x6324000 -  0x6336fff  libbsm.0.dylib ??? (???) <c43c47af4211639e8828d3e8ff93a019> /usr/lib/libbsm.0.dylib
 0x633c000 -  0x6349ffe  NetFS ??? (???) <5125b2c825637b243afa1ea942ba8220> /System/Library/Frameworks/NetFS.framework/Versions/A/NetFS
 0x6350000 -  0x6359ffc  libkxld.dylib ??? (???) <725cd6e644549962b0110493f4bb7c08> /usr/lib/system/libkxld.dylib
 0x635d000 -  0x6389fff  libxslt.1.dylib ??? (???) <da4552da93bf856897f6b399bb10806d> /usr/lib/libxslt.1.dylib
 0x6392000 -  0x6556fff +libicui18n.73.dylib ??? (???) <b242870b53a010b97e7faefc790f3af7> /opt/local/lib/libicui18n.73.dylib
 0x6649000 -  0x679affb +libicuuc.73.dylib ??? (???) <a03342215f999c02ebd90a92bbd8ff63> /opt/local/lib/libicuuc.73.dylib
 0x6804000 -  0x8688fff +libicudata.73.dylib ??? (???) <4ba02aec27f6e07f46881a1700462d95> /opt/local/lib/libicudata.73.dylib
0x8fe00000 - 0x8fe31143  dyld 113.0 (???) <ee555e655e1cf66cac80728801add419> /usr/lib/dyld
0xffff8000 - 0xffff9703  libSystem.B.dylib ??? (???) /usr/lib/libSystem.B.dylib

@pghysels
Copy link
Owner

There is a recursive function to compute the number of levels in the tree. The number of levels is too large (at least 508) so the code probably crashes with a stack overflow. We should avoid this recursion. But the tree should ideally only have ~log(N) levels, so something is wrong with the tree. The tree is determined by the matrix sparsity pattern and the matrix ordering (which is computed by METIS, or scotch).
But from the output you posted above (3 weeks ago), you can see the number of levels is 12 (for bcsstm08), so that's good. But the test is run twice, once with 32 bit integers, and then again with 64 bit integers. And it seems to fail the second time. So there is something wrong with the matrix ordering I think when using 64 bit integers. But I cannot reproduce it here on my machine.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@pghysels @barracuda156