Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expect test failures on stddev under Windows #293

Open
jmid opened this issue Jul 4, 2024 · 0 comments
Open

Expect test failures on stddev under Windows #293

jmid opened this issue Jul 4, 2024 · 0 comments

Comments

@jmid
Copy link
Collaborator

jmid commented Jul 4, 2024

When experimenting with reenabling dune runtest under Windows (MinGW) in #292 I've noticed that they fail expect tests due to different stddev's being printed.

There's a reported difference with both OCaml 4.14, 5.0, and 5.2:

 stats dist:
-  num: 100000, avg: -7215552342607541.00, stddev: 2666234426234218496.00, median -16620417636667326, min -4611371852367564818, max 4611613630315464842
+  num: 100000, avg: -7215552342607541.00, stddev: 2666234426234218500.00, median -16620417636667326, min -4611371852367564818, max 4611613630315464842

Stddev is computed as follows:

    let stddev =
      Hashtbl.fold
        (fun i res m -> m +. (float_of_int i -. !avg) ** 2. *. float_of_int res)
        tbl 0.
      |> (fun s -> if !num>0 then s /. float_of_int !num else s)
      |> sqrt

and printed as:

    Printf.bprintf out
      "  num: %d, avg: %.2f, stddev: %.2f, median %d, min %d, max %d\n"
      !num !avg stddev !median min_idx max_idx;

I've not dug further into the reason at this point, but suspect that one of the involved floating point operations is causing the difference.

Full details from the 5.2 run:
https://github.com/c-cube/qcheck/actions/runs/9796388729/job/27050729649?pr=292

File "test/core/QCheck2_expect_test.expected", line 1, characters 0-0:
"C:\Program Files\Git\bin\git.exe" --no-pager diff --no-index --color=always -u --ignore-cr-at-eol _build/default/test/core/QCheck2_expect_test.expected _build/default/test/core/QCheck2_expect_test.output
diff --git a/_build/default/test/core/QCheck2_expect_test.expected b/_build/default/test/core/QCheck2_expect_test.output
index bce5b4b..47e0420 100644
--- a/_build/default/test/core/QCheck2_expect_test.expected
+++ b/_build/default/test/core/QCheck2_expect_test.output
@@ -1405,7 +1405,7 @@ stats dist:
 +++ Stats for int dist ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 
 stats dist:
-  num: 100000, avg: -7215552342607541.00, stddev: 2666234426234218496.00, median -16620417636667326, min -4611371852367564818, max 4611613630315464842
+  num: 100000, avg: -7215552342607541.00, stddev: 2666234426234218500.00, median -16620417636667326, min -4611371852367564818, max 4611613630315464842

   -4611371852367564818..-4150222578233413331: #####################################################          5003
   -4150222578233413330..-3689073304099261843: #######################################################        5106
   -3689073304099261842..-3227924029965110355: ######################################################         5052
@@ -1430,7 +1430,7 @@ stats dist:
 +++ Stats for oneof int dist ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 
 stats dist:
-  num: 1000, avg: 4611686018427388.00, stddev: 2891078433912002560.00, median 0, min -4611686018427387904, max 4611686018427387903
+  num: 1000, avg: 4611686018427388.00, stddev: 2891078433912002600.00, median 0, min -4611686018427387904, max 4611686018427387903

   -4611686018427387904..-4150517416584649089: #################                                               198
   -4150517416584649088..-3689348814741910273:                                                                   0
   -3689348814741910272..-3228180212899171457:                                                                   0
@@ -1459,7 +1459,7 @@ random seed: 153870556
 +++ Stats for int_dist_empty_bucket ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 
 stats dist:
-  num: 1000, avg: -38152533987928128.00, stddev: 1828925617669212928.00, median 9, min -4606614955625884935, max 4611686018427387903
+  num: 1000, avg: -38152533987928128.00, stddev: 1828925617669212900.00, median 9, min -4606614955625884935, max 4611686018427387903

   -4606614955625884935..-4145699906923221320: ##                                                               27
   -4145699906923221319..-3684784858220557704: ##                                                               22
   -3684784858220557703..-3223869809517894088: ##                                                               29
File "test/core/QCheck_expect_test.expected", line 1, characters 0-0:
"C:\Program Files\Git\bin\git.exe" --no-pager diff --no-index --color=always -u --ignore-cr-at-eol _build/default/test/core/QCheck_expect_test.expected _build/default/test/core/QCheck_expect_test.output
diff --git a/_build/default/test/core/QCheck_expect_test.expected b/_build/default/test/core/QCheck_expect_test.output
index ed35317..e7f0b76 100644
--- a/_build/default/test/core/QCheck_expect_test.expected
+++ b/_build/default/test/core/QCheck_expect_test.output
@@ -1361,7 +1361,7 @@ stats dist:
 +++ Stats for int dist ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 
 stats dist:
-  num: 100000, avg: -7215552342607541.00, stddev: 2666234426234218496.00, median -16620417636667326, min -4611371852367564818, max 4611613630315464842
+  num: 100000, avg: -7215552342607541.00, stddev: 2666234426234218500.00, median -16620417636667326, min -4611371852367564818, max 4611613630315464842

   -4611371852367564818..-4150222578233413331: #####################################################          5003
   -4150222578233413330..-3689073304099261843: #######################################################        5106
   -3689073304099261842..-3227924029965110355: ######################################################         5052
@@ -1386,7 +1386,7 @@ stats dist:
 +++ Stats for oneof int dist ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 
 stats dist:
-  num: 1000, avg: 4611686018427388.00, stddev: 2891078433912002560.00, median 0, min -4611686018427387904, max 4611686018427387903
+  num: 1000, avg: 4611686018427388.00, stddev: 2891078433912002600.00, median 0, min -4611686018427387904, max 4611686018427387903

   -4611686018427387904..-4150517416584649089: #################                                               198
   -4150517416584649088..-3689348814741910273:                                                                   0
   -3689348814741910272..-3228180212899171457:                                                                   0
@@ -1415,7 +1415,7 @@ random seed: 153870556
 +++ Stats for int_dist_empty_bucket ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
 
 stats dist:
-  num: 1000, avg: -38152533987928128.00, stddev: 1828925617669212928.00, median 9, min -4606614955625884935, max 4611686018427387903
+  num: 1000, avg: -38152533987928128.00, stddev: 1828925617669212900.0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant