Can't run tests in CI: TestItemRunner.compute_line_column(content, 1) == (line = 1, column = 1) error

[ianfiske]

I can use TestItemRunner locally fine. It works both in VSCode's UI and also doing ] test in my package using a basic runtests.jl

using TestItemRunner

@run_package_tests

However, when I run in our CI system, the tests fail because it's trying to run a test from within TestItemRunner itself.

  Test threw exception
  Expression: TestItemRunner.compute_line_column(content, 1) == (line = 1, column = 1)
  UndefVarError: `TestItemRunner` not defined
  Stacktrace:
   [1] macro expansion
     @ /usr/local/julia/share/julia/stdlib/v1.10/Test/src/Test.jl:669 [inlined]
   [2] top-level scope
     @ /builds/internal/MyPackage.jl/.julia/packages/TestItemRunner/Qioo0/src/TestItemRunner.jl:508

I tried adding a filter to ensure it wouldn't pick up tests in TestItemRunner:

using TestItemRunner

# Filter necessary to avoid some Gitlab errors where it
# was detecting tests in TestItemRunner itself.
@run_package_tests filter = ti -> contains(dirname(ti.filename), "MyPackage")

but this didn't work. Any ideas?

[ianfiske]

In the (abridged) report, I see

Test Summary:                                                | Pass  Error  Total   Time
Package                                                      |   57      9     66  36.6s
  test/basic_tests.jl                                        |   16            16  22.4s
  test/components.jl                                         |   29            29   9.1s
  .julia/packages/TestItemRunner/Qioo0/src/TestItemRunner.jl |           9      9   2.1s
    compute_line_column                                      |           9      9   2.1s

so TestItemRunner is clearly picking up tests within its own source code.

[ianfiske]

I was able to work around this and get tests to pass in CI with this

@run_package_tests filter = ti -> !contains(dirname(ti.filename), "TestItemRunner")

but I don't understand why it's picking these tests up in the first place and the workaround is needed.

[davidanthoff]

It looks to me as if the Julia depot being used is a child folder of the package your are testing? This path /builds/internal/MyPackage.jl/.julia/packages/TestItemRunner/Qioo0/src/TestItemRunner.jl suggests that.

So what happens then is that all the packages that you are using are cloned into MyPackage.jl/.julia/packages etc and in general the test item framework picks up any test items that are in a Julia folder that is a sub folder of MyPackage.

Is there a reason why the depot ends up being there? That is a bit of a weird choice ;)

[davidanthoff]

But apart from that, if we fix #41 then this should also address the situation here.

[ianfiske]

Ah, good catch! Thanks for the explanation. Indeed we did intentionally set JULIA_DEPOT_PATH: "./.julia". This was done to allow Gitlab to cache the precompiled dependencies to speed up our CI jobs. Maybe there is another way. Or we can just wait for the #41 fix. Or we're good with my workaround for right now.