compiletest.sh fails on main

[sarahec]

Since I need to test my own changes with compiletest, I set up a reference implementation: new VM running Ubuntu 20.04, and installation following the openasip installation instructions. This is using a fresh checkout of the main branch.

compiletest.sh -q passes compilation but fails on the unit and system tests, see compiletest.error.log and difference.txt.

[sarahec]

compiletest.sh (long) also fails compilation. See attached
compiletest.error.log

[pjaaskel]

It looks you might be using LLVM 14. If you are building with LLVM 14, it might not pass all tests (see the release notes) due to the opaque ptr transfer in which we didn't want to spend time to fix 14 test output just for the sake of this release cycle. The fix is to use LLVM 15 instead.

[sarahec]

Using LLVM 15 fixes the quick compile test. I had to add RISC-V to get the long test working.

Both the quick and standard compiletest seem to have unit test failures after adding RISC-V (using the directions in the README). I'm not finding anything in the logs to help isolate those errors. The system tests are working.

@pjaaskel did the release notes mention the LLVM 14 issues, or just recommended LLVM 15? I'm not finding anything about LLVM 14 issues in the README, CHANGES, or TROUBLESHOOTING files. (I'm wondering if that note hasn't been merged yet.)

[pjaaskel]

@sarahec can you open an issue of the RISC-V failures? I haven't personally tested without RISC-V myself for a while. @karihepola have you?

For LLVM 15 it "recommends" LLVM 15 indeed. LLVM 14 still should work, but we didn't fix the systemtest checks as the issues were minor and we'll drop LLVM 14 fully in the next release. Sorry for being unclear here, I referred to this sentence in the download page: "Due to the switch to opaque pointers in LLVM 15, we consider the support for the previous LLVM version 14 "best-effort" and recommend that you use OpenASIP together with LLVM 15. "

[karihepola]

The test suite should still work even without RISC-V support. It will check during the test cases whether RISC-V GNU Toolchain has been installed and exit without error if not. If it tries to run tests that require a RISC-V compiler, it is indeed a bug. I think the last time I ran the long tests was during the release. Notice that if you install the RISC-V prerequisites after installing OpenASIP, you might have to run "make distclean" before "make" and "make install" to make sure you are building newlib for RISC-V.

For the unit tests, you can run "make" in the test directory and check the output that will tell you which test case has an error. I don't think the RISC-V support should affect any of the unit tests. Is it possible that you are getting some warnings from the compiler and that is why the unit tests fail?

[sarahec]

Thank you @karihepola. I found the unit test errors: EnvironmentTest.hh is over-specifying the directory:

EnvironmentTest.hh:104: Error: Expected (path == correctPath), found ("/home/sec/local/share/openasip//data/ProDe" != "/home/sec/development/openasip/openasip/data/ProDe")

It looks like the test may be in error, as it's comparing TCE_SRC_ROOT with the installation location.

[karihepola]

I forgot to mention that you must set export TCE_DEVEL_MODE=1 before calling make in the test directory to fix the search paths. The system test script does that automatically. That should fix the error you are having.

There's also a Wiki page for the test suite but it does not go through it very thoroughly. Maybe we could expand it a little.

https://github.com/cpc/openasip/wiki/Contributor-Info

[sarahec]

TCE_TEST_MODE=1 fixed the unit tests when run via make, but not when run via compiletest.sh. I will keep looking.

On a related note, would it make sense to have a top-level INSTALL.md file that puts the setup instructions in one place (including a note for people who plan to develop code)?

[pjaaskel]

TCE_TEST_MODE=1 fixed the unit tests when run via make, but not when run via compiletest.sh. I will keep looking.

You probably meant TCE_DEVEL_MODE. It's a known issue that some warnings with never gccs will spill out to the log and make compiletest.sh "fail" although those are only warnings. It would be nice to silence them, but no one has just had the time.

On a related note, would it make sense to have a top-level INSTALL.md file that puts the setup instructions in one place (including a note for people who plan to develop code)?

Yes, documentation updates are more than welcome!

[sarahec]

Sorry, yes, TCE_DEVEL_MODE.

Let me take a look at those GCC warnings. Some of them are boost/timer deprecation warnings, which I need to see how they crept back in.