Benchmarking OpenBlas on windows

[rajith-ravikumar-mcw]

I executed commands below which complete successfully

C:\Users\HCKTest\OpenBLAS\benchmark\pybench\openblas_wrap>python generate_f2pymod.py blas_lapack.pyf.src -o .
C:\Users\HCKTest\OpenBLAS\benchmark\pybench>meson setup builddir
C:\Users\HCKTest\OpenBLAS\benchmark\pybench>meson compile -C builddir

But on running

pytest benchmark/pybench/benchmarks/bench_blas.py

___________________________________________________________ ERROR collecting openblas_bench_build/OpenBLAS/benchmark/pybench/benchmarks/bench_blas.py ____________________________________________________________
ImportError while importing test module 'C:\Users\HCKTest\Desktop\OpenBLAS\benchmark\pybench\benchmarks\bench_blas.py'.
Hint: make sure your test modules/packages have valid Python names.
Traceback:
..\..\..\..\AppData\Local\Programs\Python\Python312-arm64\Lib\importlib\__init__.py:90: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
benchmark\pybench\benchmarks\bench_blas.py:3: in <module>
    import openblas_wrap as ow
E   ModuleNotFoundError: No module named 'openblas_wrap'

there are files generated in the builddir post meson compile but I'm unsure of how to use them to fix above error.

[martin-frbg]

looks like the openblas_wrap module is not in your python search path (or local site-packages folder) ? but I must admit I have never tried running this on Windows... perhaps try copying the openblas_wrap module into the benchmarks folder to see if it gets found there

[rajith-ravikumar-mcw]

Hi Martin,

That was the fix thanks!

Moving the meson built openblas_wrap (OpenBLAS\benchmark\pybench\builddir) module to the benchmarks folder fixed the error.

But there is another openblas_wrap module that comes with the repo.
(OpenBLAS\benchmark\pybench\openblas_wrap). If we move this to the benchmarks folder doesn't fix the error.

The steps to benchmark on windows are as follows:

C:\OpenBLAS\benchmark\pybench>meson setup builddir
C:\OpenBLAS\benchmark\pybench>meson compile -C builddir

Move the meson built openblas_wrap module from

OpenBLAS\benchmark\pybench\builddir To OpenBLAS\benchmark\pybench\benchmarks Directory

Run the test

pytest benchmark/pybench/benchmarks/bench_blas.py

[rajith-ravikumar-mcw]

Adding to the steps in
OpenBLAS\benchmark\pybench\meson.build
I had to modify the fortran_std parameter from 'legacy' to 'none' since it throws errors on meson setup command

Replace:

project(
  'openblas-wrap',
  'c', 'fortran',
  version: '0.1',
  license: 'BSD-3',
  meson_version: '>= 1.1.0',
  default_options: [
    'buildtype=debugoptimized',
    'b_ndebug=if-release',
    'c_std=c17',
    'fortran_std=legacy',
  ],
)

With

project(
  'openblas-wrap',
  'c', 'fortran',
  version: '0.1',
  license: 'BSD-3',
  meson_version: '>= 1.1.0',
  default_options: [
    'buildtype=debugoptimized',
    'b_ndebug=if-release',
    'c_std=c17',
    'fortran_std=none',
  ],
)

[martin-frbg]

Thank you very much for the feedback. The second part sounds like it might be related to the Fortran compiler you used - I think this benchmark was developed using gfortran, while you may be using flang ? (At least googling "meson fortran_std=legacy" brings up several hits including an open github ticket (12306) in the mesonbuild project which state that meson does not fully support flang yet). Probably just documenting this workaround in the local README.md would be best.

I'll also add a note to the README.md about copying the openblas_wrap module to the correct folder on Windows - I'm not sure yet if changing the script to do this automatically would be correct for all platforms. This pybench was primarily written to be run from a CI script on github which checks every new pull request for its impact on the performance of key functions, so I'd rather not risk breaking that functionality. (Edited to add: this may also be the reason why you see an additional openblas_wrap already present in the repo - though I think this contains only the sources for the actual wrapper to be built by meson ?)

[Harishmcw]

I have followed the same steps mentioned in the discussion (linked above) for running the benchmark tests on x64. However, I am encountering the following error when executing the benchmark script:

Traceback (most recent call last):  
  File "c:\OpenBLAS\benchmark\pybench\benchmarks\bench_blas.py", line 3, in <module>  
    import openblas_wrap as ow  
  File "c:\OpenBLAS\benchmark\pybench\benchmarks\openblas_wrap\__init__.py", line 9, in <module>  
    from . import _flapack  
ImportError: DLL load failed while importing _flapack: The specified module could not be found.  

Could you please guide me on how to resolve this error? Are there additional steps I should follow to ensure that the benchmark script can locate and load the required _flapack module?

[martin-frbg]

_flapack is part of the openblas_wrap module, maybe that was not build correctly or you need to copy it into the benchmarks folder to have it in your PATH

[Harishmcw]

The _flapack module is successfully generated during the build process and is present within the openblas_wrap folder.
I moved the openblas_wrap folder to the correct path (benchmarks), as instructed, but I am still encountering the DLL load failed error.

Could you please clarify the difference between importing _flapack module and importing scipy.linalg.blas? Are they built differently, or do they serve distinct purposes in terms of functionality or dependencies?

[martin-frbg]

As far as I know, the wrappers in _flapack are a small subset of what is in scipy.linalg.blas, simply to avoid pulling in all of scipy for what was primarily intended as a CI job. I'm not aware of anyone having this problem of _flapack not loading (as far as I can tell from the source, it should not have any other dependencies) and unfortunately I cannot test it in a WoA environment right now.
The only caveat that I know of is that one has to change the fortran_std=legacy in the toplevel meson.build file to fortran_std=none when compiling with flang-new

[Harishmcw]

Hi Martin,

I just wanted to clarify that I am working on x64, not ARM. I'm encountering the "DLL load failed" error when trying to load the _flapack module, despite having the module successfully generated during the build process and moving the openblas_wrap folder to the correct path as instructed.

I understand that the _flapack module is a small subset of the scipy.linalg.blas module, but the issue persists specifically on x64. Could you suggest any additional steps or possible reasons why the _flapack module is not loading in this environment?

Thanks for your help!

[martin-frbg]

I'll need to try to reproduce the problem myself - the pybench files were contributed by the Scipy team, and I'm not as familiar with Meson as them. Steps I did to build the wrapper on Linux were

meson setup build
cd build
meson compile
meson install

the meson install step copied the wrapper to my python site-packages folder, with that done I was able to run the benchmarks without problems - not sure if it is just as simple on Windows, the "copy to the benchmark folder" was just a wild guess at the time but it appears to have solved the problem your colleague had. Maybe you copied the wrong things by accident ?

[Mugundanmcw]

@martin-frbg The issue is with the _flapack module requires OpenBLAS.dll to be loaded dynamically on Windows. This can be done through two ways:

• Adding the path to OpenBLAS dll in Environmental Path variable.
• It can be explicitly added in the init.py script of openblas_wrap folder by using os.add_dll_directory("path to openblas.dll")
  
  Following any one of the step would actually fix this potentially blocker

Thanks!

[martin-frbg]

Ah, I see, thank you very much. I had not gotten around to testing that on Windows yet, and probably the Scipy guys didn't give much thought to eventually running this on Windows either. On Linux, and certainly in the CI job for which this was originally written, the shared library could normally be expected to be found from the standard search path or the pkgconfig information.
I'll update the README.md in the pybench folder accordingly...

[Mugundanmcw]

@martin-frbg Meanwhile I am trying to have patch that can move this openblas_wrap module to corresponding benchmark folder in meson script that could run benchmark without any issues on Windows!