Can't link in /librcl_interfaces__rosidl_generator_py.so due to it not linking in libpython

[tony-p]

Solution to issue cannot be found in the documentation.

• I checked the documentation.

Issue

I'm having an issue on linux (humble) that when it tries to link in the librcl_interfaces__rosidl_generator_py.so it is missing a number of symbols resulting in undefined reference symbols

env/lib/librcl_interfaces__rosidl_generator_py.so: undefined reference to `PyLong_AsSize_t'

It is essentially the same problem as this https://robotics.stackexchange.com/questions/25039/generated-rosidl-generator-py-so-files-issue

And it is the same for the robostack provided version, however the workarounds don't seem to work and I think it is because it is already too late and should have been applied to these packages or the ones in between.

$ ldd -r .pixi/env/lib/librcl_interfaces__rosidl_generator_py.so
	linux-vdso.so.1 (0x00007ffcd3fd2000)
	libbuiltin_interfaces__rosidl_generator_py.so => /home/me/workspace/Workcell-Automation-Core/.pixi/env/lib/./libbuiltin_interfaces__rosidl_generator_py.so (0x00007f51715d8000)
	librcl_interfaces__rosidl_generator_c.so => /home/me/workspace/Workcell-Automation-Core/.pixi/env/lib/./librcl_interfaces__rosidl_generator_c.so (0x00007f51715be000)
	librosidl_runtime_c.so => /home/me/workspace/Workcell-Automation-Core/.pixi/env/lib/./librosidl_runtime_c.so (0x00007f51715b2000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f5171200000)
	libbuiltin_interfaces__rosidl_generator_c.so => /home/me/workspace/Workcell-Automation-Core/.pixi/env/lib/././libbuiltin_interfaces__rosidl_generator_c.so (0x00007f517159b000)
	librcutils.so => /home/me/workspace/Workcell-Automation-Core/.pixi/env/lib/././librcutils.so (0x00007f5171584000)
	/lib64/ld-linux-x86-64.so.2 (0x00007f51715ef000)
	libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f517157f000)
	librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f517157a000)
undefined symbol: PyObject_GetAttrString	(/home/me/workspace/Workcell-Automation-Core/.pixi/env/lib/./libbuiltin_interfaces__rosidl_generator_py.so)
undefined symbol: PyObject_SetAttrString	(/home/me/workspace/Workcell-Automation-Core/.pixi/env/lib/./libbuiltin_interfaces__rosidl_generator_py.so)
undefined symbol: _Py_Dealloc	(/home/me/workspace/Workcell-Automation-Core/.pixi/env/lib/./libbuiltin_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_FromLong	(/home/me/workspace/Workcell-Automation-Core/.pixi/env/lib/./libbuiltin_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_AsLong	(/home/me/workspace/Workcell-Automation-Core/.pixi/env/lib/./libbuiltin_interfaces__rosidl_generator_py.so)
undefined symbol: PyImport_ImportModule	(/home/me/workspace/Workcell-Automation-Core/.pixi/env/lib/./libbuiltin_interfaces__rosidl_generator_py.so)
undefined symbol: PyObject_CallObject	(/home/me/workspace/Workcell-Automation-Core/.pixi/env/lib/./libbuiltin_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_AsUnsignedLong	(/home/me/workspace/Workcell-Automation-Core/.pixi/env/lib/./libbuiltin_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_FromUnsignedLong	(/home/me/workspace/Workcell-Automation-Core/.pixi/env/lib/./libbuiltin_interfaces__rosidl_generator_py.so)
undefined symbol: PyList_New	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_FromUnsignedLongLong	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyBuffer_Release	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_AsUnsignedLongLong	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyObject_GetAttrString	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyUnicode_AsUTF8String	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyBytes_FromStringAndSize	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyObject_SetAttrString	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyBuffer_ToContiguous	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: _Py_Dealloc	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyExc_RuntimeError	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyErr_SetString	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyFloat_FromDouble	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PySequence_Size	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyObject_CheckBuffer	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyBool_FromLong	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_AsSize_t	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyImport_ImportModule	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyObject_CallObject	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyList_SetItem	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyObject_CallFunctionObjArgs	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyObject_Size	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyUnicode_DecodeUTF8	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: _Py_TrueStruct	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyObject_GetBuffer	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_AsUnsignedLong	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_FromUnsignedLong	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_AsLongLong	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_FromLongLong	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PySequence_Fast	(.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)

Would patching the librcl_interfaces__rosidl_generator_py build to force link in libpython3 be the best solution here?

Installed packages

pixi Lock file is too long too paste here, will try find a more concise way to paste, but the relevant package is

- platform: linux-64
  name: ros-humble-rcl-interfaces
  version: 1.2.1
  category: main
  manager: conda
  dependencies:
  - libgcc-ng >=12
  - libgcc-ng >=12
  - libstdcxx-ng >=12
  - numpy >=1.21.6,<2.0a0
  - python
  - python_abi 3.10.* *_cp310
  - ros-humble-builtin-interfaces
  - ros-humble-ros-workspace
  - ros-humble-rosidl-default-runtime
  - ros2-distro-mutex 0.3.*
  - ros2-distro-mutex 0.3.* humble
  url: https://repo.prefix.dev/robostack-staging/linux-64/ros-humble-rcl-interfaces-1.2.1-py310h7c61026_3.tar.bz2
  hash:
    md5: 7123a8c0b2d39745f42ee72e444e8caf
    sha256: d66ad32b9a592a056b7b6a4eda2840a3b67d3b506a7ee3e29528bd81da9731c5
  build: py310h7c61026_3
  arch: x86_64
  subdir: linux-64
  build_number: 3
  size: 343519
  timestamp: 1675688249136

Environment info

pixi info
pixi 0.10.0

Platform            : linux-64
Virtual packages    : __unix=0=0
                    : __linux=6.2.0=0
                    : __glibc=2.35=0
                    : __archspec=1=x86_64
Cache dir           : /home/me/.cache/rattler/cache
Auth storage        : /home/me/.rattler/rattler_auth_store.json

Project
------------

Manifest file       : /home/me/workspace/project/pixi.toml
Dependency count    : 49
Last updated        : 19-12-2023 16:24:34
Target platforms    : win-64
                    : linux-64

[tony-p]

Update: force linking in the output of $(python3-config --ldflags --embed) (from the linked issue) in the failing package does work, although as a temporary fix for another linking issue I had added the env/lib in LD_LIBRARY_PATH and then it tried to link in the wrong version of 'libfreebl3.so` which required a newer glibc

[Tobias-Fischer]

Hi @tony-p - could you please try again with the recently built updated packages?

[tony-p]

Tried again very naively and same issue occurs however checked and is taking build 3 instead of 5 of rcl-interfaces 1.2.1 so will look into a bit more when I have a little more time

[tony-p]

Dug a little deeper, and now have the build 5 and end up with the same errors. Further the ldd shows that these still aren't linked (below)

Further I encountered the following issues:

• for some reason the dependency resolution was choosing an old version of sysroot that had libc v2.12. Forcing that sysroot >=2.17 solved it without other dependency conflicts so don't know why it wasn't chosen in the first place. This also means that there is the sysroot package and the virtual __glibc package both providing libc.
• many ament_*lint executables are in the lib/<tool> directory which results in them not being found by cmake as they are not on the path. Manually adding each tool to the path (temporarily) solved that issue

ldd -r .pixi/env/lib/librcl_interfaces__rosidl_generator_py.so
        linux-vdso.so.1 (0x00007ffc8c0c7000)
        libbuiltin_interfaces__rosidl_generator_py.so => /home/tonypaulussen/workspaces/Workcell-Automation-Core/.pixi/env/lib/./libbuiltin_interfaces__rosidl_generator_py.so (0x00007f375c73b000)
        librcl_interfaces__rosidl_generator_c.so => /home/tonypaulussen/workspaces/Workcell-Automation-Core/.pixi/env/lib/./librcl_interfaces__rosidl_generator_c.so (0x00007f375c721000)
        librosidl_runtime_c.so => /home/tonypaulussen/workspaces/Workcell-Automation-Core/.pixi/env/lib/./librosidl_runtime_c.so (0x00007f375c715000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f375c4e6000)
        libbuiltin_interfaces__rosidl_generator_c.so => /home/tonypaulussen/workspaces/Workcell-Automation-Core/.pixi/env/lib/././libbuiltin_interfaces__rosidl_generator_c.so (0x00007f375c4de000)
        librcutils.so => /home/tonypaulussen/workspaces/Workcell-Automation-Core/.pixi/env/lib/././librcutils.so (0x00007f375c4c7000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f375c751000)
        libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f375c4c2000)
        librt.so.1 => /lib/x86_64-linux-gnu/librt.so.1 (0x00007f375c4bd000)
undefined symbol: PyObject_GetAttrString        (/home/tonypaulussen/workspaces/Workcell-Automation-Core/.pixi/env/lib/./libbuiltin_interfaces__rosidl_generator_py.so)
undefined symbol: PyObject_SetAttrString        (/home/tonypaulussen/workspaces/Workcell-Automation-Core/.pixi/env/lib/./libbuiltin_interfaces__rosidl_generator_py.so)
undefined symbol: _Py_Dealloc   (/home/tonypaulussen/workspaces/Workcell-Automation-Core/.pixi/env/lib/./libbuiltin_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_FromLong       (/home/tonypaulussen/workspaces/Workcell-Automation-Core/.pixi/env/lib/./libbuiltin_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_AsLong (/home/tonypaulussen/workspaces/Workcell-Automation-Core/.pixi/env/lib/./libbuiltin_interfaces__rosidl_generator_py.so)
undefined symbol: PyImport_ImportModule (/home/tonypaulussen/workspaces/Workcell-Automation-Core/.pixi/env/lib/./libbuiltin_interfaces__rosidl_generator_py.so)
undefined symbol: PyObject_CallObject   (/home/tonypaulussen/workspaces/Workcell-Automation-Core/.pixi/env/lib/./libbuiltin_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_AsUnsignedLong (/home/tonypaulussen/workspaces/Workcell-Automation-Core/.pixi/env/lib/./libbuiltin_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_FromUnsignedLong       (/home/tonypaulussen/workspaces/Workcell-Automation-Core/.pixi/env/lib/./libbuiltin_interfaces__rosidl_generator_py.so)
undefined symbol: PyList_New    (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_FromUnsignedLongLong   (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyBuffer_Release      (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_AsUnsignedLongLong     (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyObject_GetAttrString        (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyUnicode_AsUTF8String        (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyBytes_FromStringAndSize     (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyObject_SetAttrString        (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyBuffer_ToContiguous (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: _Py_Dealloc   (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyExc_RuntimeError    (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyErr_SetString       (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyFloat_FromDouble    (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PySequence_Size       (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyObject_CheckBuffer  (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyBool_FromLong       (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_AsSize_t       (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyImport_ImportModule (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyObject_CallObject   (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyList_SetItem        (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyObject_CallFunctionObjArgs  (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyObject_Size (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyUnicode_DecodeUTF8  (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: _Py_TrueStruct        (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyObject_GetBuffer    (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_AsUnsignedLong (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_FromUnsignedLong       (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_AsLongLong     (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PyLong_FromLongLong   (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)
undefined symbol: PySequence_Fast       (.pixi/env/lib/librcl_interfaces__rosidl_generator_py.so)

[Tobias-Fischer]

I wonder if this should be public:

ros-humble/patch/ros-humble-rosidl-generator-py.patch

Line 61 in 52756fd

+target_link_libraries(${_target_name_lib} PRIVATE Python3::NumPy Python3::Module)

Any ideas @traversaro @wolfv?

[Tobias-Fischer]

I've just pushed 7d0d8be which I have locally confirmed fixes this issue. If you rebuild ros-humble-rosidl-generator-py and then ros-humble-rcl-interfaces the issue will go away. Unfortunately at the moment I have not the capacity to build this in CI; but it would be easy to do (please see Contributing guidelines if you'd like to open a PR @tony-p).

[tony-p]

Great thanks, will try find some time to try soon (also limited capacity) and if fixes and have time will definitely look into the CI build

[EzraBrooks]

Hi, what's left to get this across the finish line? I'm pretty blocked by this issue and would be happy to help.

[tony-p]

Hi, what's left to get this across the finish line? I'm pretty blocked by this issue and would be happy to help.

I suggest to read the thread in this PR #169

In short the PR ended up reverting the change due to downstream build dependency issues. Wolf rewrote the cmake to more modern standards, but I haven't had the chance to rebuild try if that also fixed the issue (so that would be step 1).

Given that the python libraries are still not public, I fear the issue may still be there.

The issue for the downstream builds was that they failed to find python as dependency (as opposed to missing symbols), One (hacky?) solution to that was to include rosidl_generator as a dependency purely as it has a additional exported cmake file that includes finding the python dependencies (

ros-humble/patch/ros-humble-rosidl-generator-py.patch

Line 13 in 71d4b56

+find_package(Python3 REQUIRED COMPONENTS Development NumPy)

).

So if it doesn't work with the updates from Wolf, when the interface pacakges are themselves a dependency, they need to be able to give this information through, ideally in a cleaner way then adding another dependency that happens to do that.

As a short term hack fix in your own builds that are causing issues you can add something like the following to force the un found packages to be linked

if(UNIX)
  # Hack to link in missing/can't find libraries on linux
  execute_process(COMMAND bash "-c" "echo $(python3-config --ldflags --embed)" OUTPUT_VARIABLE BUILD_PYTHON_LINK_FLAGS)
  separate_arguments(FORCE_LINK_FLAGS UNIX_COMMAND ${BUILD_PYTHON_LINK_FLAGS})
  list(APPEND FORCE_LINK_FLAGS
  -lstd_msgs__rosidl_generator_py
  )
  add_link_options(${FORCE_LINK_FLAGS})
endif()

[EzraBrooks]

Can you clarify what you mean by "include rosidl_generator as a dependency"? Do you mean as a package.xml dependency, an ament_cmake dependency, a pure CMake dependency, or a source build within the same workspace - or some combination of those?

Just adding it as a source build in my workspace doesn't seem to help (I assume because the other packages don't have direct dependencies on it, so it's not resolved to be built first)

[tony-p]

An ament dependency in the cmake file that is used to generate the interfaces that are built into a conda package.

So this is how I managed to fix it for the conda builds:
https://github.com/tony-p/ros-humble/blob/d73ed6077cccd4d35f3ec6f260cf1c5be43e19de/patch/ros-humble-rosidl-generator-py.patch#L133

So this is actually during the packaging process, once you get to your own builds, the last hack I presented above is (as far as I have found) the only way I have found to resolve the issue.

If you want to look into fixing it properly, this is a good resource as to how to start https://robostack.github.io/Contributing.html

[EzraBrooks]

I will look into fixing it properly.

Do we have a hunch on what the source of this issue is? Is there some "magic" being done by Ament, etc. when using these packages from the upstream that RoboStack doesn't do (and is probably correct in not doing)? It would be good to know if there's a gotcha here that will recur elsewhere.

[tony-p]

The problem is that python is not an ament package, and so not in the ament index. Ament handles the finding of recursive dependencies, so long as they are in the index (from what I understand).

When a library is linked private (on linux) it isn't added on the rpath, so isn't necessarily required to be known by child dependencies, but (in this case) the local implementation that is 2 dependencies removed from the privately linked python libs, it tries to ensure that all symbols are defined, but then finds symbols that don't seem to have an implementation which are provided by the python library.

When the python libs are linked public, a direct implementing library needs to explicitly link the python library, but then it needs to be found by cmake, and cmake hasn't "found" python with a find_package statement so that fails. This is why declaring rosidl_generator_py as a dependency in the generation cmake solves the issue, because it exports a cmake file that finds python for all packages that have rosidl_generator_py as a dependency.

At least that is my rough understanding.

It may be that the package building environment has the python libraries/symbols always available/findable which is why it doesn't cause an issue for building packages.

[EzraBrooks]

Thanks for that explanation. I'm starting to take a deeper dive into the actual patches being applied to the CMake here to better understand the differences between the upstream and RoboStack's version.

I'm still trying to get up-to-date on the state of this project - when you earlier said "I haven't had the chance to rebuild try if that also fixed the issue" w/r/t @wolfv's changes, does that mean those changes are not yet public on the Conda channel? If so, is there something blocking that that y'all could use help with? If the Conda packages are outdated that seems higher priority to me than getting this fixed, though to make my application work with RoboStack I'll need this. I have way too many dependencies to be able to shoehorn the linking workaround into all of them, so the underlying issue will need fixed (which I'm happy to do or have a member of my team do).

[tony-p]

I think the last published full humble rebuild was nearly a year ago (Jan/Feb).

Not sure if you are aware of the rattler-build/pixi vs boa/micromamba ecosystems, but Wolf started migrating to the recipe formats used by rattler-build in a branch of Vinca. He used jazzy as his test target and there he ran into the issues caused by making the python dependency public and reverted the public/private and updated the cmake patch in general.

[EzraBrooks]

My investigation into RoboStack is also somewhat tied into an investigation of Pixi, so I should probably learn the rattler stuff along the way.

Does that jazzy build system work need to be backported to Humble to make it easier for the two to be continuously maintained? I'd like to be on Jazzy, but I have customers stuck on earlier versions of Ubuntu that can't run Jazzy (which is one of the reasons I'm looking into RoboStack in the first place).

[traversaro]

If so, is there something blocking that that y'all could use help with?

Sure, thanks! The latest attempt to rebuild humble is conda-forge/ompl-feedstock#39 . The blocking issue there is that we were not able to upgrade to the latest boost in ompl, as with the latest boost macos tests fail. Hopefully I will try to look into that in the next weeks, as also in my organization we will really like to unblock the humble rebuild.

[traversaro]

@EzraBrooks I just read from #222 that you are an Apple Silicon users. So if you want to look into conda-forge/ompl-feedstock#43 feel free to do so!

[tony-p]

Does that jazzy build system work need to be backported to Humble to make it easier for the two to be continuously maintained? I'd like to be on Jazzy, but I have customers stuck on earlier versions of Ubuntu that can't run Jazzy (which is one of the reasons I'm looking into RoboStack in the first place).

Just saw this question wasn't answered. I think as soon as it works for jazzy, I think the intention is also to roll it out for humble. I think the only backport work that would be required is updating the contribution instructions, and the additional recipes that have been made by hand. I think a number of those recipes will also be useful for jazzy and likely almost the same