REP 149: Package Manifest Format Three Specification

[dirk-thomas]

Opening this draft for reference purposes. The goal is to cover:

• group dependencies REP 149: group dependencies #141
• ABI / compatibility version REP 149: compatibility version #142
• conditional dependencies (e.g. across ROS 1 and ROS 2) REP 149: conditional dependencies #143

In order to not mix the discussion on the different subjects I have created three separate issues. Please comment on the referenced issues for topic specific feedback.

[nuclearsandwich]

I feel like the abi_version attribute belongs on the <version> element rather than the <package> element. Because of its adjacency with the package manifest format it seems like the property is specific to the package manifest rather than the package itself.

Is there language that defines abi_version compatibility within the scope of the package manifest attribute or is it intended to be discretionary?

[dirk-thomas]

I feel like the abi_version attribute belongs on the <version> element rather than the <package> element. Because of its adjacency with the package manifest format it seems like the property is specific to the package manifest rather than the package itself.

Once we have a consensus on the proposed semantic we can shuffle the information around wherever we want it to be.

Is there language that defines abi_version compatibility within the scope of the package manifest attribute or is it intended to be discretionary?

Packages built against a previous version of the package will continue to work. That is generally what ABI compatibility expresses.

[tfoote]

We could probably keep the dependencies but have a shortcut/check for ABI compatibility at the top of the build and it will just pass without a rebuild if an ABI compatible version already exists.

[dirk-thomas]

This would require downstream jobs to be triggered just in order to determine that they should not be rebuild. I would rather avoid that overhead and not trigger any downstream jobs in the first place. Also the logic should be contained within the job of the package declaring the abi compatibility imo.

[nuclearsandwich]

We either have to keep the explicit job ordering and short circuit for ABI compatibility switch to programatically scripted jobs and deal with potential race conditions when a package build is triggered by one upstream build but other upstreams are still building.

I would guess that the overhead of queuing and short circuiting is less than trying to iron out all of the behavior of a system where we write our own triggers.

It's also possible that there may be options or plugins that allow us to keep the explicit dependency relationship between jobs in Jenkins but only trigger jobs in certain conditions.

[tfoote]

Do we want to allow partial matches, aka "MAJOR.MINOR" would match "MAJOR.MINOR.*" which would allow declarations like semantic versioning? http://semver.org/

[dirk-thomas]

I don't see a use case where that would be beneficial over specifying MAJOR.MINOR.0. Therefore I would say "no".

[dirk-thomas]

I have updated the REP draft based on the recent discussion of group dependencies as well as ABI / compatibility version.

@ros-infrastructure/ros_team Please take a closer look and feel free to fix / improve spelling / wording where you see fit and propose concrete additions if you think more information / context would be helpful.

[wjwwood]

Example should be updated?

[dirk-thomas]

I am not sure what would be a good example to use both of the new tags. I am open to suggestions.

[wjwwood]

rebuilt instead of recompiled? recompiled implies to me compiled languages, but this would also apply to interpreted languages and other build artifacts.

[wjwwood]

"The content is the name of a dependency group on which the package depends."?

[wjwwood]

"The content is the name of a dependency group of which the package is a member."?

[wjwwood]

The behavior I would expect is:

• if the compatibility attribute exists:
  • ask the user if it should be changed to the current version or left as-is
• if not:
  • let they user know that this release will cause downstream packages to be rebuilt (let them know not having compatibility between releases causes extra effort)

Since it is opt in anyways, why would the tool automatically opt out again? It could have an interactive message to make sure the user knows the version change needs to be stable unless this attributed is updated (to notify new maintainers that they package previously had compatibility control).

---

Is this just because build.ros.org doesn't process this attribute yet? In that case I'd just advocate for a warning (perhaps with a confirmation so they can cancel if they want) that this is the case, but offering to remove it seems unnecessary to me.

[dirk-thomas]

Just because the previous release of a package was compatible was earlier versions doesn't imply the next release will be too. I think it would be safer to assume that it is not until the maintainer explicitly says otherwise.

[wjwwood]

I would rather see the compatibility version updated each time (same behavior as removing it), rather than removed. So long as the tool is automating it, either way should be fine I just think it would produce a more normal history to continuously update the compatibility version rather than periodically removing it and restoring it. I think it's a reasonable assumption that the default behavior after releasing once with a compatibility version will be to continue using the version even if every release is not compatible.

This seems, however, to be a lot of detail on one tools behavior for this document. It might just be sufficient to say "the tool should default to updating the compatibility version each release, requiring the user to explicitly opt into a backwards-compatible release".

[wjwwood]

I know what you mean here, but I have a hard time believing this is enough for others to understand the situation you're talking about. Without the example of the package that depends on all message packages (e.g. ros1_bridge), it's hard to imagine. So maybe you should describe that scenario to make it clearer why a reverse dependency makes it hard to change the name of the reverse dependent.

[wjwwood]

"separate identity" or "separate identifier"?

[wjwwood]

I think this would be clearer if we had an example use case (like the ros1_bridge or a made up "message collector" package of some kind). With that you could reword this to be clearer (in my opinion):

The group dependencies can be expressed in two parts:

• a package can declare a dependency on a group of packages, e.g. the ros1_bridge can depend on "message_packages"
• other packages can declare themselves as a member of a group of packages, e.g. sensor_msgs can declare itself part of the "message_packages" group

[wjwwood]

I don't think this is the case anymore.

[wjwwood]

Has this section been updated?

[dirk-thomas]

Thank, I applied most of the feedback in bca3965.

[dirk-thomas]

The REP draft now also describes the condition expression which can be used on dependency and membership tags.

@ros-infrastructure/ros_team Please take another close look and feel free to fix / improve spelling / wording where you see fit and propose concrete additions if you think more information / context would be helpful. After that I would like to get feedback from the community on this. I might create three separate issues for the three different parts of the REP to keep each thread focus on one topic.

[dirk-thomas]

@clalancette @dhood @Karsten1987 @mikaelarguedas @nuclearsandwich @sloretz @tfoote @wjwwood Please provide feedback on the current state of the draft today. If you have no comments and think the draft doesn't need any changes please at least add a reaction to this comment for visibility.

I plan to send an announcement to discourse later today to get feedback from the ROS community if there are no objections.

[nuclearsandwich]

In general I think the draft is ready for community feedback. 👍

I have one comment / proposal related to the Environment Variables section.

The package exporting the necessary environment should be a dependency of almost all ROS 2 packages to ensure that the information is available even when only some packages are installed. The package rcl seems to be a good place for this.

Another possibility would be to make the ros-workspace package that we've been using to bootstrap the beta releases more official and give it this responsibility. Its only root dependencies are ament_package and ament_cmake_core its sole purpose is providing the base environment and setup files.

Right now bloom has a hack in place to make it a dependency of every other package for ROS 2 distros. I would love to either retire it in favor of an accepted root workspace package, rcl or otherwise, or clean it up and canonicalize its status.

Package compatibility support will make rebuilding rcl as a root package less impactful but rcl is actually quite high level in the package hierarchy. These are its direct upstreams on the buildfarm:

• ament_cmake
• ament_cmake_gtest
• ament_cmake_nose
• ament_cmake_ros
• ament_lint_auto
• ament_lint_common
• example_interfaces
• launch
• rcl_interfaces
• rcutils
• rmw
• rmw_implementation
• rmw_implementation_cmake
• rosidl_default_runtime
• rosidl_generator
• std_msgs

We could, of course, keep using the ros-workspace package to do what its doing and just add the ROS_DISTRO and ROS_VERSION envars to rcl, but it makes more sense to me to provide the environment variables in one consistent place.

[tfoote]

A few suggestions but otherwise lgtm to circulate

[tfoote]

I'd suggest that we require naming rules to be followed.

[dirk-thomas]

Done: 2e7172f

[tfoote]

Since it's not required to switch and they can run side by side. I'd suggest

format,..." it is an update to the versions"

[dirk-thomas]

Rephrased: 2e7172f

[tfoote]

The buildsystem's don't explicitly need these changes. I'd suggest:
"The extensions are relevant to supporting more complex use cases in ROS2 as well as to support easier code sharing between ROS1 and ROS2."

[dirk-thomas]

I removed the sentence (2e7172f) since most is not specifically "more complex" and the rest of the document should cover it already.

[gavanderhoorn]

Not a content thing, but could I ask for the deuteranomalous and deuteranopes among us to use a different colour here?

[nuclearsandwich]

Thanks for raising this. I should have caught it but since GitHub filters out html tags the draft I was reviewing wasn't colorized.
My CSS overrides for GitHub.com use #b85c00 in place of dark red and #3269a0 for dark green. It's also worth noting that these indicators will be of no utility for readers using text-to-speech. So perhaps additional indicators should be used for significant changes.

[dirk-thomas]

I switched the green to blue: 2e7172f

For the final document the coloring will likely be removed.

[dirk-thomas]

I will merge the draft as-is since the generated HTML page is much easier for users to read.

[peci1]

Would it be possible to put online some example, e.g. how would the std_msgs package.xml change utilizing the new possibilities? Without an example, it is still too abstract to me...

[dirk-thomas]

@peci1 The message packages don't use any of the features yet.

rmw_implementation e.g. uses the group dependencies which can serve as an example: https://github.com/ros2/rmw_implementation/pull/32/files#diff-e20e58462fdb7b39b7dba23ae765bab7R23

Specific implementations like rmw_opensplice_cpp can then declare themselves as being a member of that group: https://github.com/ros2/rmw_opensplice/pull/210/files#diff-416ef50d50e6a77d8de4e69360278131R38

If a package wants to target ROS 1 as well as ROS 2 from the same code base it can use conditional dependencies to avoid branching just for the manifest. This could look like the following:

<build_depend condition="$ROS_VERSION==1">message_generation</build_depend>
<exec_depend condition="$ROS_VERSION==1">message_runtime</exec_depend>

<build_depend condition="$ROS_VERSION==2">rosidl_default_generators</build_depend>
<build_depend condition="$ROS_VERSION==2">rosidl_default_runtime</build_depend>
<depend condition="$ROS_VERSION==2">builtin_interfaces</depend>

Whatever would be necessary on CMake and source code level is not in the scope of this PR though.

[peci1]

Shouldn't there be a better definition of "evaluates"? In which language? In which semantics? In what context (e.g. what variables are available and how to set them)? Is it possible to compare against strings? Should strings be quoted? Why no "<", "<=", ">", ">=" ?

It would seem nice to me if there would be an agreement about this in-XML conditional expressions language and expressivity. Right now I know about this, about roslaunch, and about Xacro; maybe there are even more tools evaluating conditions inside XML files. It starts to be difficult to keep track about where I can use what...

[dirk-thomas]

Shouldn't there be a better definition of "evaluates"?

The string describes an expression which can be "evaluated" to a boolean value. I am not sure what kind of definition you would like to see. Can you please propose a text which clarifies your expectation.

In which language?

Evaluating could happen in any language. E.g. ament_package provides a function in Python which parses the condition and can evaluate it based on some input variables.

In which semantics? In what context (e.g. what variables are available and how to set them)?

Any tool evaluating these conditions has to define that. E.g. ament_tools passes all environment variables in. If a tool like rosdep needs to also support to be invoked without environment being set already then it could offer command line options like --ros-version (similar to the existing option --ros-distro.

Is it possible to compare against strings? Should strings be quoted? Why no "<", "<=", ">", ">=" ?

All variables are interpreted as strings. The same as e.g. environment variables are being used in a shell. Since there is no typing beyond strings the meaning of comparison operators is less useful. Since there wasn't a use case requiring this those operators where not specified in the current draft.

It would seem nice to me if there would be an agreement about this in-XML conditional expressions language and expressivity. Right now I know about this, about roslaunch, and about Xacro; maybe there are even more tools evaluating conditions inside XML files. It starts to be difficult to keep track about where I can use what...

For complex conditions in XML would be one way. Imo the way roslaunch exposes those especially with the desire to express even more and more complicated conditions is not a sustainable way. The prototype of launch files in ROS 2 doesn't use XML for that reason. A script like Python is much better suited to express complex conditions and logic to decide if a node should be started or what the arguments / parameters should be. Pushing that complexity into XML is from my point of view not desired.

[peci1]

Well, it seems to me that what's in this specification is just the syntax, but not the semantics of the expression.

There's e.g. not specified the meaning of the parentheses (of course it is "apparent", but unless written down, somebody may interpret it differently).

I suggest adding the following paragraph (after the list of tokens):

An expression syntactically correct by the previous definition will be evaluated as follows. First, all variables are substituted by their values and treated as strings. All literals are also treated as strings. The resulting expression is then evaluated as a Python 3 interpreter would evaluate it.

Beware, in Python "1" != 1! Empty string evaluates to False.

Then I suggest adding a few hints about error handling. I just don't know which of these to add:

1. When a syntactically incorrect condition expression is encountered, the processing tool should stop with an error.
2. When a syntactically incorrect condition expression is encountered, the condition attribute is ignored and the processing tool should produce a warning or alert the developer (not necessarily the user).

Also, error handling for the variables should be defined:

1. When a variable is used in the condition expression, whose value is not known to the processing tool, it should stop processing and issue an error.
2. When a variable is used in the condition expression, whose value is not known to the processing tool, it should ignore the condition attribute and issue a warning to the developer.
3. When a variable is used in the condition expression, whose value is not known to the processing tool, it is substituted with and empty string (and a warning is issued to the developer).

As a final suggestion, an example could be given.

As an example, a dependency might only be needed in ROS 1 build environment. Such dependency could read <depend condition="$ROS_VERSION==1">roscpp</depend>, where the value of $ROS_VERSION can be taken from the shell environment by catkin/ament.

I still think more comparison operators would be better, to be e.g. able to write $ROS_DISTRO >= kinetic. However I know that this would greatly increase the complexity of this specification. And it is not wise to just let there be any Python expression, because then even the C++ processing tools would need to emulate/access the Python interpreter. So, I am for a constrained syntax, but maybe not that much constrained.

[dirk-thomas]

@peci1 Thank you for your feedback. I have applied some of it in 71ccf1f.

Then I suggest adding a few hints about error handling.

I am not sure those "rules" make sense for each and every tool. Therefore I would rather not specify this in the REP but let each tool decide what behavior makes most sense based on the context.

I still think more comparison operators would be better, to be e.g. able to write $ROS_DISTRO >= kinetic.

Especially for the ROS distro name the operator can't be implemented easily. Simply because there is no knowledge about the order of the names. It would require a new release of a software package which defines those names whenever a new distro is being released (currently that is not the case).

In general instead of defining an "open-ended" interval you can invert the logic and specify the known inverted set of distro names. In this case $ROS_DISTRO != indigo and $ROS_DISTRO != jade. While it is more verbose it doesn't require any further knowledge / update in the future.

[jack-oquin]

The order of ROS distros would be alphabetical, one supposes.

[dirk-thomas]

The order of ROS distros would be alphabetical, one supposes.

That assumption might hold for a while. But at some point as you can currently see for Ubuntu that won't be the case anymore.

[peci1]

It might come handy to mention that rosdep currently ignores the version_* tags. Or add a link to issue ros-infrastructure/rosdep#325 . Again, an example use-case might help. Here, we could mention that Debian package version constraints can be generated from these tags.

[dirk-thomas]

This part as not changed since the previous REP. Therefore I won't focus on this in this REP but concentrate on the newly added attributes and tags. Please consider providing a PR to amend the existing REPs to add some example use case.

[jack-oquin]

The order of ROS distros would be alphabetical, one supposes.

That assumption might hold for a while. But at some point as you can currently see for Ubuntu that won't be the case anymore.

Ten years from now, when that becomes a problem, we could apply something like a rot13 adjustment on the leading letter.

Not to belabor a minor point, but I do think writing $ROS_DISTRO >= kinetic would be useful.

[dirk-thomas]

@jack-oquin Please see #146 which add these comparison operators.

[dirk-thomas]

Please review ros/ros_environment#1 which creates the new ROS package described in this REP.