From 1435e8857365b9030763b833336101ae45d01e19 Mon Sep 17 00:00:00 2001 From: Dirk Thomas Date: Tue, 10 Oct 2017 11:54:34 -0700 Subject: [PATCH 01/10] add draft for REP 149: Package Manifest Format Three Specification --- rep-0149.rst | 973 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 973 insertions(+) create mode 100644 rep-0149.rst diff --git a/rep-0149.rst b/rep-0149.rst new file mode 100644 index 00000000..f794810a --- /dev/null +++ b/rep-0149.rst @@ -0,0 +1,973 @@ +REP: 149 +Title: Package Manifest Format Three Specification +Author: Dirk Thomas +Status: Draft +Type: Standards Track +Content-Type: text/x-rst +Created: 11-Oct-2017 + +Outline +======= + +#. Abstract_ +#. Motivation_ +#. Rationale_ +#. `Data Representation`_ +#. Compatibility_ +#. Schema_ +#. References_ +#. Copyright_ + +.. raw:: html + + The current draft highlights significant changes compared to REP-0140 [2]_ + in red. + Trivial changes appear in green. + + +Abstract +======== + +.. raw:: html + + + +This REP specifies the third ``package.xml`` format, replacing the ones +specified in REP-0127 [1]_ and REP-0140 [2]_. +It is relevant for packages using the catkin as well as the ament build system. + +.. raw:: html + + + + +Motivation +========== + +.. raw:: html + + + +This REP covers three separate topics which are described in the following +subsections. + +Dependency groups +----------------- + +Several ROS packages aggregate dependencies of a specific type, e.g. +`message_generation` in ROS 1 or `rmw_implementation` in ROS 2. +They explicitly list the dependencies which are being used during the release +process. +But when building from source they need to be updated to add custom message +generators or rmw implementations. + +The goal is to allow specifying this kind of group dependency in a way that a +from source build can process the custom packages accordingly to make them +available to the package even though the package doesn't list an explicit +dependency on the custom package. + +ABI version on the package level +-------------------------------- + +When building binary packages on the ROS buildfarm there is no assumption +about ABI compatibility. +When a new Debian package has been built all downstream packages are being +rebuild too in order to ensure they still work together. +This makes e.g. releasing a core package like `catkin` or `roscpp` +particularly costly. +Due to that overhead changes which should be released rather sooner than +later, e.g. a documentation fix, simple bugfix in a `.cpp` file, are being +hold back to group them with other changes to make the rebuild effort +worthwhile. + +In order to allow package releases without enforcing a rebuild of all +downstream packages a package should be able to declare if such a rebuild +is necessary or not. + +Dependencies across ROS 1 and ROS 2 +----------------------------------- + +In order for a ROS package to work with ROS 1 and ROS 2 from a single source +the dependencies listed in the package manifest are a problem. +E.g. in ROS 1 a package needs to depend on `roscpp` where as in ROS 2 it needs +to depend on `rclcpp`. +Currently the manifest doesn't support this use case. + +.. raw:: html + + + + +Rationale +========= + +.. raw:: html + + + +REP-0127 [1]_ and REP-0140 [1]_ provide the ``package.xml`` design rationale +for format one and two, which is not repeated here. + +.. raw:: html + + + +.. raw:: html + + + +Group dependencies +------------------ + +While "normal" dependencies are specified in the package which expresses a +dependency on another package this is not adequate for the targeted goal. +Instead packages need to be able to declare that they are a dependency of +another package. + +For that kind of "reverse" dependency a package could use the name of the other +package. +But that approach would fail if the other package gets forked under a different +name since the reverse dependency would still reference the original package +name. + +Therefore a reverse dependency is not using the package name but rather a +separate identify which will be called "group dependency name". +The reverse dependency can then be specified in two parts: + + * *(A)* packages declaring a dependency on a "group dependency name", e.g. + "message_packages" + + * *(B)* packages declaring to be part of a group dependency identified by its name + +During a from source build the build tool can use the information from all +package manifests in the workspace to identify group dependencies. +It can use these information to process group members before the packages +declaring the group dependency as if the dependency was declared explicitly +using the existing `depend` tags. + +The group dependency (A) is declared with a `group_depend` tag. +The content of the group dependency tag is the name of the group. +Leading and trailing whitespace is being ignore from the name and for +consistency it is recommended to following the naming rules for packages. + +In order to avoid confusion with the existing `depend` tags (which are also +being used during the release process) the group dependency information is +stored in new tags within the `export` section. + +The membership of a group (B) is declared with a `member_of_group` tag. + +ABI version attribute +--------------------- + +The early draft of REP 127 proposed an attribute (`abi_version`) to identify +the package version which is considered "compatible". + +Since no consensus could be reached about specifying ABI compatibility +in general (for packages as well as libraries) the proposed attribute was +removed from the draft (see related discussions [13]_, [14]_, [15]_). + +Since then the number of packages in a ROS distributions has increased +(e.g. Indigo has more than 2500 packages) as well as the number of platforms +ROS provides binary packages for. +Therefore the need to avoid unnecessary rebuilds has increased. +It is also desired to be able to encourage more frequent releases if they +don't require downstream packages to be rebuild. + +Dependencies across ROS 1 and ROS 2 +----------------------------------- + +.. note:: figure out if there is a way to support this use case without + significant drawbacks + +.. raw:: html + + + + +Data Representation +=================== + +The package manifest is an XML file, with restricted syntax. + +The only top-level element allowed is ````. Immediately +subordinate to that are several required or optional elements, defined +here. No other tags are permitted directly under the ```` +element. + +.. raw:: html + + + + +-------------------- + +The ```` tag is the unique top-level tag in a ``package.xml`` +file. All other tags are nested under it. + +.. raw:: html + + + +Attributes +'''''''''' + + ``format="NUMBER"`` + + Specifying the ``package.xml`` format being used. If not set, + ``format="1"`` would be assumed, which is *not* the format described + here. For this interface, you *must* specify ``format="3"``. If + you are using a different format, please refer to the relevant + specification. REP-0127 [1]_ described format one, REP-0140 [12]_ format two. + +Required Tags +------------- + +The required tags in a ``package.xml`` file provide package meta-data: + + * `\`_ + * `\`_ + * `\`_ + * `\ (multiple, but at least one)`_ + * `\ (multiple, but at least one)`_ + +Optional Tags +------------- + +Meta-information +'''''''''''''''' + +Optional, but recommended, meta-data include the names of the original +authors and links to support documentation. + + * `\ (multiple)`_ + * `\ (multiple)`_ + +Dependencies +'''''''''''' + +The dependencies and relations to other packages and system packages +have been discussed in [7]_. They are described using: + + * `\ (multiple)`_ + * `\ (multiple)`_ + * `\ (multiple)`_ + * `\ (multiple)`_ + * `\ (multiple)`_ + * `\ (multiple)`_ + * `\ (multiple)`_ + * `\ (multiple)`_ + * `\ (multiple)`_ + * `\ (multiple)`_ + +These dependency tags are used with both system packages and ROS +packages. For system dependencies specify the rosdep key name, for +ROS dependencies use the package name. + +The dependency graph must be acyclic. No package may directly or +indirectly depend on itself. + +Group dependencies and memberships +'''''''''''''''''''''''''''''''''' + + .. raw:: html + + + + The group dependencies and membership of groups are described using: + + * `\ (multiple)`_ + * `\ (multiple)`_ + + These group dependencies are only applied in from-source builds and are being + ignore in the release process. + + The dependency graph must be acyclic even when considering group dependencies. + + .. raw:: html + + + +Various +''''''' + +There is a need for additional meta-data in the manifest for other +tools that work with packages, like message generators and plugin +discovery. Tags for that kind of information are wrapped within this +tag: + + * `\`_ + +Some ```` tags used by catkin are defined below. Others are +defined by various tools, which must specify their own specific tag +structures. + +Example +------- + +:: + + + my_package + 1.2.3 + + This is my package's description. + + Someone + + BSD + LGPL + + http://wiki.ros.org/my_package + http://www.github.com/my_org/my_package + http://www.github.com/my_org/my_package/issues + John Doe + Jane Doe + + catkin + genmsg + + roscpp + + libgstreamer0.10-dev + libgstreamer0.10-dev + libgstreamer0.10-0 + + gtest + + doxygen + + alternative_implementation + + my_old_package + + + ... + + + + + +------ + +The package name must start with a letter and contain only lowercase +alphabetic, numeric or underscore characters [2]_. The package name +should be unique within the ROS community. It may differ from the +folder name into which it is checked out, but that is *not* recommended. + +The following recommended exemptions apply, which are optional for +implementations: + +- Dashes may be permitted in package names. This is to support + maintaining a consistent dependency name when transitioning back + and forth between a system dependency and in-workspace package, + since many rosdep keys contain dashes (inherited from the + Debian/Ubuntu name). +- In support of some legacy packages, capital letters may also be + accepted in the package name, with a validation warning. + + + +--------- + +The version number of the package in the format ``MAJOR.MINOR.PATCH`` +where each part is numeric only. + +Attributes +'''''''''' + + .. raw:: html + + + + ``compatibility="MAJOR.MINOR.PATCH"`` + + Specifying the version the package is compatible with. + If not set, the same value as specified in the ``version`` tag is assumed. + + "Compatibility" in this context guarantees that downstream packages built + against the older version will continue to work with a newer version + without the need to be recompiled. + This includes but is not limited to ABI compatibility. + Changes in other parts of a package (CMake, Python, etc.) could also + require downstream packages to be rebuild and therefore not qualify as + "compatible". + + .. raw:: html + + + + +------------- + +The description of the package. It can consist of multiple lines and +may contain XHTML. But depending on where the description is used +XML tags and multiple whitespaces might be stripped. + + + (multiple, but at least one) +----------------------------------------- + +The name of the person maintaining the package. All packages require +a maintainer. For orphaned packages see below. + +Attributes +'''''''''' + + ``email="name@domain.tld"`` *(required)* + + Email address of the maintainer. + +An orphaned package is one with no current maintainer. +Orphaned packages should use the following maintainer information to +guide volunteers how they can claim maintainership: + +Example +''''''' + +:: + + Unmaintained see http://wiki.ros.org/MaintenanceGuide#Claiming_Maintainership + + + (multiple, but at least one) +-------------------------------------- + +Name of license for this package, e.g. BSD, GPL, LGPL. In order to +assist machine readability, only include the license name in this tag. +For multiple licenses multiple separate tags must be used. A package +will have multiple licenses if different source files have different +licenses. Every license occurring in the source files should have +a corresponding ```` tag. For any explanatory text about +licensing caveats, please use the ```` tag. + +Most common open-source licenses are described on the +`OSI website `_. + +Commonly used license strings: + + - Apache 2.0 + - BSD + - Boost Software License + - GPLv2 + - GPLv3 + - LGPLv2.1 + - LGPLv3 + - MIT + - Mozilla Public License Version 1.1 + + (multiple) +---------------- + +A Uniform Resource Locator for the package's website, bug tracker or +source repository. + +It is a good idea to include ```` tags pointing users to these +resources. The website is commonly a wiki page on ``ros.org`` where +users can find and update information about the package. + +Attributes +'''''''''' + + ``type="TYPE"`` *(optional)* + + The type should be one of the following identifiers: ``website`` + (default), ``bugtracker`` or ``repository``. + + + (multiple) +------------------- + +The name of a person who is an author of the package, as +acknowledgement of their work and for questions. + +Attributes +'''''''''' + + ``email="name@domain.tld"`` *(optional)* + + Email address of author. + + +Dependency tags +--------------- + + (multiple) +''''''''''''''''''''''''' + +Declares a rosdep key or ROS package name that this package requires +at build-time. For system packages, the rosdep key will normally +specify the "development" package, which frequently ends in ``"-dev"``. + +The ``build`` and ``buildtool`` dependencies are used to determine +the build order of multiple packages. + +Attributes +.......... + + All dependencies and relationships may restrict their applicability + to particular versions. For each comparison operator an attribute + can be used. Two of these attributes can be used together to + describe a version range. + + ``version_lt="VERSION"`` *(optional)* + + The dependency to the package is restricted to versions less than + the stated version number. + + ``version_lte="VERSION"`` *(optional)* + + The dependency to the package is restricted to versions less or + equal than the stated version number. + + ``version_eq="VERSION"`` *(optional)* + + The dependency to the package is restricted to a version equal than + the stated version number. + + ``version_gte="VERSION"`` *(optional)* + + The dependency to the package is restricted to versions greater or + equal than the stated version number. + + ``version_gt="VERSION"`` *(optional)* + + The dependency to the package is restricted to versions greater than + the stated version number. + + + (multiple) +'''''''''''''''''''''''''''''''' + +Declares a rosdep key or ROS package name that this package needs as +part of some build interface it exports. For system packages, the +rosdep key will normally specify the "development" package, which +frequently ends in ``"-dev"``. + +The ```` declares a transitive build dependency. A +common example is when one of your dependencies provides a header file +included in some header exported by your package. Even if your +package does not use that header when building itself, other packages +depending on your header *will* require those transitive dependencies +when they are built. + +Attributes +.......... + + The same attributes as for `\ (multiple)`_. + + + (multiple) +''''''''''''''''''''''''''''' + +Declares a rosdep key or ROS package name for a tool that is executed +during the build process. For cross-compilation, one must distinguish +these from normal build dependencies, which may be linked with your +package and must be compiled for the target architecture, not the +build system. For system packages, the rosdep key will normally +specify the "development" package, which frequently ends in +``"-dev"``. + + +Attributes +.......... + + The same attributes as for `\ (multiple)`_. + + + (multiple) +'''''''''''''''''''''''''''''''''''' + +Declares a rosdep key or ROS package name that this package exports +which must be compiled and run on the build system, not the target +system. For cross-compilation, one must distinguish these from target +build dependencies, which may be linked with your package and must be +compiled for the target architecture. + + +Attributes +.......... + + The same attributes as for `\ (multiple)`_. + + + (multiple) +'''''''''''''''''''''''' + +Declares a rosdep key or ROS package name that this package needs at +execution-time. For system packages, the rosdep key will normally +*not* specify the "development" package, so it will generally lack the +``"-dev"`` suffix. + +The ```` is needed for packages providing shared +libraries, executable commands, Python modules, launch scripts or any +other files required for running your package. It is also used by +metapackages for grouping packages. + +Attributes +.......... + + The same attributes as for `\ (multiple)`_. + + + (multiple) +''''''''''''''''''' + +Declares a rosdep key or ROS package name that this package needs for +multiple reasons. A ```` tag is equivalent to specifying +````, ```` and ````, +all on the same package or key. The ```` tag cannot be used +in combination with any of the three equivalent tags for the same +package or key name. + +Attributes +.......... + + The same attributes as for `\ (multiple)`_. + + + (multiple) +''''''''''''''''''''''' + +Declares a rosdep key or ROS package name that your package needs for +building its documentation. A ```` *may* reference a +package also declared as some other type of dependency. + +The current version of the buildsystem does not provide any +documentation specific functionality or targets but may do so in the +future, similar to how the unit tests are integrated into the configure +and make steps. Other infrastructure (like the documentation jobs on +the buildfarm) will utilize these additional doc dependencies. + +Generated Debian packages are built without the documentation or the +documentation dependencies. + +Attributes +.......... + + The same attributes as for `\ (multiple)`_. + + + (multiple) +'''''''''''''''''''''''' + +Declares a rosdep key or ROS package name that your package needs +for running its unit tests. A ```` *may* reference a +package also declared as some other type of dependency. + +All tests and their dependencies will be built if the CMake variables +``CATKIN_ENABLE_TESTING=1`` and ``CATKIN_SKIP_TESTING=0``, the default +settings. ``CMakeLists.txt`` should only define its test targets when +``CATKIN_ENABLE_TESTING=1`` [9]_. + +When building with testing enabled, the ```` packages are +available for configuring and building the tests as well as running +them. Generated Debian packages are built without the unit tests or +their dependencies. + +Attributes +.......... + + The same attributes as for `\ (multiple)`_. + + + (multiple) +''''''''''''''''''''' + +Declares a rosdep key or ROS package name with which your package +conflicts. This package and the conflicting package should not be +installed at the same time. This has no effect on source builds, but +maps to ``Conflicts`` when creating Debian or RPM packages. + +For a detailed explanation how these relationships are used see +[4]_ and [5]_. + +Attributes +.......... + + The same attributes as for `\ (multiple)`_. + + + (multiple) +'''''''''''''''''''' + +Declares a rosdep key or ROS package name that your package replaces. +This has no effect on source builds, but maps to ``Replaces`` when +creating Debian packages and ``Obsoletes`` for RPM packages. + +Attributes +.......... + + The same attributes as for `\ (multiple)`_. + + +.. raw:: html + + + + +Group dependency tags +--------------------- + + (multiple) +''''''''''''''''''''''''' + +The content identifies the name of the dependency group. +The group name should follow the same rules as the `\`_ tag. + + (multiple) +'''''''''''''''''''''''''''' + +The content identifies the name of the dependency group. + +.. raw:: html + + + + + +-------- + +This tag serves as a container for additional information various +packages and subsystems need to embed. To avoid potential collisions, +an export tag should have the same name as the package which is meant +to process it. The content of that tag is up to the package to define +and use. + +Existing rosbuild export tags for tools using ``pluginlib`` remain +unchanged. For example, a package which implements an rviz plugin +might include this:: + + + + + +The following are some tags used within an ```` for various +package and message generation tasks. + + +''''''''''''''''''''''''''' + +This empty tag indicates that your package produces no +architecture-specific files once built. +That information is intended for allowing optimization of packaging. + +Specifying ```` is recommended for +metapackages and for packages defining only ROS messages and services. +Python-only packages are reasonable candidates, too. +It is not appropriate for any package which compiles C or C++ code. + +Be sure to remove this tag if some subsequent update adds +architecture-dependent targets to a formerly independent package. + + +'''''''''''' + +Various tools use this tag to determine how to handle a package. It +was defined in REP-0134 [10]_, which currently specifies only two +valid values:: + + catkin + cmake + +If no ```` is provided, ``catkin`` is assumed. + +When the build type is ``cmake``, the package is handled as a +non-catkin CMake project. It cannot be included in a normal catkin +workspace, but can instead use ``catkin_make_isolated``, which +configures and builds a different kind of workspace in which +``cmake``, ``make``, and ``make install`` are invoked separately for +each package. See REP-0134 for details. + +Further build types may eventually be defined, such as: "make", +"autotools", "rosbuild", or "custom". + + + +'''''''''''' + +This tag indicates that your package is deprecated, enabling tools to +notify users about that fact. The tag may be empty or may optionally +contain an arbitrary text providing user more information about the +deprecation:: + + + + This package will be removed in ROS Hydro. Instead, use package + FOO, which provides similar features with a different API. + + + + +''''''''''''''''''' + +The content defines the *identifier* for the language bindings +generated by this package, i.e. in ``gencpp`` this is set to ``cpp``:: + + + cpp + + + +'''''''''''''' + +This empty tag declares a special kind of catkin package used for +grouping other packages. Metapackages only provide execution-time +dependencies. They cannot be used for catkin builds and compile +nothing themselves. Metapackages may not install any code or other +files, although ``package.xml`` does get installed automatically. +They can depend on other metapackages, but regular catkin packages +cannot. + +A good use for metapackages is to group the major components of your +robot and then provide a comprehensive grouping for your whole system. +Package installation tools like ``apt-get`` or ``yum`` can +automatically install all the packages on which a metapackage directly +or indirectly depends. Metapackages can also be used to resolve +dependencies declared by legacy rosbuild stacks not yet converted to +catkin. + +Every metapackage must have a ``CMakeLists.txt`` containing these +commands:: + + cmake_minimum_required(VERSION 2.8.3) + project(PACKAGE_NAME) + find_package(catkin REQUIRED) + catkin_metapackage() + +Because the metapackage ``CMakeLists.txt`` contains a catkin macro, +its ``package.xml`` must declare a buildtool dependency on catkin:: + + catkin + +Additional buildtool, build or test dependencies are not permitted. + +Because metapackages only supply execution-time dependencies, they use +```` to list the packages in their group:: + + your_custom_msgs + your_server_node + your_utils + another_metapackage + + +Compatibility +============= + +.. raw:: html + + + +Support for format three +------------------------ + +Existing tools supporting up to format two will need to be updated to +honor the new information provided by format three. +If they are not checking that the format is two or lower they will +simply ignore the format three specific information when processing a +package with a format three manifest. + +catkin_prepare_release +'''''''''''''''''''''' + +As long as a manifest doesn't specify a `compatibility` version the tools +doesn't need to change. +For packages which do specify a `compatibility` version the tool should +probably by default remove the attribute and only after confirmation from +the user offer to keep it. + +ros_buildfarm +''''''''''''' + +The dependencies between packages is directly mapped to upstream / +downstream jobs in Jenkins. +In order to consider the `compatibility` attribute the downstream job +dependencies must be changed to be conditional which are only triggered +when the new package version is not compatible to the previous version. + + +Backward compatibility +---------------------- + +Format one or two packages following REP-0127 [1]_ or REP-0140 [12]_ are +not affected unless they are updated to declare ````. + +Since format three only adds new functionality and doesn't modify any +existing markup a migration to format three only makes sense when the +package wants to use any of the new features. + +.. raw:: html + + + + +Schema +====== + +.. raw:: html + + + +A schema defining the structure specified in this document is available +at [11]_. +To specify the schema within a manifest you can reference a self +contained schema file like this: + + + + + +.. raw:: html + + + + +References +========== + +.. [1] REP-0127 + (http://ros.org/reps/rep-0127) +.. [2] ROS naming conventions + (http://www.ros.org/wiki/ROS/Patterns/Conventions#Naming_ROS_Resources) +.. [3] ros-infrastructure/catkin_pkg#43: `"add support for depend tag" + `_ +.. [4] Declaring relationships between packages (Debian Policy Manual) + (http://www.debian.org/doc/debian-policy/ch-relationships.html) +.. [5] Advanced RPM Packaging (Fedora Documentation) + (http://docs.fedoraproject.org/en-US/Fedora_Draft_Documentation/0.1/html/RPM_Guide/ch-advanced-packaging.html) +.. [6] Buildsystem mailing list discussion: `"adding syntax to package.xml" + `_ +.. [7] Buildsystem mailing list discussion: `"Dependency tag types for REP 127" + `_ +.. [8] Buildsystem mailing list discussion: `"dev/non-dev packages and required meta information" + `_ +.. [9] Buildsystem mailing list discussion: `"REP-0140: internal review" + `_ +.. [10] REP-0134 + (http://ros.org/reps/rep-0134) +.. [11] Schema file + (https://github.com/ros-infrastructure/rep/blob/master/xsd/package_format3.xsd) +.. [12] REP-0140 + (http://ros.org/reps/rep-0140) +.. [13] Discussion on REP-0127 + (https://groups.google.com/forum/#!topic/ros-sig-buildsystem/_jRvhXFfsVk) +.. [14] Related topic of versioning ROS libraries + (https://groups.google.com/forum/#!topic/ros-sig-buildsystem/Q9BK3MGFY_U) +.. [15] SO versioning from a package perspective + (https://groups.google.com/forum/#!topic/ros-sig-buildsystem/jTB9r3zu580) + + +Copyright +========= + +This document has been placed in the public domain. + + + +.. + Local Variables: + mode: indented-text + indent-tabs-mode: nil + sentence-end-double-space: t + fill-column: 70 + coding: utf-8 + End: From b98829729af2b3103a6d2297dc64fbdbbd58bc51 Mon Sep 17 00:00:00 2001 From: William Woodall Date: Fri, 20 Oct 2017 12:29:50 -0700 Subject: [PATCH 02/10] Update rep-0149.rst --- rep-0149.rst | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/rep-0149.rst b/rep-0149.rst index f794810a..d9306654 100644 --- a/rep-0149.rst +++ b/rep-0149.rst @@ -72,12 +72,12 @@ ABI version on the package level When building binary packages on the ROS buildfarm there is no assumption about ABI compatibility. When a new Debian package has been built all downstream packages are being -rebuild too in order to ensure they still work together. +rebuilt too in order to ensure they still work together. This makes e.g. releasing a core package like `catkin` or `roscpp` particularly costly. Due to that overhead changes which should be released rather sooner than -later, e.g. a documentation fix, simple bugfix in a `.cpp` file, are being -hold back to group them with other changes to make the rebuild effort +later, e.g. a documentation fix or simple bugfix in a `.cpp` file, are being +held back to group them with other changes to make the rebuild effort worthwhile. In order to allow package releases without enforcing a rebuild of all @@ -141,7 +141,7 @@ The reverse dependency can then be specified in two parts: During a from source build the build tool can use the information from all package manifests in the workspace to identify group dependencies. -It can use these information to process group members before the packages +It can use this information to process group members before the packages declaring the group dependency as if the dependency was declared explicitly using the existing `depend` tags. @@ -171,7 +171,7 @@ Since then the number of packages in a ROS distributions has increased ROS provides binary packages for. Therefore the need to avoid unnecessary rebuilds has increased. It is also desired to be able to encourage more frequent releases if they -don't require downstream packages to be rebuild. +don't require downstream packages to be rebuilt. Dependencies across ROS 1 and ROS 2 ----------------------------------- @@ -389,7 +389,7 @@ Attributes without the need to be recompiled. This includes but is not limited to ABI compatibility. Changes in other parts of a package (CMake, Python, etc.) could also - require downstream packages to be rebuild and therefore not qualify as + require downstream packages to be rebuilt and therefore not qualify as "compatible". .. raw:: html From bca396513fa434fe33592c1c0be069d3b1007391 Mon Sep 17 00:00:00 2001 From: Dirk Thomas Date: Mon, 23 Oct 2017 14:27:29 -0700 Subject: [PATCH 03/10] update based on feedback --- rep-0149.rst | 39 ++++++++++++++++++--------------------- 1 file changed, 18 insertions(+), 21 deletions(-) diff --git a/rep-0149.rst b/rep-0149.rst index d9306654..64c1b831 100644 --- a/rep-0149.rst +++ b/rep-0149.rst @@ -119,25 +119,26 @@ for format one and two, which is not repeated here. Group dependencies ------------------ -While "normal" dependencies are specified in the package which expresses a -dependency on another package this is not adequate for the targeted goal. -Instead packages need to be able to declare that they are a dependency of -another package. +Packages can already express direct dependencies on other individual packages, +but this is not sufficient for cases where a package needs to be built after +all instances of a particular kind of package. +Instead packages need to be able to declare a dependency on packages without +using their name directly. -For that kind of "reverse" dependency a package could use the name of the other -package. -But that approach would fail if the other package gets forked under a different -name since the reverse dependency would still reference the original package -name. +One alternative is to "reverse" the dependency declaration so that a package +states that it "wants" to be a dependency of another package. +But that approach does not work in the case where the other package gets forked +under a different name since the "reverse" dependency would still only +reference the original package name. -Therefore a reverse dependency is not using the package name but rather a -separate identify which will be called "group dependency name". -The reverse dependency can then be specified in two parts: +Therefore the dependency declaration is being decoupled into two parts: * *(A)* packages declaring a dependency on a "group dependency name", e.g. - "message_packages" + `ros1_bridge` can depend on the group "message_packages" - * *(B)* packages declaring to be part of a group dependency identified by its name + * *(B)* packages declaring to be part of a group dependency identified by its + name, e.g. `sensor_msgs` can declare itself part of the "message_packages" + group During a from source build the build tool can use the information from all package manifests in the workspace to identify group dependencies. @@ -150,10 +151,6 @@ The content of the group dependency tag is the name of the group. Leading and trailing whitespace is being ignore from the name and for consistency it is recommended to following the naming rules for packages. -In order to avoid confusion with the existing `depend` tags (which are also -being used during the release process) the group dependency information is -stored in new tags within the `export` section. - The membership of a group (B) is declared with a `member_of_group` tag. ABI version attribute @@ -386,7 +383,7 @@ Attributes "Compatibility" in this context guarantees that downstream packages built against the older version will continue to work with a newer version - without the need to be recompiled. + without the need to be rebuilt. This includes but is not limited to ABI compatibility. Changes in other parts of a package (CMake, Python, etc.) could also require downstream packages to be rebuilt and therefore not qualify as @@ -711,13 +708,13 @@ Group dependency tags (multiple) ''''''''''''''''''''''''' -The content identifies the name of the dependency group. +The content is the name of a dependency group on which the package depends. The group name should follow the same rules as the `\`_ tag. (multiple) '''''''''''''''''''''''''''' -The content identifies the name of the dependency group. +The content is the name of a dependency group of which the package is a member. .. raw:: html From 9d1936b096a4e023078caba7288e9fa376c95292 Mon Sep 17 00:00:00 2001 From: Dirk Thomas Date: Tue, 31 Oct 2017 10:14:38 -0700 Subject: [PATCH 04/10] add information about dependencies for ROS 1 and ROS 2 --- rep-0149.rst | 65 +++++++++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 62 insertions(+), 3 deletions(-) diff --git a/rep-0149.rst b/rep-0149.rst index 64c1b831..a00ac8a9 100644 --- a/rep-0149.rst +++ b/rep-0149.rst @@ -91,7 +91,13 @@ In order for a ROS package to work with ROS 1 and ROS 2 from a single source the dependencies listed in the package manifest are a problem. E.g. in ROS 1 a package needs to depend on `roscpp` where as in ROS 2 it needs to depend on `rclcpp`. -Currently the manifest doesn't support this use case. +Currently the manifest doesn't support this use case and requires two different +code bases (usually two separate branches). + +The necessary part outside of the manifest to make a package compatible with +ROS 1 as well as ROS 2 is explicitly not part of this document. +In general it is possible to conditionally handle those cases in programming +languages like CMake, C++, Python, etc. .. raw:: html @@ -173,8 +179,59 @@ don't require downstream packages to be rebuilt. Dependencies across ROS 1 and ROS 2 ----------------------------------- -.. note:: figure out if there is a way to support this use case without - significant drawbacks +The differences in dependencies in ROS 1 and ROS 2 could be handled in various +ways. + +Use the same package names +'''''''''''''''''''''''''' + +One option (A) would be to not allow different dependencies depending on the +ROS version. +That would imply that for the previously described example of `roscpp` and +`rclcpp` a "dummy" package could be provided to provide the same package names. +This could be either a ROS 1 package named `rclcpp` which simply depends on +`roscpp` or a ROS 2 package named `roscpp` which depends on `rclcpp`. +That would allow all downstream packages to use a single name as their +dependency. +While that would be possible it would require dummy packages for every naming +difference between ROS 1 and ROS 2. + +But option (A) would become even more cumbersome if a package needs to declare +a dependency only in one of the ROS versions. +It would require a dummy package in both ROS version where as in one of them +the dummy would be empty and in the other it would declare a dependency. + +Annotate dependencies +''''''''''''''''''''' + +Another option (B) is to annotate the dependencies in the manifest if they are +specific to a ROS version. +The annotation could happen in various ways in the markup. +One way is an optional attribute for each dependency tag. +Another way would be a new conditional tag under which one or multiple +dependency tags would be grouped. +In case a dependency doesn't have such an annotation it is being used +unconditionally. + +In this case various tools will need to be aware of the condition choosing +which dependencies should be used: + + * `bloom` + * `rosdep` + * `rosinstall_generator` + * the build tool + +The first three tools are all operating on a specific ROS distribution. +Assume that a ROS distribution "knows" which ROS version it represents (which +would require adding a new field to the distribution file specified in REP 143 +[16]_) the necessary information should be available to decide the condition. + +The build tool does not have access to the ROS distribution metadata. +It could either use information provided by an environment variable or fall +back to consider *all* dependencies (independent of their annotation) and work +on the assumption that the workspace contains a set of consistent packages and +additional dependencies from other ROS versions are not available in the +workspace. .. raw:: html @@ -951,6 +1008,8 @@ References (https://groups.google.com/forum/#!topic/ros-sig-buildsystem/Q9BK3MGFY_U) .. [15] SO versioning from a package perspective (https://groups.google.com/forum/#!topic/ros-sig-buildsystem/jTB9r3zu580) +.. [16] REP-0143 + (http://ros.org/reps/rep-0143) Copyright From ea84a97702804989d017460ecc10e2f83d3d84f3 Mon Sep 17 00:00:00 2001 From: Dirk Thomas Date: Tue, 31 Oct 2017 11:24:42 -0700 Subject: [PATCH 05/10] clone package format 2 schema to format 3 --- xsd/package_format3.xsd | 55 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 55 insertions(+) create mode 100644 xsd/package_format3.xsd diff --git a/xsd/package_format3.xsd b/xsd/package_format3.xsd new file mode 100644 index 00000000..b6a110b1 --- /dev/null +++ b/xsd/package_format3.xsd @@ -0,0 +1,55 @@ + + + + + + + Specified in REP 140 (see http://www.ros.org/reps/rep-0140.html). + + + + + + + + The package name should be unique within the ROS community. + It may differ from the folder name into which it is checked out, + but that is not recommended. + It must start with a lower-case letter and consist of only + lower-case letters, numbers and underscores. + It must not have two consecutive underscores. + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + From 31d52bc08aa69eb4d6bdeeefcb6189291a07edc6 Mon Sep 17 00:00:00 2001 From: Dirk Thomas Date: Tue, 31 Oct 2017 11:52:21 -0700 Subject: [PATCH 06/10] update format 3 schema --- xsd/package_format3.xsd | 16 ++++++++++++++-- 1 file changed, 14 insertions(+), 2 deletions(-) diff --git a/xsd/package_format3.xsd b/xsd/package_format3.xsd index b6a110b1..9a9ba131 100644 --- a/xsd/package_format3.xsd +++ b/xsd/package_format3.xsd @@ -1,10 +1,19 @@ + + + + + + + + + - Specified in REP 140 (see http://www.ros.org/reps/rep-0140.html). + Specified in REP 149 (see http://www.ros.org/reps/rep-0149.html). @@ -47,9 +56,12 @@ + + + - + From d26abde44252d1b784bc37d4eb585ae25e99d64c Mon Sep 17 00:00:00 2001 From: Dirk Thomas Date: Tue, 7 Nov 2017 10:50:24 -0800 Subject: [PATCH 07/10] add condition attribute --- rep-0149.rst | 28 ++++++++++++++++++++++++++++ 1 file changed, 28 insertions(+) diff --git a/rep-0149.rst b/rep-0149.rst index a00ac8a9..639de357 100644 --- a/rep-0149.rst +++ b/rep-0149.rst @@ -588,6 +588,24 @@ Attributes The dependency to the package is restricted to versions greater than the stated version number. + ``condition="CONDITION_EXPRESSION"`` + + Every dependency can be conditional on a condition expression. + If the condition expression evaluate to "true" the dependency is being used + and considered as if it doesn't have a condition attribute. + If the condition expression evaluate to "false" the dependency is being + ignored and considered as if it doesn't exist. + + The condition expression can consist of: + + * parenthesis (which must be balanced) + * logical operators `and` and `or` + * comparison operators `==` and `!=` + * variable names which start with a `$` sign and are followed by + alphanumerics and underscores + * literals which can only contain alphanumerics, undercsores and dashes + * arbitrary whitespaces between these tokens + (multiple) '''''''''''''''''''''''''''''''' @@ -768,11 +786,21 @@ Group dependency tags The content is the name of a dependency group on which the package depends. The group name should follow the same rules as the `\`_ tag. +Attributes +.......... + + The `condition` attribute as defined for `\ (multiple)`_. + (multiple) '''''''''''''''''''''''''''' The content is the name of a dependency group of which the package is a member. +Attributes +.......... + + The `condition` attribute as defined for `\ (multiple)`_. + .. raw:: html From d25ea6f07066b50bcf83bd08a835a8f8077e7f97 Mon Sep 17 00:00:00 2001 From: Dirk Thomas Date: Tue, 7 Nov 2017 15:02:22 -0800 Subject: [PATCH 08/10] add section about to be defined environment variables --- rep-0149.rst | 30 ++++++++++++++++++++++++++++++ 1 file changed, 30 insertions(+) diff --git a/rep-0149.rst b/rep-0149.rst index 639de357..55dc1608 100644 --- a/rep-0149.rst +++ b/rep-0149.rst @@ -932,6 +932,34 @@ Because metapackages only supply execution-time dependencies, they use another_metapackage +Related Changes +=============== + +Environment variables +--------------------- + +In order to identify the ROS distribution ROS 1 already defines an environment +variable `ROS_DISTRO` ([17]_). +ROS 2 should expose the same environment variable. +The package exporting the necessary environment should be a dependency of +almost all ROS 2 packages to ensure that the information is available even when +only some packages are installed. +The package `rcl` seems to be a good place for this. + +Additionally for the condition expressions to allow distinguishing ROS 1 and +ROS 2 an environment variable identifying the major version is necessary. +Therefore the environment variable `ROS_VERSION` is used with the value being +either `1` or `2`. +This new environment variable can be defined beside the `ROS_DISTRO` one. + +Once a specific ROS environment has been sourced all tools can determine the +ROS major version as well as the distribution name and therefore evaluate +conditions which use those to limit the scope of dependencies. + +If now environment has been source some tools might require that the necessary +information is being specified explicitly when being invoked. + + Compatibility ============= @@ -1038,6 +1066,8 @@ References (https://groups.google.com/forum/#!topic/ros-sig-buildsystem/jTB9r3zu580) .. [16] REP-0143 (http://ros.org/reps/rep-0143) +.. [17] ROS_DISTRO environment variable + (https://github.com/ros/ros/blob/b202645dc6bea6d4b9ca408dc703c8c7cc8204d9/core/roslib/env-hooks/10.ros.sh.em#L16) Copyright From 11f7bf8471206ba268c45af0a015e979ea67ca32 Mon Sep 17 00:00:00 2001 From: Dirk Thomas Date: Tue, 7 Nov 2017 15:24:46 -0800 Subject: [PATCH 09/10] mention need to update ROS 1 tools --- rep-0149.rst | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/rep-0149.rst b/rep-0149.rst index 55dc1608..76420374 100644 --- a/rep-0149.rst +++ b/rep-0149.rst @@ -976,6 +976,10 @@ If they are not checking that the format is two or lower they will simply ignore the format three specific information when processing a package with a format three manifest. +In order to enable packages to declare different dependencies for ROS 1 and ROS +2 in a single manifest the tools in ROS 1 (`catkin_pkg`, `rosdep`, `bloom`, +etc.) should be updated to support format 3. + catkin_prepare_release '''''''''''''''''''''' From 2e7172fb5e658afa1d9e5e6a06fee73948ea962a Mon Sep 17 00:00:00 2001 From: Dirk Thomas Date: Fri, 10 Nov 2017 19:48:09 -0800 Subject: [PATCH 10/10] apply feedback --- rep-0149.rst | 17 ++++++++--------- 1 file changed, 8 insertions(+), 9 deletions(-) diff --git a/rep-0149.rst b/rep-0149.rst index 76420374..d61ade29 100644 --- a/rep-0149.rst +++ b/rep-0149.rst @@ -22,7 +22,7 @@ Outline The current draft highlights significant changes compared to REP-0140 [2]_ in red. - Trivial changes appear in green. + Trivial changes appear in blue. Abstract @@ -30,11 +30,10 @@ Abstract .. raw:: html - + -This REP specifies the third ``package.xml`` format, replacing the ones -specified in REP-0127 [1]_ and REP-0140 [2]_. -It is relevant for packages using the catkin as well as the ament build system. +This REP specifies the third ``package.xml`` format, which is an update to the +previous versions specified in REP-0127 [1]_ and REP-0140 [2]_. .. raw:: html @@ -109,7 +108,7 @@ Rationale .. raw:: html - + REP-0127 [1]_ and REP-0140 [1]_ provide the ``package.xml`` design rationale for format one and two, which is not repeated here. @@ -155,7 +154,7 @@ using the existing `depend` tags. The group dependency (A) is declared with a `group_depend` tag. The content of the group dependency tag is the name of the group. Leading and trailing whitespace is being ignore from the name and for -consistency it is recommended to following the naming rules for packages. +consistency it is required to follow the naming rules for packages. The membership of a group (B) is declared with a `member_of_group` tag. @@ -250,7 +249,7 @@ element. .. raw:: html - + -------------------- @@ -1019,7 +1018,7 @@ Schema .. raw:: html - + A schema defining the structure specified in this document is available at [11]_.