T5083: extend xml schema definitions to support child requirements #3575

thomasfinstad · 2024-06-02T15:00:47Z

Change Summary

Adds declarative child requirements support to XML schema definitions, and converts the container service to utilize this new functionality as a proof of concept.

Types of changes

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Code style update (formatting, renaming)
Refactoring (no functional changes)
Migration from an old Vyatta component to vyos-1x, please link to related PR inside obsoleted component
Other (please describe):

Related Task(s)

https://vyos.dev/T5083

Related PR(s)

vyos/vyos1x-config#29

Component(s) name

container

Proposed changes

This extends the XML schema to include an optional block where a node/tagnode can declaratively define simple requirements for children.

Currently 4 requirement types are supported:

require (child must be configured)
conflict (child X can not be configured if child Y is configured)
atLeastOneOf (a list of children where at least one of them must be present)
depend (if child X is configured then child Y must also be configured)

This feature allows the simpler checks to be moved out of the .py file to clear it up and make more room for the more complex checks without extra clutter. It will also allow 3rd party tooling to better understand vyos command / api structure via the schemas.

While this PR contains changes for the container service, other services can be migrated over time.

How to test

Extra tests to assure expected behavior has been added to the container smoke test.

Smoketest result

$ /usr/libexec/vyos/tests/smoke/cli/test_container.py 
test_basic (__main__.TestContainer.test_basic) ... ok
test_dual_stack_network (__main__.TestContainer.test_dual_stack_network) ... 
IP address "192.0.2.1" can not be used for a container, reserved for the
container engine!

ok
test_image (__main__.TestContainer.test_image) ... 
[container name image-test] Requires "image" to be configured

ok
test_ipv4_network (__main__.TestContainer.test_ipv4_network) ... 
IP address "192.0.2.1" can not be used for a container, reserved for the
container engine!

ok
test_ipv6_network (__main__.TestContainer.test_ipv6_network) ... 
IP address "2001:db8::1" can not be used for a container, reserved for
the container engine!

ok
test_network_or_host_networks (__main__.TestContainer.test_network_or_host_networks) ... 
[container name image-test] "allow-host-networks" and "network" can not
be configured at the same time

ok
test_network_required (__main__.TestContainer.test_network_required) ... 
[container name image-test] Requires at least one of "allow-host-
networks" or "network" to be configured

ok
test_uid_gid (__main__.TestContainer.test_uid_gid) ... 
[container name uid-test] Can not configure "gid" without "uid"

ok

----------------------------------------------------------------------
Ran 8 tests in 122.401s

OK

Checklist:

I have read the CONTRIBUTING document
I have linked this PR to one or more Phabricator Task(s)
I have run the components SMOKETESTS if applicable
My commit headlines contain a valid Task id
My change requires a change to the documentation
I have updated the documentation accordingly

dmbaturin

The idea is interesting, but there are some points to think about and discuss.

dmbaturin · 2024-06-02T16:43:30Z

schema/interface_definition.rnc

+# requirements of node or tagnode children.
+childRequirements = element childRequirements {
+    (element require { child+ } )? &
+    (element conflict { nodeNameAttr, child+ })* &


One issue I see is that nodeNameAttr is now assumed to always be a single word, but conflicts are often between children at different levels

My feeling is that there are actually two different situations we need to handle, if we want to generate helpful error messages.

The first one is a simple conflict when two nodes cannot appear together. For example, route-reflector-client and route-server-client in BGP — a neighbor cannot be both, under any circumstances. We may use something like <mutuallyExclusiveChildren> for that. In that case if both are found, we can show a message like $nodeA and $nodeB cannot be configured at the same time if any nodes from the list are found.

The second situation is a context-dependent conflict. For example, in OpenVPN, the server node and any of its children like server topology conflict with mode client and mode site-to-site. It's possible for nodes to conflict with nodes in general or with specific valued of leaf nodes.

If we find a way to handle both, that may indeed limit the need for in-script validation to ultra-context-dependent things like route-reflector-client allowed only if the local AS is the same as the remote AS (i.e., the peer is an iBGP peer).

The way that you are proposing will lead to a combinatorial explosion when there are more than two conflicting options.

The first one is a simple conflict when two nodes cannot appear together. For example, route-reflector-client and route-server-client in BGP — a neighbor cannot be both, under any circumstances. We may use something like <mutuallyExclusiveChildren> for that. In that case if both are found, we can show a message like $nodeA and $nodeB cannot be configured at the same time if any nodes from the list are found.

It would not cover cases where there are 3 children X, Y and Z where X can not be configured with Y or Z, but Y and X can both be configured as long as X is not.
This might however be less useful that what you suggest having a simple list of all mutually exclusive / conflicting children.
I have no issue with reworking (element conflict { nodeNameAttr, child+ })* & to be (element mutuallyExclusiveChildren { child+ } )? & if that is preferred by those of you that know the system and its needs better than me.

One issue I see is that nodeNameAttr is now assumed to always be a single word, but conflicts are often between children at different levels
[...]
The second situation is a context-dependent conflict. For example, in OpenVPN, the server node and any of its children like server topology conflict with mode client and mode site-to-site. It's possible for nodes to conflict with nodes in general or with specific valued of leaf nodes.

If we find a way to handle both, that may indeed limit the need for in-script validation to ultra-context-dependent things like route-reflector-client allowed only if the local AS is the same as the remote AS (i.e., the peer is an iBGP peer).

While what I have here does not currently support this level of logic I am willing to take a stab at creating the required logic if you believe it will be a useful feature to have declarative configurations for conflicting descendants of different levels. The reason I did not add it is because my use-case will not benefit from it, but I would be happy to add it.

Does these changes sounds reasonable to accept?

dmbaturin · 2024-06-02T16:45:21Z

schema/interface_definition.rnc

+    (element require { child+ } )? &
+    (element conflict { nodeNameAttr, child+ })* &
+    (element atLeastOneOf { child+ } )* &
+    (element depend { nodeNameAttr, child+ })*


I'm not sure if we should use both require and depend, from the maintainability perspective.

I assume what you mean here is that since both fields can be intepreted as "this node must have these children configured" it can be confusing when not taken with extra context, which will waste mental bandwidth.

If that is the case what about (element requiredChildren { child+ } )? & and (element childDependsOn { nodeNameAttr, child+ })* instead?

github-actions · 2024-06-11T03:51:18Z

This pull request has conflicts, please resolve those before we can evaluate the pull request.

jestabro

I agree that this is an interesting idea; the implementation is nicely done. However, I am concerned that it simply internalizes the complexity of inter-node dependence from the conf-mode script verify stage to the XML definitions and overhead of the ConfigDict class: this poses two immediate issues:
(1) will authors of conf-mode scripts tend to write verify checks simply, rather than properly, according to this design, leaving it underused
(2) the introduction of the ConfigDict to handle defaults tracking (T5330) was meant as a lightweight overhead --- subsequent problems have suggested packing class attributes with further bookkeeping, but in each case I considered that to indicate a mistaken design: my sense is that if we had a general definition of inter-node dependencies (cf. comments of @dmbaturin), we could justify the overhead, but lacking that, I am hesitant.

I agree this requires further discussion to consider.

thomasfinstad · 2024-06-17T15:03:01Z

I am currently working on expanding the capabilities as suggested by @dmbaturin where I am looking to add features to allow internode specifications as well as some basic support for leafnode values eg: if mode = X then child Y must be configured. (but is not intended to replace the allowed values constraint <constraint><regex>(tcp|udp)</regex></constraint>)

Due to these new features renamed it from child_requirements to child_specification, but the idea is the same.

I hope that all non-dynamic type of requirements would then be possible to have declaratively defined. Currently this is the schema I am working towards supporting:

properties = element properties
{
...
    # These are meaningful only for tag and node nodes
    childSpecification? &
...
}

childSpecificationDescendant = (
    (
        (element descendant { nodeNameAttr, childSpecificationDescendant+ })+ |
        (element child { text })+
    )+
)

childSpecificationDescendantWithValue = (
    (
        (element descendant { nodeNameAttr, childSpecificationDescendantWithValue+ })+ |
        (element child { text })+ |
        (element value { text })+
    )+
)

childSpecification = element childSpecification
{
    (element requiredChildren { childSpecificationDescendant } )? &
    (element atLeastOneOf { childSpecificationDescendant })* &
    (element mutuallyExclusiveChildren { childSpecificationDescendantWithValue })* &
    (element mutuallyDependantChildren { childSpecificationDescendantWithValue })* &
    (element oneWayDependantChildren {
        element dependants { childSpecificationDescendantWithValue }+ &
        element dependees { childSpecificationDescendantWithValue }+
    })*
}

So here is a snippet of vyos-1x/interface-definitions/interfaces_openvpn.xml.in:

<?xml version="1.0"?>
<interfaceDefinition>
  <node name="interfaces">
    <children>
      <tagNode name="openvpn" owner="${vyos_conf_scripts_dir}/interfaces_openvpn.py">
        <properties>
...
          <childSpecification>
            <mutuallyExclusiveChildren>
              <child>server</child>
              <descendant name="mode">
                <value>site-to-site</value>
                <value>client</value>
              </descendant>
            </mutuallyExclusiveChildren>
            <mutuallyExclusiveChildren>
              <descendant name="mode">
                <value>client</value>
              </descendant>
              <descendant name="tls">
                <child>dh-params</child>
              </descendant>
            </mutuallyExclusiveChildren>
            <mutuallyDependantChildren>
              <descendant name="authentication"><child>username</child></descendant>
              <descendant name="authentication"><child>password</child></descendant>
            </mutuallyDependantChildren>
          </childSpecification>
        </properties>
        <children>
          #include <include/interface/authentication.xml.i>
          #include <include/generic-description.xml.i>
...

As you can see setting the username under authentication will require password to be set, and vise versa. As you can also see this enables configuration of elements that are included from generic interface files that might be used in cases where the children are not in need of the same configuration. (I do not know if that is the case with <include/interface/authentication.xml.i>, but I did not wish to go full hog on implementing the changes until I know if the changes will be accepted.

I would love to know if these changes are acceptable, if changes need to be made, or if this is a dead end and I need to find a different way to fulfill my use-case. (which is a terraform provider: https://registry.terraform.io/providers/thomasfinstad/vyos-rolling/latest/docs)

github-actions · 2024-06-19T22:03:18Z

Conflicts have been resolved. A maintainer will review the pull request shortly.

github-actions · 2024-06-19T22:03:24Z

👍
No issues in PR Title / Commit Title

thomasfinstad · 2024-06-19T22:10:25Z

I pushed up what I currently have to get some more input.

What I think might be a good idea:

add "and" type option for one-way-dependancy for possibility for "if X and Y is configured then Z must also be configured"
add "and" type option for a new one-way-exclusive config for "if X and Y is configured then Z can not be configured
change the decendant element name to in to make it easier to read, and waste less space

I hope these changes (current new code + what I suggest above) gives enough of a picture of how it will end up if we wish move forward with this.

Edit
I also think it might be easier to use and read if I merge the child and value elements into one <child name="X" value="Y"> where the value part is optional. changing for example this:

<mutuallyExclusiveChildren>
  <descendant name="mode">
    <value>client</value>
  </descendant>
  <descendant name="tls">
    <child>dh-params</child>
  </descendant>
<mutuallyExclusiveChildren>

into this

<mutuallyExclusiveChildren>
  <child name="mode" value="client"/>
  <in name="tls">
    <child name="dh-params"/>
  </in>
<mutuallyExclusiveChildren>

thomasfinstad · 2024-06-21T13:17:14Z

add "and" type option for one-way-dependancy for possibility for "if X and Y is configured then Z must also be configured"

add "and" type option for a new one-way-exclusive config for "if X and Y is configured then Z can not be configured

One other option would be to utilize logical operators to fulfill required flexibility instead. Meaning adding in generic and, or and not blocks containing the children. This would allow for very flexible configuration.

dmbaturin

@thomasfinstad Please don't feel discouraged — this is just a really big and interesting problems that takes time and iterations! I've been thinking about it a lot lately, now I'm ready to offer some suggestions.

I'm wondering if we can find a better terminology that will allow more concise and expressive syntax.

I left a comment with an example there in the review. My thinking is that we may want to think in terms of just two entities: requirements and conflicts.

Here's what a requirement may look like:

<requirement>
  <source node="mode" value="client" />
  <requiredSibling>remote-host</requiredSibling>
  <requiredDescendant>...

Conflicts also have a source and a bunch of things that must not appear when the source node exists (and, optionally, has a particular value) — see an example above.

I've also been thinking about mutually exclusive children, and I wonder if the possibility of them is a sign that the CLI can be improved. For example, in BGP we have set protocols bgp neighbor x.x.x.x address-family ipv4-unicast route-server-client and set protocols bgp neighbor x.x.x.x address-family ipv4-unicast route-reflector-client.

Should we? If we make both those options values or something like role (as it's done elsewhere in BGP), the illegal state will be unrepresentable.

BGP is also an example where full validation is impossible without a Turing-complete check. One cannot set route-reflector-client unless local-as == ../system-as, since it only makes sense in iBGP — eBGP has no concept of route reflectors. So some checks will always reside in the code, and I don't think we should try to make our XML Turing-complete.

dmbaturin · 2024-07-04T12:26:04Z

.gitignore

@@ -149,3 +149,6 @@ python/vyos/xml_ref/pkg_cache/*_cache.py
 # We do not use pip
 Pipfile
 Pipfile.lock
+
+# KDE


This change might be sensible for KDE users (not for me — I use MATE, btw; but @jestabro does use KDE :).
But it's out of scope of this PR, could you make it separate please?

dmbaturin · 2024-07-04T12:27:12Z

.vscode/settings.json

@@ -1,26 +0,0 @@
-{


If this file doesn't prevent the changes from working (I believe it doesn't), its removal shouldn't be in this PR.

dmbaturin · 2024-07-04T12:28:05Z

interface-definitions/container.xml.in

+              <child>allow-host-networks</child>
+              <child>network</child>
+            </atLeastOneOf>
+            <oneWayDependantChildren>


The word "dependent" is misspelled here (see https://en.wiktionary.org/wiki/dependent)

I also wonder if it should be just "dependentChildren" rather than "oneWayDependentChildren".

dmbaturin · 2024-07-04T12:31:07Z

interface-definitions/interfaces_openvpn.xml.in

+            <oneWayDependantChildren>
+              <dependants><descendant name="mode"><value>client</value></descendant></dependants>
+              <dependees>
+                <child>remote_host</child>


Underscore, not a hyphen?

dmbaturin · 2024-07-04T12:33:00Z

interface-definitions/interfaces_openvpn.xml.in

+            <oneWayDependantChildren>
+              <dependants><descendant name="mode"><value>site-to-site</value></descendant></dependants>
+              <dependees>
+                <child>remote-host</child>


Site-to-site mode doesn't actually require remote-host. One side of a site-to-site tunnel is a listener.

dmbaturin · 2024-07-04T12:40:27Z

interface-definitions/interfaces_openvpn.xml.in

+              <descendant name="mode"><value>server</value></descendant>
+              <child>authentication</child>
+            </mutuallyExclusiveChildren>
+            <mutuallyExclusiveChildren>


This sequence of <mutuallyExclusiveChildren> tags is very tedious to read, to be honest.

I'm also thinking that <mutuallyExclusiveChildren> should only be for children that simply cannot appear together under any circumstances, like server and remote-host.

In a situation when a node value is incompatible with a bunch of children, we may want to call it something else.

At the very least I'd like a more compact syntax, like:

<conflict> <source> <descendant name="mode" value="server"> </source> <excludes>remote-host</excludes> <excludes>remote-port</excludes> ... </conflict>

github-actions · 2024-07-22T17:25:12Z

This pull request has conflicts, please resolve those before we can evaluate the pull request.

thomasfinstad · 2024-07-24T12:21:04Z

Note for passers by, the discussion is currently taking place in slack until we land on how to move forward.

github-actions · 2024-07-25T10:10:37Z

This pull request has conflicts, please resolve those before we can evaluate the pull request.

github-actions · 2024-12-04T00:24:52Z

This PR is stale because it has been open 30 days with no activity. The PR will be reviewed by a maintainer and may be closed

sever-sever · 2024-12-18T19:55:35Z

@thomasfinstad could you check requested changes and rebase the PR?

github-actions bot assigned thomasfinstad Jun 2, 2024

github-actions bot added the current label Jun 2, 2024

thomasfinstad force-pushed the T5083 branch from d3150c8 to 38a1913 Compare June 2, 2024 15:07

thomasfinstad mentioned this pull request Jun 2, 2024

vyconf: T5083: add childRequirement to reference tree vyos/vyos1x-config#29

Open

10 tasks

thomasfinstad force-pushed the T5083 branch from 38a1913 to 5995cfe Compare June 2, 2024 16:37

thomasfinstad marked this pull request as ready for review June 2, 2024 16:38

thomasfinstad requested a review from a team as a code owner June 2, 2024 16:38

thomasfinstad requested review from dmbaturin, sarthurdev, zdc, jestabro, sever-sever and c-po June 2, 2024 16:38

dmbaturin requested changes Jun 2, 2024

View reviewed changes

github-actions bot added the state: conflict label Jun 11, 2024

jestabro reviewed Jun 17, 2024

View reviewed changes

thomasfinstad force-pushed the T5083 branch from 5995cfe to 2f34d36 Compare June 19, 2024 22:03

github-actions bot removed the state: conflict label Jun 19, 2024

T5083: extend xml schema definitions to support child specifications

7ca203c

thomasfinstad force-pushed the T5083 branch from 2f34d36 to 7ca203c Compare June 20, 2024 07:21

andamasov requested review from jestabro and dmbaturin June 21, 2024 13:22

dmbaturin requested changes Jul 4, 2024

View reviewed changes

github-actions bot added the state: conflict label Jul 22, 2024

github-actions bot added the conflicts label Jul 25, 2024

sever-sever added the stale PR has become inactive or needs attention label Sep 4, 2024

github-actions bot added the rebase label Sep 4, 2024

github-actions bot added the state: stale label Dec 4, 2024

github-actions bot removed the state: stale label Dec 19, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

T5083: extend xml schema definitions to support child requirements #3575

T5083: extend xml schema definitions to support child requirements #3575

thomasfinstad commented Jun 2, 2024 •

edited

Loading

dmbaturin left a comment

dmbaturin Jun 2, 2024

thomasfinstad Jun 2, 2024

dmbaturin Jun 2, 2024

thomasfinstad Jun 2, 2024 •

edited

Loading

github-actions bot commented Jun 11, 2024

jestabro left a comment •

edited

Loading

thomasfinstad commented Jun 17, 2024

github-actions bot commented Jun 19, 2024

github-actions bot commented Jun 19, 2024 •

edited

Loading

thomasfinstad commented Jun 19, 2024 •

edited

Loading

thomasfinstad commented Jun 21, 2024

dmbaturin left a comment

dmbaturin Jul 4, 2024

dmbaturin Jul 4, 2024

dmbaturin Jul 4, 2024

dmbaturin Jul 4, 2024

dmbaturin Jul 4, 2024

dmbaturin Jul 4, 2024

github-actions bot commented Jul 22, 2024

thomasfinstad commented Jul 24, 2024

github-actions bot commented Jul 25, 2024

github-actions bot commented Dec 4, 2024

sever-sever commented Dec 18, 2024

T5083: extend xml schema definitions to support child requirements #3575

Are you sure you want to change the base?

T5083: extend xml schema definitions to support child requirements #3575

Conversation

thomasfinstad commented Jun 2, 2024 • edited Loading

Change Summary

Types of changes

Related Task(s)

Related PR(s)

Component(s) name

Proposed changes

How to test

Smoketest result

Checklist:

dmbaturin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thomasfinstad Jun 2, 2024 • edited Loading

Choose a reason for hiding this comment

github-actions bot commented Jun 11, 2024

jestabro left a comment • edited Loading

Choose a reason for hiding this comment

thomasfinstad commented Jun 17, 2024

github-actions bot commented Jun 19, 2024

github-actions bot commented Jun 19, 2024 • edited Loading

thomasfinstad commented Jun 19, 2024 • edited Loading

thomasfinstad commented Jun 21, 2024

dmbaturin left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Jul 22, 2024

thomasfinstad commented Jul 24, 2024

github-actions bot commented Jul 25, 2024

github-actions bot commented Dec 4, 2024

sever-sever commented Dec 18, 2024

thomasfinstad commented Jun 2, 2024 •

edited

Loading

thomasfinstad Jun 2, 2024 •

edited

Loading

jestabro left a comment •

edited

Loading

github-actions bot commented Jun 19, 2024 •

edited

Loading

thomasfinstad commented Jun 19, 2024 •

edited

Loading