Distinguish between "desired" and "twin" Feature properties

[thjaeckle]

In order to add another important Digital Twin feature to Ditto it requires to distinguish between different state perspectives.

We already have the "twin" and "live" perspective being an important differentiation between "cached state" and "actual live state".
However if a user wants to configure on a Digital Twin the desired state (e.g. "I want that this light is switched on") no matter if it is currently receiving the desired change or not, Ditto needs a place to store that.

I would suggest that

• the "twin" perspective would always track the "reported" state of the device
• the "live" perspective stays as it is: by using "live", a device is directly addressed
  • if it cannot receive the command (e.g. in order to retrieve the "on" state), the user of the API gets a timeout.
• we add a new perspective "desired" (name to be discussed) which Ditto saves separately

I would also like to propose to have a "common API" which handles the "twin/live/desired" under the hood.
E.g. that a user may trigger API calls in which she defines:

• give me the current value of the "on" property (first trying via "live", falling back to "twin" after a timeout when live doesn't answer in time)
• change the current value of the "on" property to true (first trying via "live", falling back to "desired" after a timeout when live doesn't answer in time)

A device may retrieve the state is has still to apply in order to be "in sync" with the Digital Twin.
E.g.:

• device comes online and knows it has state revision 42
• devices asks Ditto: "hey, I am on revision 42, give me the delta of "desired" property changes I missed)
• Ditto answers with a diff/delta

What is your opinion on that?

[geglock]

I played around with our existing concepts and scribbled a sequence diagram that shows a potential approach without yet extending the core of Eclipse Ditto but by accompanying it with a separate microservice that helps on the synchronization, i.e. creating a "patch" between desired and reported state.

This approach is not based on revision numbers but on a second "copy" of state - called "Feature@desired" for each "Feature" of a Thing. This second Feature is is also fully managed within Ditto and also usable by applications.

Devices/gateway have to do some "co-operation":

• confirm changes when they are applied
• request a "patch" on re-connects (or periodically)

See my proposal:

[geglock]

For use cases with more capable devices and not extremely limiting bandwidths to the devices the described approach above can be simplified without the need of a "patch" functionality:

Use two Features for each element that has to distinguish between "reported" and "desired". Name the first one just "featureX" and the second one "featureX@desired".
Let the application use both to manage these perspectives.
This is exactly as described above.

On device side, implement a re-connect mechanism that retrieves the featureX@desired via a respective Ditto protocol "retrieve" command (see https://www.eclipse.org/ditto/protocol-specification-things-retrieve.html#retrieve-all-properties-of-a-feature) . This is needed to catch up with eventually missed updates.
After retrieving the result the device should check which configurations have to be applied by comparing them to it's known state. After successful applying, send respective "modify" commands on the featureX (without @desired). Optionally the device could in addition to that also send "delete" commands to remove properties of featureX@desired to indicate that there is no difference between reported and desired.

The discussed issue here would then be an optimization that tries to save bandwidth and potentially also automates the process of sending the change to the device on each re-connect. But you could use this approach already for now without waiting for this issue.

[sophokles73]

Based on the concepts implemented in Hono, it seems desirable for Ditto to be able to react to telemetry/event messages from devices that contain a time-till-disconnect property which is used in Hono to indicate that a device is online and willing to receive commands. Ditto could use this trigger to send pending patch/update command(s) to the device via Hono. If Ditto uses a request/response command for that purpose, it would also get notified about the outcome of processing the command on the device.

[thjaeckle]

We added an issue #492 in order to being able to reflect the Hono ttd information in Ditto's digital twins.
Sending patch/update commands based on desired state (the feature this issue is about) would be another step (the holy grail? ;)) which I think will be quite challenging as we would have to define means in order to find out when exactly and for which devices to send what information upon connection.

With #492, IoT applications however can react on Hono's online indications and send a command down to the device themselves.

[BobClaerhout]

As stated during the conf call last time, we are very much interested in this feature. @thjaeckle asked me to write down my thoughts about this. Here we go:

• We will be using this to send messages to the devices coming from the application as documented in the comment from @geglock.
• Our devices send and receive binary payload messages. This implies some sort of translator is in between hono and ditto (Map Hono device connection status to Thing feature #492). We are looking into Vorto for this since our binary payload is quite complicated. I don't know whether this is useful information here, but it somewhat describes our setup.
• What does this "request patch" look like if our device is sending binary payload?
• Will there be some mechanism to determine which feature@desired will result in a downlink and which feature does not result in a downlink? Some clarification: in some cases the feature has more of a functional meaning. Say you have temperature sensor somewhere which reports the temperature in a feature. The application decides the temperature should change but of course the sensor is not capable to do that. Therefore this should not result in a downlink command but a warning/notification in the business application. See next point for possible solution to this.
• This example shows how the ditto object can use a vorto model in it's definition. How do these definitions relate to generating downlinks? Based on the (non-)existence of the configuration and/or the operation functionblock, we can hold back the downlink command. However, if we are working with a separate feature (@Desired), how does this work together? Should we have multiple definitions? Would be nice if both projects could integrate on this and have the possibility to co-exist without a hard requirement for each other.
• Can we already agree on the used convention? Right now we have a use case where we already want to set the desired state and compare them in the business application. Would be nice to have the correct naming convention already. However, this greatly depends on the previous point as well.

[thjaeckle]

Thanks @BobClaerhout for your input.
I think we should split up this (rather very large) issue and see the following sub-parts.

sub parts

Part 1: Persisting "desired" state and providing API for that

• enhance the ThingPersistenceActor by - in addition to twin - persisting desired feature properties
• provide API (e.g. HTTP, but also Ditto Protocol) in order to manage/access the desired view
  • could e.g. in case of HTTP be a query param desired
• no "magic" is applied anywhere, this is just an alternative view on a thing
  • no patch/delta calculation
  • no automatic "downlink" message containing delta upon device connection

Part 2: requesting a diff from twin/desired view

• add and handle command(s) (and an HTTP API) requesting a diff
  • use case: to be invoked by a device when it connects in order to retrieve its "missed" desired changes to apply
  • this would be a Ditto Protocol message, this has nothing to do with binary payload, this is plain JSON which could be e.g. transformed by Ditto's JavaScript based payload mapping (@BobClaerhout, you asked that ;))
• add and handle commands which "resets" the desired view for when the reported (twin) was updated

tbd:

• it does not make sense to create a diff on outdated twin data, so I assume that the API would require the device to always either
  • send its twin state upon connection (at least the parts the device is interested in diffs for) - @BobClaerhout: that could be the "mechanism to determine" which desired feature would create a downlink delta message
  • send its last known revision number upon connection
    • downside on that is that Ditto would have to load all historic events between the requested last known revision number and the current desired -> Ditto currently does not keep that data but removes it

Part 3: automated downlink delta for connected devices

Currently I don't know if we even can or want support that to be honest - as mentioned in "Part 2" this does only make sense if the device says something about its currently known state .. and that way a automatic mechanism will never work out-of-the-box because it operates on outdated "reported" data of a connected device.

Can we already agree on the used convention?

Before we discuss that I would like to discuss and get the community's input on some alternatives we have.
Please also think about more positives and negatives for each solution, I just started with the ones which came into my mind.
Maybe someone also sees another alternative.

I will gladly enhance this list with community input.

Implementation alternatives for Part 1

@desired as additional feature

Like described already in this issue above.

{
  "features": {
    "lamp": {
      "definition": [ "org.eclipse.ditto:Lamp:1.0.0" ],
      "properties": {
        "on": true
      }
    },
    "lamp@desired": {
      "definition": [ "org.eclipse.ditto:Lamp:1.0.0" ],
      "properties": {
        "on": false
      }
    }
  }
}

+

• is currently possible by convention
• already existing features with @desired suffix could be supported when Ditto eventually adds this feature
• policy may be written in a way which defines different access rights for twin/desired features

-

• payload duplication (e.g. of the definition)
• feature and feature@desired would not even have to share the same definition -> big potential for mistakes
• when applying Vorto model for feature@desired the "required" would have to be ignored - as desired properties must be optionally settable,
  so if we strictly apply JsonSchema, another schema for the @desired feature would have to be generated setting all "required" fields to optional
• @desired features must be excluded when using the "twin" API and the other way around

@desired on property level

{
  "features": {
    "lamp": {
      "definition": [ "org.eclipse.ditto:Lamp:1.0.0" ],
      "properties": {
        "on": true,
        "on@desired": false
      }
    }
  }
}

+

• is currently possible by convention
• already existing feature properties with @desired suffix could be supported when Ditto eventually adds this feature
• it does not always make sense to define a complete feature as @Desired, only some properties
• defines fine grained which deltas should be sent downlink
• Ditto Vorto generator could be enhanced to generate <property-name>@desired with same type as <property-name> and always optional
• policy may be written in a way which defines different access rights for twin/desired properties
• almost no payload duplication

-

• @desired properties must be excluded when using the "twin" API and the other way around

Impl. ideas:

• We could add a special Predicate<JsonField> (similar to FieldType) for all properties ending with @desired
  • that way the toJson() of a Thing could use this in order to either
    • only include non-@desired tagged properties
    • only include @desired tagged properties
    • include all, @desired and non-@desired tagged properties
  • that could be used at HTTP API level to select the channel e.g. for GETs

desired not visible in thing JSON

This approach would basically save an additional desired thing in the persistence.

Twin:

{
  "features": {
    "lamp": {
      "definition": [ "org.eclipse.ditto:Lamp:1.0.0" ],
      "properties": {
        "on": true
      }
    }
  }
}

Desired Twin:

{
  "features": {
    "lamp": {
      "definition": [ "org.eclipse.ditto:Lamp:1.0.0" ],
      "properties": {
        "on": false
      }
    }
  }
}

+

• internally handled what is the "twin" and what "desired twin"
• "twin" and "desired twin" could potentially even have different _modified timestamps and _revision counters
• Ditto Vorto generator would also have to treat the properties of the "desired twin" as "optional"

-

• is currently NOT possible by convention
• massive duplication (e.g. of the definition)

[w4tsn]

@thjaeckle I myself like the approach "@desired on property level" the most out of the proposed solutions. With a write permission on the actual property and a read permission on the @desired property there is no circularity. On low bandwith devices it is important to not receive another message when the actual state is set and only receive a message if the desired state is set.

We also observe a difference in sending desired states over live or twin channel. Let me give you my user story to clarify this.

A user (through a frontend) or microservice wants to influence the device. It may do this through the digital twin or the live channel. There are (at least) two cases I can think of right now:

1. The state should be applied immediately and is only valid for the current actual state (e.g. user invokes it directly through a UI or microservice computes the desired state from current state)
2. The state may be applied at any time and has no dependencies
3. The state may be applied at any time, but has dependencies (to complex right now to follow from my point of view)

We solve the first case right now by using live messages, so we have a command/response API and are able to leverage timeouts and do the computation based on the methods (which might not refer to a single property per sé). Another solution would be to set properties over the live channel with a more limited API also resulting in a loss of the desired state if the device is offline (which is desired in this case).

In the second case we want to set a desired state for properties, which will result in a success for this "correlation", but may fail when the state is applied. We then would have to emit an event message or something similar. I first thought, that another channel would be good to filter out only @Desired messages to reduce bandwith, but this could also be achieved via policies. A device may only have read on it's desired state properties but not on the actual ones having only write permissions on them.

I think we can further make a distinction between desired state of configuration or status. A configuration@desired may be set directly and have or have no side effects. If a status is set however, this implies some more complex computation (maybe also configuration changes invoked by the device) to meet the desired status state. I'm just pointing this out right now, but I don't have any more ideas or thoughts on this at the moment. It's just fascinating :)

Other Ideas

• the @Desired property could hold an object of the form { value: any, _modified: date } to get a bit more context of single properties

I really don't like the duplication of the last approach and it's error potentials around e.g. 'definition' and the vorto model.

[nicodeceulaer]

@thjaeckle When using vorto to define the twin schema, there seems to already be a logical split up between configuration/status/events/operations.
Following that logic, it makes sense to only apply the desired twin split for the configuration block:

{
    "definition": "com.bosch.iot.suite.examples.digitaltwin:DigitaltwinExample:1.0.0",
    "attributes": {
        "modelDisplayName": "DigitaltwinExample"
    },
    "features": {
        "Device": {
            "definition": [
                "com.bosch.iot.suite.examples.digitaltwin:D100:1.0.0"
            ],
            "properties": {
                "status": {
                    "temperature": 0.0
                },
                "configuration": {
                    "threshold": 0.0
                },
                "configuration@desired": {
                    "threshold": 0.0
                }
            }
        }
    }
}

But maybe I'm missing something in my first steps in the world of Vorto ?

[thjaeckle]

@nicodeceulaer that would be correct if Ditto would always rely on Vorto's models. But that's not the case (a definition is completely optional and may also reference another type model), so we can't assume that there is a configuration section at all.

I would like to completely leave the semantics of what is a property useful to have a desired state to the data modeler and applications using Ditto.

[w4tsn]

It just came to my mind, that we could use approach 1 @desired feature together with approach 3 in the way that ditto could handle the desired twin internally, but the interface to it is done via @Desired. This would mean that it can be implemented with current methods but the underlaying data-structure could then allow additional things that are only possible, if the desired state is kept in it's own entity. @thjaeckle what do you think?

[thjaeckle]

@w4tsn that sounds for me like choosing the worst stuff from approach 1 (problems with Vorto defintion, etc.) and 3 (much effort, duplication) :D
Meanwhile, personally, I would go with approach 2, if we would start with that issue. This however seems to be not around the corner, to be honest.

What could also be another approach, which might be interesting for solving other problems as well:

Managing metadata for scalar JSON fields

Idea: add another "_meta" field which copies the JSON structure, but uses a Json object on the JSON leaves where additional meta information like modification timestamp, revision and desired state are located:

{
  "thingId": "org.eclipse.ditto:foo1",
  "policyId": "org.eclipse.ditto:foo1",
  "features": {
    "lamp": {
      "definition": [ "org.eclipse.ditto:Lamp:1.0.0" ],
      "properties": {
        "on": true
      }
    }
  },
  "_meta": {
    "features": {
      "lamp": {
        "definition": {
          "lastModified": "...",
          "revision": 42
         },
        "properties": {
          "on": {
            "lastModified": "...",
            "revision": 50,
            "desired": false
          }
        }
      }
    }
}

Just another idea which might come in very useful for other topics (e.g. updates based on the modification timestamp of a single value) as well ..

[thjaeckle]

We'll close this longly discussion about different formats/requirements, etc. and start over with #696 in order to track the overall concept.
This feature addition is an integral part of a digital twin framework and will be quite some work, so this is just to have a better management of the topic.

Issue #697 shall be the one I mentioned above being the Part 1: Persisting "desired" state and providing API for that.
We had another great idea for the format and we can of course continue our discussion there.

My last comment was already moved to another issue: #680 in which adding additional metadata for feature properties is tracked.