[epic] Distinguish between reported and desired state in twins

[thjaeckle]

Problem statement

A feature which was long on the roadmap of Eclipse Ditto (see also former GH issue #125 where previous discussions were done) is the ability to distinguish between reported and desired twin state.
reported twin state can be seen as data/state coming from the actual device (the current "truth") whereas the desiredProperties state is something an application in the backend or a mobile app would set as the new requested target state for a property.

Example

For example, take a room thermostat:

• the actual device reports that the measured-temperature (reported) is 20°
• the currently configured target-temperature (reported) is also 20°
• the user feels chilly, so via a mobile app the (desired) target-temperature is set to 22°
• the actual device receives this desired target-temperature as change notification and sets its reported target-temperature also to 22
• when this change was done and reported = desired for the target-temperature, the desired target-temperature is removed from the twin
• the measured-temperature in this moment is still 20, and the device will now start heating up the room until the measured-temperature reaches the (reported) target-temperature
• now if the device was offline during the desired target-temperature was configured, upon reconnection it can ask its twin "give me a delta for the desired state to apply" and can apply those missed changes

Sequence diagram

The following sequence diagram shall illustrate the interaction pattern:

Show sequencediagram.org source of sequence diagram

title Eclipse Ditto reported/desired feature state

participant Mobile App
participant Ditto
participant Device

==Device has connection==

Mobile App->Ditto:set "desired" property\nThermo/desiredProperties/configuration/targetTemp
activate Mobile App
activate Ditto
activate Device
Ditto->Ditto:update twin\nThermo/desiredProperties/configuration/targetTemp
Mobile App<-Ditto:response
deactivate Mobile App

Ditto->Device:send modified event regarding\nThermo/desiredProperties/configuration/targetTemp
deactivate Ditto

Device->Device:apply config change

Device->Ditto:confirm by setting actual property\nin "properties" with same name:\nThermo/properties/configuration/targetTemp
activate Ditto

Ditto->Ditto:update twin\nThermo/properties/configuration/targetTemp
Mobile App<--Ditto:send modified event regarding\nThermo/properties/configuration/targetTemp

Ditto->Ditto:remove twin\nThermo/desiredProperties/configuration/targetTemp
Mobile App<--Ditto:send modified event regarding\nThermo/desiredProperties/configuration/targetTemp

deactivate Ditto
deactivate Device

==Device looses connection==

Mobile App->Ditto:set "desiredProperties" property\nThermo/desiredProperties/configuration/targetTemp
activate Mobile App
activate Ditto
Ditto->Ditto:update twin\nThermo/desiredProperties/configuration/targetTemp
Mobile App<-Ditto:response
deactivate Mobile App

note right of Ditto:ineffective modified event
deactivate Ditto

==on Device re-connect==

Device->Ditto:request outstanding desired changes
activate Ditto
activate Device
Ditto->Device:send back all outstanding (non-deleted) \ndesired properties
deactivate Ditto

Device->Device:apply delta\nchanges to\ndesired
Device->Ditto:confirm by setting adjusted actual property\nin "properties" with same name:\nThermo/properties/configuration/targetTemp
activate Ditto

Ditto->Ditto:update twin\nThermo/properties/configuration/targetTemp
Mobile App<--Ditto:send modified event regarding\nThermo/properties/configuration/targetTemp

Ditto->Ditto:remove twin\nThermo/desiredProperties/configuration/targetTemp
Mobile App<--Ditto:send modified event regarding\nThermo/desiredProperties/configuration/targetTemp

deactivate Ditto
deactivate Device

====
note over Mobile App,Device:Sequence diagram created using the awesome online tool http://sequencediagram.org/

Benefits

Currently, Eclipse Ditto foresees features of a thing (aka the digital twin) to contain data/state from real devices in the properties of the features.
It is however not defined if those properties are reported or desired state. Up to now this had to be defined by the application developer modeling the data
to store in Ditto.

By defining in Ditto what is reported and what desired property state, Ditto can provide several benefits for application developers:

• at any time (e.g. when a device is not connected) the reported and desired properties are clearly distinguishable in the twin
• devices which are offline during changing desiredProperties can retrieve all desired state of a digital twin, e.g. on (re-)connection of the device
• delta changes can be calculated by Ditto in order to get a list of changes to be done for an out-of-sync-device to reach its configured desired state
• the definition (the information about the contained properties and their types) of a feature applies for both, reported and desired state
• automatic cleanup/deletion of desiredProperties state by Ditto when reported state eventually reaches the desired state
• policies may be applied based on reported/desired, so e.g. a device may be prohibited to change the desiredProperties state whereas an end user may be prohibited from changing the reported state
• search capabilities based on the desired state, e.g. finding all twins where the desired state is not yet met
• automatic delta calculation and publishing to the device upon device connection
• maintaining order (modification timestamp) of desired/reported changes in order to not apply a desired change when a change done to the actual device was done at a later point in time

This GitHub issue shall act as the overall "epic" for all issues regarding the "desiredProperties" feature addition.

EPIC issues

Phase 1

• Enhance model and APIs with "desiredProperties" for Features #697: Enhance model and APIs with "desiredProperties" for Features
• Support wildcard in "fields" selectors in order to receive all "desired" properties of all features of a thing #700: Support wildcard in "fields" selectors in order to receive all "desired" properties of all features of a thing
• Delete value of a "desired" feature property once the actual "property" was set to the same value #698: Delete value of a "desiredProperties" feature property once the actual "property" was set to the same value
• Convenient creation of parent objects for modify commands with nested JSON paths #701: Convenient creation of parent objects for modify commands with nested JSON paths

Phase 1+x

• Support for using wildcard in place of feature name in policies #708: Support for using wildcard in place of feature name in policies
• Search for things/features with pending "desired" properties #702: Search for things/features with pending "desiredProperties"
• Provide information about desired property changes which could not be applied #704: Provide information about desired property changes which could not be applied
• Don't apply desired property if "applied" property was changed by device more recently #706: Don't apply desired property if "applied" property was changed by device more recently
• Automatically publish remaining desired state upon device connection #703: Automatically publish remaining desired state upon device connection
• Emit a special event once a single and/or all "desired" properties were cleared #705: Emit a special event once a single and/or all "desiredProperties" were cleared
• Provide a command + APIs in order to calculate+retrieve deltas of desired vs. properties #699: Provide a command + APIs in order to calculate+retrieve deltas of desiredProperties vs. properties

Please feel free to start discussions about the overall concept in this issue and regarding the more specific topics in the linked issues.

[jufickel-b]

At least one corner case came to my mind: what happens if a property with a desired state gets deleted or renamed?

Deletion, probably, is no problem because the desired state gets deleted as well.

However when a property is renamed will its desired property renamed, too? If so, how does Ditto maintain the association of the former name to the new name for received state changes?

[thjaeckle]

At least one corner case came to my mind: what happens if a property with a desired state gets deleted or renamed?

I think that this is a cornercase which can only happen during development - once the model/structure of a feature is defined by a fixed "definition", there is no renaming of properties.

When updating the major version of a definition (according to semantic versioning this would be the only way to rename a field), there would have to be a migration done anyways, so that migration must migrate properties as well as desired fields the same way.