Happenstance is a specification and sample implementation for a federated, decentralized status network. The features of this network are strong and discoverable identity, status update feeds, update publish & subscribe, mentions, re-post and private messaging.
The network is defined as a number of interoperating systems, that together form the network, but which can be implemented independently. All communication in the network is done over HTTP using JSON as the message payload.
http://www.claassen.net/geek/blog/category/happenstance
As I am writing the sample implementations of the components and getting feedback, I am of course finding holes and oversights. I am striving to not let those findings detract me from keeping it simple and small. Until I have a working set of samples, I will keep this document in its informal state and write up a more traditional spec for each component at that point.
Before going into details of how the network is constructed, here's a 10k ft. view of the messaging workflow for status updates.
- Sending user creates new feed entry in Status Feed system
- Status Feed pushes entry to PubSub
- PubSub pushes entry to recipients
- entry is pushed to all its subscribers, generally Aggregation servers
- Identity Entities in entries ( mentions ) are resolved via the appropriate Identity server to locate the resposible Aggregation servers, and pushed to those servers
- Aggregation combines and de-dupes entries and presents them as a status timeline to the recipient user
The Identity server provides the status feed lookup for {user}@{host} identities. It MUST provide a single API endpoint for looking up users, although there is no prohibition against alternative lookup endpoints and methods.
GET:/who/joe
{
_sig: {
name: 'key2',
sig: 'xfZ4DmrcLbz8qPJoTwYg/wIqggIKBBtzqnaiUu1Wess82wKdge+UsQEqU1hY2/0OrzgtUnzgn8nSWWPJtd6qtKbOTkPQqYDf2uVk6WTHYjwpysHmMj8fzrMkpE0ZPkPD8N7kEn1Rmt85CeXMDjYDN14H3Ep4iRNc7qxeNSR7xH8='
},
id: '[email protected]',
name: 'Joe Smith',
status_uri: 'http://droogindustries.com/status',
feed_uri: 'http:/droogindustries.com/status/feed'
}The above specifies the minimum required information, although the entire feed metadata section may be served by this call.
A domain's status nameserver is expected to live at status.{host}, unless a DNS SRV record exists for the {host}:
_happenstance-ns._tcp.{host}. TTL IN SRV 5 0 {port} {status host}
The canonical location of the status is always an HTML document including a link element identifying the status feed document. This abstraction serves two purposes:
- The canonical location can stay the same while the feed management provider can be changed independently
- The HTML document should serve as a human readable rendering of the feed document
The link takes the form of:
<link rel="alternate" type="application/hsf+json" href="http://droogindustries.com/status/feed"/>The document itself contains two sets of information, the feed meta data and a subset of the feed entries.
{
author: {metadata},
entries: [{entry}, ...]
subscriptions: [
{
id: '[email protected]',
name: 'Bob Jones',
profile_image.uri: 'http://droogindustries.com/bob.jpg',
status_uri: 'http://droogindustries.com/bob/status',
feed_uri: 'http://droogindustries.com/bob/status/feed',
},
...
]
}Subscriptions is an optional way to display what users the feed owner follows, but is not required. Generally subscriptions are handled by aggregation servers and do not have to be public.
For paging, two additional keys that may exist in the feed are previous_uri and next_uri. These keys allow callers to page through the feed without needing to know the implementation of the feed's API.
The minimum requirement for the meta data is defined below. Additional information can be added as desired by implemented, although it is generally good form to use a single key and stuff the desired content into a sub-object.
{
_sig: {
name: 'key2',
sig: 'xfZ4DmrcLbz8qPJoTwYg/wIqggIKBBtzqnaiUu1Wess82wKdge+UsQEqU1hY2/0OrzgtUnzgn8nSWWPJtd6qtKbOTkPQqYDf2uVk6WTHYjwpysHmMj8fzrMkpE0ZPkPD8N7kEn1Rmt85CeXMDjYDN14H3Ep4iRNc7qxeNSR7xH8='
},
id: '[email protected]',
name: 'Joe Smith',
profile_image_uri: 'http://droogindustries.com/joe.jpg',
status_uri: 'http://droogindustries.com/bob/status',
feed_uri: 'http:/droogindustries.com/bob/status/feed',
public_keys: {
key2: {
key: '-----BEGIN RSA PRIVATE KEY----- ...'
},
key1: {
key: '-----BEGIN RSA PRIVATE KEY----- ...',
expired: '2012-07-30T11:31:00Z'
}
},
altnerate_ids: [ '[email protected]' ],
publishers: ['http://publish.droogindustries.com/publish/RPqdvmtOHmEPbJ+kX'],
aggregators: ['http://aggro.droogindustries.com/aggro/AgMBAAEwDQYJKoZIhvcNA']
}Entries are the status updates. They should be considered write-only, since once published, copies will exist in many downstream data-stores. There is a mechanism for advisory updates and deletes using updates_id and deletes_id keys, but it is up to the downstream implementer to determine whether those entries are respected, used to create revision history or applied permanently. The minimum set (except for deletes_id entries) is:
{
_sig: {
name: 'key2',
sig: 'xfZ4DmrcLbz8qPJoTwYg/wIqggIKBBtzqnaiUu1Wess82wKdge+UsQEqU1hY2/0OrzgtUnzgn8nSWWPJtd6qtKbOTkPQqYDf2uVk6WTHYjwpysHmMj8fzrMkpE0ZPkPD8N7kEn1Rmt85CeXMDjYDN14H3Ep4iRNc7qxeNSR7xH8='
},
id: '4AQlP4lP0xGaDAMF6CwzAQ'
href: 'http://droogindustries.com/joe/status/feed/4AQlP4lP0xGaDAMF6CwzAQ',
created_at: '2012-07-30T11:31:00Z',
author: {
id: '[email protected]',
name: 'Joe Smith',
profile_image.uri: 'http://droogindustries.com/joe.jpg',
status_uri: 'http://droogindustries.com/joe/status',
feed_uri: 'http://droogindustries.com/joe/status/feed',
},
text: 'Hey {{bob}}, current status {{beach}} {{vacation}}',
entities: {
beach: 'http://droogindustries.com/images/beach.jpg',
bob: '[email protected]',
vacation '#vacation'
}
}For re-posting someone else's status update, a repost_href key containing the original posts href can be included. In addition, it is recommended to also include the full entry under the _repost key (omitting it from signature requirements). The text of the status update can be used to add additional commentary.
Additional optional fields are specified in the separate message spec, as well as details on options for entities.
While there is no way to enforce content size, since the content will replicated across the network and is meant as a status network not forum, the text field is assumed to be limited to 1000 characters and implementers are free to drop of truncate (which invalidates the signature) long messages.
While the feed is designed that it can be represented by static files, the authoring tool responsible for creating the status content bears the additional responsibility of pushing messages to the PubSub servers listed in author.publishers as described in the PubSub section.
Aggregation server collect status updates for users to create their timeline. In general, aggregation servers will receive events from PubSub servers, but the same expected API is also used to push mentions and private messages to a user.
The only required API for an aggregation server is rooted at any uri provided in the author.aggregators list in a user's feed. This endpoint is expected to accept POST messages with a body of { entries: [{entry}, ...] }.
Since this endpoint will likely be the target of spammers, implementers are advised to include some defensive mechanisms, such as whitelisting senders to subscribed feeds only or checking the entries for a valid source feed and signature, etc.
The PubSub server is responsible to publishing status updates to all subscribers. It defines two REST APIs, one for creating and managing publication resources and one for creating and managing subscription resources per publication resource. Only the subscription API is required to be implemented, i.e. a pubsub server could be an integral part of the feed content management system and have no publicly exposed API for creating publications.
The responsibility of PubSub is twofold:
- deliver the entries posted to publication endpoints to all its subscribers, and
- deliver the mentions enumerated in the entities object.
The latter involves looking up the feed for a name entity via the nameserver and then post the mentioned entries to the aggregators listed in author.aggregators.
The subscription REST API is always rooted at the uris provided by the author.publishers list in the feed and. A subscription is created via a POST to a publisher uri.
POST:
{
callback_uri: 'http://aggro.droogindustries.com/aggro/AgMBAAEwDQYJKoZIhvcNA',
status_uri: 'http://droogindustries.com/joe/status',
feed_uri: 'http://droogindustries.com/joe/status/feed',
headers: {
token: 'wNTI1MDIwODUxWjAjMQs'
}
}Response:201 Created
{
subscription_uri: 'http://publish.droogindustries.com/publish/RPqdvmtOHmEPbJ+kX/subscription/wDQYJKoZIhvcNAQEBBQADgY0AM',
access_key: 'Tj2DzR4EEXgEKpIvo8VBs/3+sHLF3ESgAhAgM'
}The response also contains the subscription_uri in the Location header. The access_key must be provided in modification requests as the Access header.
status_uri and feed_uri are optional, but highly encouraged if the subscription is on behalf of a user rather another subsystem, such as an indexing service. Subscriber count and subscribing feeds is reported back to the publisher and used to generate follower lists.
The optional headers can be any key value pairs and will be included in each POST to the callback_uri.
The current subscription resource can be fetched with a GET against the subscription_uri:
GET:{subscription_uri}
The subscription can be modified via PUT to the subscription uri with the Access header using the same request message format as the POST.
PUT:{subscription_uri}
The subscription can be deleted via DELETE to the subscription uri with the Access header.
DELETE:{subscrition_uri}
The callback uri is expected to accept POST requests with a body of { entries: [{entry}, ...] }. The PubSub implementation may choose to deliver each entry in this body, accumulate multiple from a feed or even multiple from different feeds as long as the callback uri and headers match another subscription.
PubSub may also implement an API for setting up subscription endpoints that the feed owner uses to set up the subscription. The base uri for this API is up to the implemtation -- there exists no automated discovery mechanism at this time.
POST:
javascript
{
_sig: '80669bd0d0bc39a062f87107de126293d85347775152328bf464908430712856',
id: '[email protected]',
name: 'Joe Smith',
status_uri: 'http://droogindustries.com/bob/status',
}
_Response:**201 Created**_
```javascript
{
publication_uri: 'http://publish.droogindustries.com/publish/RPqdvmtOHmEPbJ+kX',
access_key: 'Tj2DzR4EEXgEKpIvo8VBs/3+sHLF3ESgAhAgM'
}
The response also contains the publication_uri in the Location header. The access_key must be provided in modification requests as well as entry publication as the Access header.
The optional headers can be any key value pairs and will be included in each POST to the callback_uri.
The current subscription resource can be fetched with a GET against the publication_uri:
GET:{publication_uri}
Information about the subscribers can be retrieved with a GET:
GET:{publication_uri}/subscribers
{
subscription_count: 123,
subscribing_feeds: [
{
status_uri: 'http://droogindustries.com/joe/status',
feed_uri: 'http://droogindustries.com/joe/status/feed',
},
...
]
}Paging of subscribers can be handled as with feeds, using previous_uri and next_uri. Since this is not meant as a realtime query mechanism there are no page size and other options. It is meant merely for retrieval for storage at the publishers application. Paging exists as an option for PubSub implementers to manage payloads.
The publication can be modified via PUT to the publication uri with the Access header using the same request message format as the POST.
PUT:{publication_uri}
The publication can be deleted via DELETE to the publication uri with the Access header.
DELETE:{publication_uri}
POST:{publication_uri}
{
entries: [{entry}, ...]
}Discovery has no formal definition. It is comprised with organic discovery via reposts and browsing the subscriptions of the user you are subscribed out and discovery from various search indicies.
Given that anyone can get real-time feeds delivery and the feeds of anyone mentioned in an entry or can be discovered via the name-servers it is relatively simple to get access to the communications of the ecosystem for permanent or ephemeral, complete or partial indexing as a service.
Aggregators will likely want to integrate search capabilities greater than the data that passes through them, so they are the most likely customers of such services, but such integration will be custom and would not benefit for a formal definition at this stage of the ecosystem.
It should be noted that especially early in the ecosystem's life, it is highly desirable to have a central indexer of the global feed and user base.
See spec/Signing.md