data-point-service

DataPoint Factory

Creates a DataPoint instance with redis support

Requirements

• Node 8 LTS (or higher)
• Redis (Optional for development)

Install

$ npm install data-point-cache

factory.create

Create a new DataPoint Service Object

factory.create(options):Promise<Service>

Properties of the options argument:

option	type	description
cache	Object	cache specific settings
cache.isRequired	Boolean	false by default, if true it will throw an error
cache.redis	Object	redis settings passed to the ioredis constructor. For key prefixing, set keyPrefix through cache.redis.keyPrefix; if the value is not provided then os.hostname() will be used.

This method returns a Promise that resolves to a Service Object.

The Service Object holds a reference to a dataPoint instance.

const options = {
  entities: {
    "reducer:foo": (input, acc) => "bar"
  }
};
factory
  .create(options)
  .then(service => {
    return service.dataPoint.transform("reducer:foo");
  })
  .then(output => {
    console.log(output);
    // bar
  });

Express example

const express = require("express");
const DataPoint = require("data-point");
const DataPointService = require("data-point-service");

function server(dataPoint) {
  const app = express();

  app.get("/api/hello-world", (req, res) => {
    dataPoint.resolve(`entry:HelloWorld`, req.query).then(output => {
      res.send(output);
    });
  });

  app.listen(3000, function() {
    console.log("listening on port 3000!");
  });
}

function createService() {
  return DataPointService.create({
    DataPoint,
    entities: {
      "reducer:HelloWorld": (input, acc) => "Hello World!!"
    }
  }).then(service => {
    return service.dataPoint;
  });
}

createService().then(server);

Configure entity's cache settings

To configure an entity's cache settings you must set cache configuration through the params object.

'<type>:<entity-name>': {
  params: {
    cache: {
      ttl: String|Number,
      staleWhileRevalidate: String|Number|Boolean,
      revalidateTimeout: String|Number
    }
  }
}

cache.ttl

ttl: String | Number;

Use cache.ttl to set an entity's cache entry Time To Live value. When this value is a string it is expected to be written using the format supported by ms.

Example:

Creates a request entity that when called will append an entry to redis with a TTL of 20 minutes. When the entry expires, the next request that comes through will have to re-fetch again to respond, upon success a new cache entry with same TTL will be created.

DataPointService.create({
  DataPoint,
  entities: {
    'request:getPlanets': {
      url: 'https://swapi.co/api/planets/'
      params: {
        cache: {
          ttl: '20m', // 20 minutes
        }
      }
    }
  }
})

cache.staleWhileRevalidate

staleWhileRevalidate: String|Number|Boolean (defaults to `5s` if set to `true`)

staleWhileRevalidate = time to serve stale while revalidating

staleWhileRevalidate value is expected to be written as a string following the format supported by ms. Alternately it may also be set to true, which tells the entity to use double the time of the value of its ttl.

Use cache.staleWhileRevalidate in conjunction with a valid cache.ttl to use the Stale While Revalidate cache pattern. When present, caches MAY serve the response in which it appears after it becomes stale, up to the indicated ttl. The cache SHOULD attempt to revalidate it asynchronously while still serving stale responses. If staleWhileRevalidate time has passed without the cached entity being revalidated, it SHOULD NOT continue to be served stale.

Example:

Creates a request entity that when called will append an entry to redis with no expiration (stale), at the same time it will create a controller cache entry that has the ttl value. On every request to this entity the stale value will be returned, when the control entry expires, the stale value will continue to be returned and a background process will be triggered to execute the entity and update the stale entry.

DataPointService.create({
  DataPoint,
  entities: {
    'request:getPlanets': {
      url: 'https://swapi.co/api/planets/'
      params: {
        cache: {
          ttl: '20m', // 20 minutes
          staleWhileRevalidate: true
        }
      }
    }
  }
})

cache.revalidateTimeout

revalidateTimeout: String | Number;

defaults to: 5s

revalidateTimeout is the time a revalidation process has before it times-out, the value is expected to be written as a string (eg. '20m') following the format supported by ms. When revalidation starts a revalidation flag is set which blocks revalidation duplicates from happening. Once the revalidation times-out the revalidation flag will be removed and the key will be unblocked for being revalidated again. If omitted it defaults to 5 seconds.

This value is only used when staleWhileRevalidate is also set.

Revalidation and concurrency

When an entity is requested for the first time it will be considered a cold lookup, once the entity is resolved a cache entry will be saved on redis. For the key's life set by the entity's ttl (and staleWhileRevalidate) the entity will return the value (considered stale) from the redis store. Once a new request is made for the key and it's ttl has expired a revalidation will be triggered on the background; when a revalidation starts a revalidation flag will be added to prevent new calls of duplicating the revalidation process.

Revalidation flags are saved locally (in memory - they get cleared once their timeout expires) and also added to the redis store to be accessed by multiple node instances sharing the same keys. The local flag is to prevent duplicate revalidation by a single instance, because it is in memory it is an instant lookup; the remote flag (redis) is to prevent multiple instances of attempting to revalidate the same request.

In the case the revalidation fails the flag will be removed to allow a new revalidation to be triggered.

It is important to know that once the staleWhileRevalidate delta has also expired a background revalidation will no longer be triggered and cold lookup will be triggered instead. For this reason it is important to carefully set the cache settings to have a solid caching strategy.

Validation

If an entity's outputType reducer is set, revalidation will execute it before storing the new entry in cache. If the validation fails the entry will not be stored.

In the case revalidation happens via stale-while-revalidate and outputType does not pass then stale value will be served until the stale value expires on its own.

Contributing

Please read CONTRIBUTING.md for details on our code of conduct, and the process for submitting pull requests to us.

License

This project is licensed under the Apache License Version 2.0 - see the LICENSE file for details