Remote data sources

This section provides API Reference, examples of remote data sources and explains how to set up your own.

1. API Reference

Here, you will find information about the available options for configuring your remote data source. Below is a table outlining the parameters:

Name	Description	Type	Default
liveData	Enable or disable live data fetching	Boolean	False
remoteDataFetchPace	Specifies a periodic fetching pace in milliseconds	Number	1000
webWorkersHTTP	Enable or disable fetching data with web workers	Boolean	False

2. Integration

To integrate your custom remote data sources, follow these steps:

• Define a use case inside the *.stories.js file.
• Add the following use case arguments:
  • isRemoteDataSource: a mandatory option set to true, indicating that the data source is remote.
  • fetchFunction: an asynchronous custom user function that fetches the data, adapts it to Palindrome.js accepted data format, and returns a promise.

2.1. Integration example

Inside the root folder, inside *.stories.js:

export const remoteDataSourceExample = createPalindrome.bind({});
remoteDataSourceExample.args = {
   isRemoteDataSource: true,
   fetchFunction: myFetchFunction
};

// in webCollectors folder

In src/webCollectors, create a file that will contain myFetchFunction function:

export async function myFetchFunction() {
    try {
      const response = await fetch(myUrl);
      const data = await response.json();
      // adaptation logic
      return data;
    } catch (error) {
      console.error('Error fetching data:', error);
      throw error;
    }
}

2.2. Accepted format

{
    "layer_0": {
        "metrics": {
            "metric_0": {
                "label": string
                "unit": string
                "min": number
                "med": number
                "max": number
                "current": number
            },
            "metric_n": {
                "label": string
                "unit": string
                "min": number
                "med": number
                "max": number
                "current": number
            }
        },
        "layer": {
            "layer_0-layer": {
                "label": string
            }
        }
    },
    "layer_n": {
        "metrics": {
            "metric_0": {
                "label": string
                "unit": string
                "min": number
                "med": number
                "max": number
                "current": number
            },
            "metric_n": {
                "label": string
                "unit": string
                "min": number
                "med": number
                "max": number
                "current": number
            }
        },
        "layer": {
            "layer1-layer": {
                "label": string
            }
        }
    }
}

Make sure to replace myFetchFunction and myUrl with the actual function and url you want to use for fetching remote data.

3. Web Workers

When dealing with a high monitoring pace, the utilization of web workers becomes advantageous for efficient data fetching. This method enables the concurrent execution of multiple HTTP queries through distinct workers, preventing potential bottlenecks.

Implementation Details

To enhance the data fetching process, we have introduced an additional web workers pool named httpRequests_pool.

Code

if (timeDifferenceInMilliseconds >= remoteDataFetchPace) {
    // Getting updates...
    if (renderingType === "workers") { // Making HTTP requests using web workers
        const worker = httpRequests_pool.getWorker();
        
        // Setting up event handling for when the worker sends a message
        worker.onmessage = function (e) {
            newData = e.data.newData;
            refreshedData["newData"] = e.data.newData;
            refreshedData["scrapperUpdateInitTime"] = new Date();

           // Releasing the worker back to the pool
           httpRequests_pool.releaseWorker(worker);
        }

        // Initiating the HTTP request by sending a message to the worker
        worker.postMessage({
            subject: "httpRequests",
            fn: conf.fetchFunction.toString()
        });
    } else { 
        // Perform the HTTP request directly on the main thread
        newData = await conf.fetchFunction();
        scrapperUpdateInitTime = new Date();
    }
}

4. Current Examples

1. Open Meteo API:
   • Scrapes weather data from api.open-meteo.com and displays it as a 3D model.
2. Local Live Monitoring:
   • Retrieves local system monitoring data every 1000 ms and displays it as a 3D model.