Constructor for the Echo State Network model. It requires the reservoir size as the input and the data for the training. It returns a struct ready to be trained with the states already harvested.
After the training this struct can be used for the prediction following the second function call. This will take as input a prediction type and the output layer from the training. The initial_conditions and last_state parameters can be left as they are, unless there is a specific reason to change them. All the components are detailed in the API documentation. More examples are given in the general documentation.
These arguments detail more deep variation of the underlying model and they need a separate call. For the moment the most complex is the Hybrid call, but this can and will change in the future. All ESN models can be trained using the following call:
Training of the built ESN over the target_data. The default training method is RidgeRegression. The output is an OutputLayer object to be fed at the esn call for the prediction.
This document was generated with Documenter.jl version 0.27.25 on Thursday 21 September 2023. Using Julia version 1.9.3.
+ last_state=esn.states[:, end])
Constructor for the Echo State Network model. It requires the reservoir size as the input and the data for the training. It returns a struct ready to be trained with the states already harvested.
After the training, this struct can be used for the prediction following the second function call. This will take as input a prediction type and the output layer from the training. The initial_conditions and last_state parameters can be left as they are, unless there is a specific reason to change them. All the components are detailed in the API documentation. More examples are given in the general documentation.
In addition to all the components that can be explored in the documentation, a couple components need a separate introduction. The variation arguments can be
These arguments detail a deeper variation of the underlying model, and they need a separate call. For the moment, the most complex is the Hybrid call, but this can and will change in the future. All ESN models can be trained using the following call:
Training of the built ESN over the target_data. The default training method is RidgeRegression. The output is an OutputLayer object to be fed to the esn call for the prediction.
Returns a Multiple RNN initializer, where multiple function are combined in a linear combination with chosen parameters scaling_factor. The activation_function and scaling_factor arguments must vectors of the same size. Multiple combinations are possible, the implementation is based upon a double activation function idea, found in [1].
[1] Lun, Shu-Xian, et al. "A novel model of leaky integrator echo state network for time-series prediction." Neurocomputing 159 (2015): 58-66.
Returns a Multiple RNN initializer, where multiple functions are combined in a linear combination with chosen parameters scaling_factor. The activation_function and scaling_factor arguments must be vectors of the same size. Multiple combinations are possible. The implementation is based upon the double activation function idea, found in [1].
[1] Lun, Shu-Xian, et al. "A novel model of leaky integrator echo state network for time-series prediction." Neurocomputing 159 (2015): 58-66.
Returns a weighted layer initializer object, that will produce a weighted input matrix with a with random non-zero elements drawn from [-scaling, scaling], as described in [1]. The scaling factor can be given as arg or kwarg.
[1] Lu, Zhixin, et al. "Reservoir observers: Model-free inference of unmeasured variables in chaotic systems." Chaos: An Interdisciplinary Journal of Nonlinear Science 27.4 (2017): 041102.
Returns a fully connected layer initializer object, that will produce a weighted input matrix with a with random non-zero elements drawn from [-scaling, scaling]. The scaling factor can be given as arg or kwarg. This is the default choice in the ESN construction.
Returns a weighted layer initializer object, that will produce a weighted input matrix with random non-zero elements drawn from [-scaling, scaling], as described in [1]. The scaling factor can be given as arg or kwarg.
[1] Lu, Zhixin, et al. "Reservoir observers: Model-free inference of unmeasured variables in chaotic systems." Chaos: An Interdisciplinary Journal of Nonlinear Science 27.4 (2017): 041102.
Returns a fully connected layer initializer object, that will produce a weighted input matrix with random non-zero elements drawn from [-scaling, scaling]. The scaling factor can be given as arg or kwarg. This is the default choice in the ESN construction.
Returns a sparsely connected layer initializer object, that will produce a random sparse input matrix with random non-zero elements drawn from [-scaling, scaling] and given sparsity. The scaling and sparsity factors can be given as args or kwargs.
Returns a weighted input layer matrix, with random non-zero elements drawn from [-scaling, scaling], where some γ of reservoir nodes are connected exclusively to the raw inputs, and the rest to the outputs of the prior knowledge model, as described in [1].
[1] Jaideep Pathak et al. "Hybrid Forecasting of Chaotic Processes: Using Machine Learning in Conjunction with a Knowledge-Based Model" (2018)
Returns a sparsely connected layer initializer object, that will produce a random sparse input matrix with random non-zero elements drawn from [-scaling, scaling] and given sparsity. The scaling and sparsity factors can be given as args or kwargs.
Returns a weighted input layer matrix, with random non-zero elements drawn from [-scaling, scaling], where some γ of reservoir nodes are connected exclusively to the raw inputs, and the rest to the outputs of the prior knowledge model, as described in [1].
[1] Jaideep Pathak et al. "Hybrid Forecasting of Chaotic Processes: Using Machine Learning in Conjunction with a Knowledge-Based Model" (2018)
Returns a fully connected layer initializer object. The matrix constructed with this initializer presents the same absolute weight value, decided by the weight factor. The sign of each entry is decided by the sampling struct. Construction detailed in [1] and [2].
[1] Rodan, Ali, and Peter Tino. "Minimum complexity echo state network." IEEE transactions on neural networks 22.1 (2010): 131-144. [2] Rodan, Ali, and Peter Tiňo. "Simple deterministically constructed cycle reservoirs with regular jumps." Neural computation 24.7 (2012): 1822-1852.
Returns a Bernoulli sign constructor for the MinimumLayer call. The p factor determines the probability of the result as in the Distributions call. The value can be passed as an arg or kwarg. This sign weight determination for input layers is introduced in [1].
[1] Rodan, Ali, and Peter Tino. "Minimum complexity echo state network." IEEE transactions on neural networks 22.1 (2010): 131-144.
Returns an irrational sign contructor for the '''MinimumLayer''' call. The values can be passed as args or kwargs. The sign of the weight are decided from the decimal expansion of the given irrational. The first start decimal digits are thresholded at 4.5, then the n-th input sign will be + and - respectively.
[1] Rodan, Ali, and Peter Tiňo. "Simple deterministically constructed cycle reservoirs with regular jumps." Neural computation 24.7 (2012): 1822-1852.
To create new input layers it suffice to define a new struct containing the needed parameters of the new input layer. This struct wiil need to be an AbstractLayer, so the create_layer function can be dispatched over it. The workflow should follow this snippet:
#creation of the new struct for the layer
+MinimumLayer(;weight=0.1, sampling=BernoulliSample(0.5))
Returns a fully connected layer initializer object. The matrix constructed with this initializer presents the same absolute weight value, decided by the weight factor. The sign of each entry is decided by the sampling struct. Construction detailed in [1] and [2].
[1] Rodan, Ali, and Peter Tino. "Minimum complexity echo state network." IEEE transactions on neural networks 22.1 (2010): 131-144. [2] Rodan, Ali, and Peter Tiňo. "Simple deterministically constructed cycle reservoirs with regular jumps." Neural computation 24.7 (2012): 1822-1852.
Returns a Bernoulli sign constructor for the MinimumLayer call. The p factor determines the probability of the result, as in the Distributions call. The value can be passed as an arg or kwarg. This sign weight determination for input layers is introduced in [1].
[1] Rodan, Ali, and Peter Tino. "Minimum complexity echo state network." IEEE transactions on neural networks 22.1 (2010): 131-144.
Returns an irrational sign constructor for the MinimumLayer call. The values can be passed as args or kwargs. The sign of the weight is decided from the decimal expansion of the given irrational. The first start decimal digits are thresholded at 4.5, then the n-th input sign will be + and - respectively.
[1] Rodan, Ali, and Peter Tiňo. "Simple deterministically constructed cycle reservoirs with regular jumps." Neural computation 24.7 (2012): 1822-1852.
To create new input layers, it suffices to define a new struct containing the needed parameters of the new input layer. This struct will need to be an AbstractLayer, so the create_layer function can be dispatched over it. The workflow should follow this snippet:
#creation of the new struct for the layer
struct MyNewLayer <: AbstractLayer
#the layer params go here
end
@@ -19,14 +15,14 @@
#dispatch over the function to actually build the layer matrix
function create_layer(input_layer::MyNewLayer, res_size, in_size)
#the new algorithm to build the input layer goes here
-end
Returns a random sparse reservoir initializer, that will return a matrix with given sparsity and scaled spectral radius according to radius. This is the default choice in the ESN construction.
Returns a Delay Line Reservoir matrix constructor to obtain a deterministi reservoir as described in [1]. The weight can be passed as arg or kwarg and it determines the absolute value of all the connections in the reservoir.
[1] Rodan, Ali, and Peter Tino. "Minimum complexity echo state network." IEEE transactions on neural networks 22.1 (2010): 131-144.
Returns a Delay Line Reservoir constructor to create a matrix with Backward connections as described in [1]. The weight and fb_weight can be passed as either args or kwargs, and they determine the only absolute values of the connections in the reservoir.
[1] Rodan, Ali, and Peter Tino. "Minimum complexity echo state network." IEEE transactions on neural networks 22.1 (2010): 131-144.
Returns a Simple Cycle Reservoir Reservoir constructor to biuld a reservoir matrix as described in [1]. The weight can be passed as arg or kwarg and it determines the absolute value of all the connections in the reservoir.
[1] Rodan, Ali, and Peter Tino. "Minimum complexity echo state network." IEEE transactions on neural networks 22.1 (2010): 131-144.
Return a Cycle Reservoir with Jumps constructor to create a reservoir matrix as described in [1]. The weight and jump_weight can be passed as args or kwargs and they determine the absolute values of all the connections in the reservoir. The jump_size can also be passed either as arg and kwarg and it detemines the jumps between jump_weights.
[1] Rodan, Ali, and Peter Tiňo. "Simple deterministically constructed cycle reservoirs with regular jumps." Neural computation 24.7 (2012): 1822-1852.
To create a new reservoir the procedure is imilar to the one for the input layers. First the definition of the new struct of type AbstractReservoir with the reservoir parameters is needed. Then the dispatch over the create_reservoir function makes the model actually build the reservoir matrix. An example of the workflow is given in the following snippet:
#creation of the new struct for the reservoir
+end
Returns a random sparse reservoir initializer, that will return a matrix with given sparsity and scaled spectral radius according to radius. This is the default choice in the ESN construction.
Returns a Delay Line Reservoir matrix constructor to obtain a deterministic reservoir as described in [1]. The weight can be passed as arg or kwarg, and it determines the absolute value of all the connections in the reservoir.
[1] Rodan, Ali, and Peter Tino. "Minimum complexity echo state network." IEEE transactions on neural networks 22.1 (2010): 131-144.
Returns a Delay Line Reservoir constructor to create a matrix with Backward connections as described in [1]. The weight and fb_weight can be passed as either args or kwargs, and they determine the only absolute values of the connections in the reservoir.
[1] Rodan, Ali, and Peter Tino. "Minimum complexity echo state network." IEEE transactions on neural networks 22.1 (2010): 131-144.
Returns a Simple Cycle Reservoir constructor to build a reservoir matrix as described in [1]. The weight can be passed as arg or kwarg, and it determines the absolute value of all the connections in the reservoir.
[1] Rodan, Ali, and Peter Tino. "Minimum complexity echo state network." IEEE transactions on neural networks 22.1 (2010): 131-144.
Return a Cycle Reservoir with Jumps constructor to create a reservoir matrix as described in [1]. The weight and jump_weight can be passed as args or kwargs, and they determine the absolute values of all the connections in the reservoir. The jump_size can also be passed either as arg or kwarg, and it detemines the jumps between jump_weights.
[1] Rodan, Ali, and Peter Tiňo. "Simple deterministically constructed cycle reservoirs with regular jumps." Neural computation 24.7 (2012): 1822-1852.
To create a new reservoir, the procedure is similar to the one for the input layers. First, the definition of the new struct of type AbstractReservoir with the reservoir parameters is needed. Then the dispatch over the create_reservoir function makes the model actually build the reservoir matrix. An example of the workflow is given in the following snippet:
#creation of the new struct for the reservoir
struct MyNewReservoir <: AbstractReservoir
#the reservoir params go here
end
@@ -34,4 +30,4 @@
#dispatch over the function to build the reservoir matrix
function create_reservoir(reservoir::AbstractReservoir, res_size)
#the new algorithm to build the reservoir matrix goes here
-end
Settings
This document was generated with Documenter.jl version 0.27.25 on Thursday 21 September 2023. Using Julia version 1.9.3.
+end
Settings
This document was generated with Documenter.jl version 1.0.1 on Sunday 24 September 2023. Using Julia version 1.9.3.
This prediction methodology allows the models to produce an autonomous prediction, feeding the prediction into itself to generate the next step. The only parameter needed is the number of steps for the prediction.
This prediction methodology allows the models to produce an autonomous prediction, feeding the prediction into itself to generate the next step. The only parameter needed is the number of steps for the prediction.
Random mapping of the input data directly in the reservoir. The expansion_size determines the dimension of the single reservoir, and permutations determines the number of total reservoirs that will be connected, each with a different mapping. The detail of this implementation can be found in [1].
[1] Nichele, Stefano, and Andreas Molund. “Deep reservoir computing using cellular automata.” arXiv preprint arXiv:1703.02806 (2017).
The training and prediction follow the same workflow of the ESN. It is important to note that at the moment we were not able to find any paper using these models with a Generative approach for the prediction, so full support is given only to the Predictive method.
Settings
This document was generated with Documenter.jl version 0.27.25 on Thursday 21 September 2023. Using Julia version 1.9.3.
Random mapping of the input data directly in the reservoir. The expansion_size determines the dimension of the single reservoir, and permutations determines the number of total reservoirs that will be connected, each with a different mapping. The detail of this implementation can be found in [1].
[1] Nichele, Stefano, and Andreas Molund. “Deep reservoir computing using cellular automata.” arXiv preprint arXiv:1703.02806 (2017).
The training and prediction follow the same workflow as the ESN. It is important to note that currently we were unable to find any papers using these models with a Generative approach for the prediction, so full support is given only to the Predictive method.
Settings
This document was generated with Documenter.jl version 1.0.1 on Sunday 24 September 2023. Using Julia version 1.9.3.
The states are extended with the input data, for the training section, and the prediction data, during the prediction section. This is obtained with a vertical concatenation of the data and the states.
The states are padded with a chosen value. Usually this value is set to one. The padding is obtained through a vertical concatenation of the padding value and the states.
The states are extended with the training data or predicted data and subsequently padded with a chosen value. Usually the padding value is set to one. The padding and the extension are obtained through a vertical concatenation of the padding value, the data and the states.
Applies the $ \text{T}_1 $ transformation algorithm, as defined in [1] and [2].
[1] Chattopadhyay, Ashesh, et al. "Data-driven prediction of a multi-scale Lorenz 96 chaotic system using a hierarchy of deep learning methods: Reservoir computing, ANN, and RNN-LSTM." (2019).
[2] Pathak, Jaideep, et al. "Model-free prediction of large spatiotemporally chaotic systems from data: A reservoir computing approach." Physical review letters 120.2 (2018): 024102.
Apply the $ \text{T}_2 $ transformation algorithm, as defined in [1].
[1] Chattopadhyay, Ashesh, et al. "Data-driven prediction of a multi-scale Lorenz 96 chaotic system using a hierarchy of deep learning methods: Reservoir computing, ANN, and RNN-LSTM." (2019).
Apply the $ \text{T}_3 $ transformation algorithm, as defined in [1].
[1] Chattopadhyay, Ashesh, et al. "Data-driven prediction of a multi-scale Lorenz 96 chaotic system using a hierarchy of deep learning methods: Reservoir computing, ANN, and RNN-LSTM." (2019).
The states are extended with the input data, for the training section, and the prediction data, during the prediction section. This is obtained with a vertical concatenation of the data and the states.
The states are padded with a chosen value. Usually, this value is set to one. The padding is obtained through a vertical concatenation of the padding value and the states.
The states are extended with the training data or predicted data and subsequently padded with a chosen value. Usually, the padding value is set to one. The padding and the extension are obtained through a vertical concatenation of the padding value, the data, and the states.
Applies the $ \text{T}_1 $ transformation algorithm, as defined in [1] and [2].
[1] Chattopadhyay, Ashesh, et al. "Data-driven prediction of a multi-scale Lorenz 96 chaotic system using a hierarchy of deep learning methods: Reservoir computing, ANN, and RNN-LSTM." (2019).
[2] Pathak, Jaideep, et al. "Model-free prediction of large spatiotemporally chaotic systems from data: A reservoir computing approach." Physical review letters 120.2 (2018): 024102.
Apply the $ \text{T}_2 $ transformation algorithm, as defined in [1].
[1] Chattopadhyay, Ashesh, et al. "Data-driven prediction of a multi-scale Lorenz 96 chaotic system using a hierarchy of deep learning methods: Reservoir computing, ANN, and RNN-LSTM." (2019).
Apply the $ \text{T}_3 $ transformation algorithm, as defined in [1].
[1] Chattopadhyay, Ashesh, et al. "Data-driven prediction of a multi-scale Lorenz 96 chaotic system using a hierarchy of deep learning methods: Reservoir computing, ANN, and RNN-LSTM." (2019).
Linear regression training based on MLJLinearModels for all the models in the library. All the parameters have to be passed into regression_kwargs, apart from the solver choice. MLJLinearModels.jl needs to be called in order to use these models.
Support vector Regression is possible using a direct call to LIBSVM regression methods. Instead of a wrapper please refer to the use of LIBSVM.AbstractSVR in the original library.
Settings
This document was generated with Documenter.jl version 0.27.25 on Thursday 21 September 2023. Using Julia version 1.9.3.
+ regression_kwargs=(;))
Linear regression training based on MLJLinearModels for all the models in the library. All the parameters have to be passed into regression_kwargs, apart from the solver choice. MLJLinearModels.jl needs to be called in order to use these models.
Support Vector Regression is possible using a direct call to LIBSVM regression methods. Instead of a wrapper, please refer to the use of LIBSVM.AbstractSVR in the original library.
Settings
This document was generated with Documenter.jl version 1.0.1 on Sunday 24 September 2023. Using Julia version 1.9.3.
+ `;
+
+ return filter_html;
+}
+
+/**
+ * Make the result component given a minisearch result data object and the value of the search input as queryString.
+ * To view the result object structure, refer: https://lucaong.github.io/minisearch/modules/_minisearch_.html#searchresult
+ *
+ * @param {object} result
+ * @param {string} querystring
+ * @returns string
+ */
+function make_search_result(result, querystring) {
+ let search_divider = ``;
+ let display_link =
+ result.location.slice(Math.max(0), Math.min(50, result.location.length)) +
+ (result.location.length > 30 ? "..." : ""); // To cut-off the link because it messes with the overflow of the whole div
+
+ if (result.page !== "") {
+ display_link += ` (${result.page})`;
+ }
+
+ let textindex = new RegExp(`\\b${querystring}\\b`, "i").exec(result.text);
+ let text =
+ textindex !== null
+ ? result.text.slice(
+ Math.max(textindex.index - 100, 0),
+ Math.min(
+ textindex.index + querystring.length + 100,
+ result.text.length
+ )
+ )
+ : ""; // cut-off text before and after from the match
+
+ let display_result = text.length
+ ? "..." +
+ text.replace(
+ new RegExp(`\\b${querystring}\\b`, "i"), // For first occurrence
+ '$&'
+ ) +
+ "..."
+ : ""; // highlights the match
+
+ let in_code = false;
+ if (!["page", "section"].includes(result.category.toLowerCase())) {
+ in_code = true;
+ }
+
+ // We encode the full url to escape some special characters which can lead to broken links
+ let result_div = `
+
+
A great deal of efforts in the ESNs field are devoted to finding an ideal construction for the reservoir matrices. With a simple interface using ReservoirComputing.jl is possible to leverage the currently implemented matrix constructions methods for both the reservoir and the input layer. In this page it is showcased how it is possible to change both of these layers.
The input_init keyword argument provided with the ESN constructor allows for changing the input layer. The layers provided in ReservoirComputing.jl are the following:
A great deal of effort in the ESNs field is devoted to finding the ideal construction for the reservoir matrices. With a simple interface using ReservoirComputing.jl it is possible to leverage the currently implemented matrix construction methods for both the reservoir and the input layer. On this page, it is showcased how it is possible to change both of these layers.
The input_init keyword argument provided with the ESN constructor allows for changing the input layer. The layers provided in ReservoirComputing.jl are the following:
In addition, the user can define a custom layer following this workflow:
#creation of the new struct for the layer
struct MyNewLayer <: AbstractLayer
#the layer params go here
end
@@ -19,7 +15,7 @@
#dispatch over the function to build the reservoir matrix
function create_reservoir(reservoir::AbstractReservoir, res_size)
#the new algorithm to build the reservoir matrix goes here
-end
Using [1] and [2] as references this section will provide an example on how to change both the input layer and the reservoir for ESNs. The full script for this example can be found here. This example was run on Julia v1.7.2.
The task for this example will be the one step ahead prediction of the Henon map. To obtain the data one can leverage the package DynamicalSystems.jl. The data is scaled to be between -1 and 1.
Using [1] and [2] as references, this section will provide an example of how to change both the input layer and the reservoir for ESNs. The full script for this example can be found here. This example was run on Julia v1.7.2.
The task for this example will be the one step ahead prediction of the Henon map. To obtain the data, one can leverage the package DynamicalSystems.jl. The data is scaled to be between -1 and 1.
using PredefinedDynamicalSystems
train_len = 3000
predict_len = 2000
@@ -49,4 +45,4 @@
output = esn(Predictive(testing_input), wout)
println(msd(testing_target, output))
end
0.0036884815121632775
-0.003480856411242415
As it is possible to see, changing layers in ESN models is straightforward. Be sure to check the API documentation for a full list of reservoir and layers.
1Rodan, Ali, and Peter Tiňo. “Simple deterministically constructed cycle reservoirs with regular jumps.” Neural computation 24.7 (2012): 1822-1852.
2Rodan, Ali, and Peter Tiňo. “Minimum complexity echo state network.” IEEE transactions on neural networks 22.1 (2010): 131-144.
Settings
This document was generated with Documenter.jl version 0.27.25 on Thursday 21 September 2023. Using Julia version 1.9.3.
+0.003480856411242415
As it is possible to see, changing layers in ESN models is straightforward. Be sure to check the API documentation for a full list of reservoirs and layers.
Deep Echo State Network architectures started to gain some traction recently. In this guide we illustrate how it is possible to use ReservoirComputing.jl to build a deep ESN.
The network implemented in this library is taken from [1]. It works by stacking reservoirs on top of each other, feeding the output on one in the next. The states are obtained by merging all the inner states of the stacked reservoirs. For a more in depth explanation refer to the paper linked above. The full script for this example can be found here. This example was run on Julia v1.7.2.
Deep Echo State Network architectures started to gain some traction recently. In this guide, we illustrate how it is possible to use ReservoirComputing.jl to build a deep ESN.
The network implemented in this library is taken from [1]. It works by stacking reservoirs on top of each other, feeding the output from one into the next. The states are obtained by merging all the inner states of the stacked reservoirs. For a more in-depth explanation, refer to the paper linked above. The full script for this example can be found here. This example was run on Julia v1.7.2.
For this example, we are going to reuse the Lorenz data used in the Lorenz System Forecasting example.
using OrdinaryDiffEq
#define lorenz system
function lorenz!(du,u,p,t)
@@ -27,7 +23,7 @@
test_data = data[:,shift+train_len+1:shift+train_len+predict_len]
Again, it is important to notice that the data needs to be formatted in a matrix with the features as rows and time steps as columns like it is done in this example. This is needed even if the time series consists of single values.
The construction of the ESN is also really similar. The only difference is that the reservoir can be fed as an array of reservoirs.
using ReservoirComputing
+ 25.6556 24.4858 23.3794 22.3394 24.3255 25.4374 26.6729
Again, it is important to notice that the data needs to be formatted in a matrix, with the features as rows and time steps as columns, as in this example. This is needed even if the time series consists of single values.
The construction of the ESN is also really similar. The only difference is that the reservoir can be fed as an array of reservoirs.
As it is possible to see, different sizes can be chosen for the different reservoirs. The input layer and bias can also be given as vectors, but of course they have to be of the same size of the reservoirs vector. If they are not passed as a vector, the value passed is going to be used for all the layers in the deep ESN.
In addition to using the provided functions for the construction of the layers the user can also choose to build their own matrix, or array of matrices, and feed that into the ESN in the same way.
The training and prediction follows the usual framework:
As it is possible to see, different sizes can be chosen for the different reservoirs. The input layer and bias can also be given as vectors, but of course, they have to be of the same size of the reservoirs vector. If they are not passed as a vector, the value passed will be used for all the layers in the deep ESN.
In addition to using the provided functions for the construction of the layers, the user can also choose to build their own matrix, or array of matrices, and feed that into the ESN in the same way.
The training and prediction follow the usual framework:
Note that there is a known bug at the moment with using WeightedLayer as the input layer with the deep ESN. We are in the process of investigating and solving it. The leak coefficient for the reservoirs has to always be the same with the current implementation. This is also something we are actively looking into expanding.
1Gallicchio, Claudio, and Alessio Micheli. "Deep echo state network (deepesn): A brief survey." arXiv preprint arXiv:1712.04323 (2017).
Settings
This document was generated with Documenter.jl version 0.27.25 on Thursday 21 September 2023. Using Julia version 1.9.3.
+ legendfontsize=12, titlefontsize=20)
Note that there is a known bug at the moment with using WeightedLayer as the input layer with the deep ESN. We are in the process of investigating and solving it. The leak coefficient for the reservoirs has to always be the same in the current implementation. This is also something we are actively looking into expanding.
While the original implementation of the Echo State Network implemented the model using the equations of Recurrent Neural Networks to obtain non linearity in the reservoir, other variations have been proposed in recent years. More specifically the different drivers implemented in ReservoirComputing.jl are the multiple activation function RNN MRNN() and the Gated Recurrent Unit GRU(). To change them it suffice to give the chosen method to the ESN keyword argument reservoir_driver. In this section some example of their usage will be given, as well as a quick introduction to their equations.
Based on the double activation function ESN (DAFESN) proposed in [1], the Multiple Activation Function ESN expands the idea and allows a custom number of activation functions to be used in the reservoir dynamics. This can be thought as a linear combination of multiple activation functions with corresponding parameters.
where $D$ is the number of activation function and respective parameters chosen.
The method to call to use the mutliple activation function ESN is MRNN(activation_function, leaky_coefficient, scaling_factor). The arguments can be used as both args or kwargs. activation_function and scaling_factor have to be vectors (or tuples) containing the chosen activation functions and respective scaling factors ($f_1,...,f_D$ and $\lambda_1,...,\lambda_D$ following the nomenclature introduced above). The leaky_coefficient represents $\alpha$ and it is a single value.
Starting the example, the data used is based on the following function based on the DAFESN paper [1]. A full script of the example is available here. This example was run on Julia v1.7.2.
For this example the type of prediction will be one step ahead. The metric used to assure a good prediction is going to be the normalized root-mean-square deviation rmsd from StatsBase. Like in the other examples first it is needed to gather the data:
train_len = 3000
+Using Different Reservoir Drivers · ReservoirComputing.jl
While the original implementation of the Echo State Network implemented the model using the equations of Recurrent Neural Networks to obtain non-linearity in the reservoir, other variations have been proposed in recent years. More specifically, the different drivers implemented in ReservoirComputing.jl are the multiple activation function RNN MRNN() and the Gated Recurrent Unit GRU(). To change them, it suffices to give the chosen method to the ESN keyword argument reservoir_driver. In this section, some examples, of their usage will be given, as well as a brief introduction to their equations.
Based on the double activation function ESN (DAFESN) proposed in [1], the Multiple Activation Function ESN expands the idea and allows a custom number of activation functions to be used in the reservoir dynamics. This can be thought of as a linear combination of multiple activation functions with corresponding parameters.
where $D$ is the number of activation functions and respective parameters chosen.
The method to call to use the multiple activation function ESN is MRNN(activation_function, leaky_coefficient, scaling_factor). The arguments can be used as both args and kwargs. activation_function and scaling_factor have to be vectors (or tuples) containing the chosen activation functions and respective scaling factors ($f_1,...,f_D$ and $\lambda_1,...,\lambda_D$ following the nomenclature introduced above). The leaky_coefficient represents $\alpha$ and it is a single value.
Starting with the example, the data used is based on the following function based on the DAFESN paper [1]. A full script of the example is available here. This example was run on Julia v1.7.2.
For this example, the type of prediction will be one step ahead. The metric used to assure a good prediction will be the normalized root-mean-square deviation rmsd from StatsBase. Like in the other examples, first it is needed to gather the data:
In order to follow the paper more closely it is necessary to define a couple of activation functions. The numbering of them follows the ones in the paper. Of course one can also use any function, custom defined, available in the base language or any activation function from NNlib.
To follow the paper more closely, it is necessary to define a couple of activation functions. The numbering of them follows the ones in the paper. Of course, one can also use any custom-defined function, available in the base language or any activation function from NNlib.
It is now possible to build different drivers, using the parameters suggested by the paper. Also in this instance the numbering follows the test cases of the paper. In the end a simple for loop is implemented to compare the different drivers and activation functions.
using ReservoirComputing, Random, StatsBase
+f4(x) = x/sqrt(1+x*x)
f4 (generic function with 1 method)
It is now possible to build different drivers, using the parameters suggested by the paper. Also, in this instance, the numbering follows the test cases of the paper. In the end, a simple for loop is implemented to compare the different drivers and activation functions.
using ReservoirComputing, Random, StatsBase
#fix seed for reproducibility
Random.seed!(42)
@@ -50,15 +46,15 @@
end
Gated Recurrent Units (GRUs) [2] have been proposed in more recent years with the intent of limiting notable problems of RNNs, like the vanishing gradient. This change in the underlying equations can be easily transported in the Reservoir Computing paradigm, switching the RNN equations in the reservoir with the GRU equations. This approach has been explored in [3] and [4]. Different variations of GRU have been proposed [5][6]; this section is subdivided into different sections that go in detail about the governing equations and the implementation of them into ReservoirComputing.jl. Like before, to access the GRU reservoir driver it suffice to change the reservoir_diver keyword argument for ESN with GRU(). All the variations that are going to be presented can be used in this package by leveraging the keyword argument variant in the method GRU() and specifying the chosen variant: FullyGated() or Minimal(). Other variations are possible modifying the inner layers and reservoirs. The default is set to the standard version FullyGated(). The first section will go in more detail about the default of the GRU() method, and the following ones will refer to it to minimize repetitions. This example was run on Julia v1.7.2.
Gated Recurrent Units (GRUs) [2] have been proposed in more recent years with the intent of limiting notable problems of RNNs, like the vanishing gradient. This change in the underlying equations can be easily transported into the Reservoir Computing paradigm, by switching the RNN equations in the reservoir with the GRU equations. This approach has been explored in [3] and [4]. Different variations of GRU have been proposed [5][6]; this section is subdivided into different sections that go into detail about the governing equations and the implementation of them into ReservoirComputing.jl. Like before, to access the GRU reservoir driver, it suffices to change the reservoir_diver keyword argument for ESN with GRU(). All the variations that will be presented can be used in this package by leveraging the keyword argument variant in the method GRU() and specifying the chosen variant: FullyGated() or Minimal(). Other variations are possible by modifying the inner layers and reservoirs. The default is set to the standard version FullyGated(). The first section will go into more detail about the default of the GRU() method, and the following ones will refer to it to minimize repetitions. This example was run on Julia v1.7.2.
Going over the GRU keyword argument it will be explained how to feed the desired input to the model.
activation_function is a vector with default values [NNlib.sigmoid, NNlib.sigmoid, tanh]. This argument controls the activation functions of the GRU, going from top to bottom. Changing the first element corresponds in changing the activation function for $\mathbf{r}(t)$ and so on.
inner_layer is a vector with default values fill(DenseLayer(), 2). This keyword argument controls the $\mathbf{W}_{\text{in}}$s going from top to bottom like before.
reservoir is a vector with default value fill(RandSparseReservoir(), 2). In a similar fashion to inner_layer, this keyword argument controls the reservoir matrix construction in a top to bottom order.
bias is again a vector with default value fill(DenseLayer(), 2). It is meant to control the $\mathbf{b}$s, going as usual from top to bottom.
variant as already illustrated controls the GRU variant. The default value is set to FullyGated().
It is important to notice that inner_layer and reservoir control every layer except $\mathbf{W}_{in}$ and $\mathbf{W}$ and $\mathbf{b}$. These arguments are given as input to the ESN() call as input_layer, reservoir and bias.
The following sections are going to illustrate the variations of the GRU architecture and how to obtain them in ReservoirComputing.jl
Similarly to before, to obtain this variation it is only needed to set inner_layer = fill(NullLayer(), 2) and bias = fill(NullLayer(), 2) while keeping variant = FullyGated().
Going over the GRU keyword argument, it will be explained how to feed the desired input to the model.
activation_function is a vector with default values [NNlib.sigmoid, NNlib.sigmoid, tanh]. This argument controls the activation functions of the GRU, going from top to bottom. Changing the first element corresponds to changing the activation function for $\mathbf{r}(t)$ and so on.
inner_layer is a vector with default values fill(DenseLayer(), 2). This keyword argument controls the $\mathbf{W}_{\text{in}}$s going from top to bottom like before.
reservoir is a vector with default value fill(RandSparseReservoir(), 2). In a similar fashion to inner_layer, this keyword argument controls the reservoir matrix construction in a top to bottom order.
bias is again a vector with default value fill(DenseLayer(), 2). It is meant to control the $\mathbf{b}$s, going as usual from top to bottom.
variant controls the GRU variant. The default value is set to FullyGated().
It is important to notice that inner_layer and reservoir control every layer except $\mathbf{W}_{in}$ and $\mathbf{W}$ and $\mathbf{b}$. These arguments are given as input to the ESN() call as input_layer, reservoir and bias.
The following sections are going to illustrate the variations of the GRU architecture and how to obtain them in ReservoirComputing.jl
Similarly to before, to obtain this variation, it is only required to set inner_layer = fill(NullLayer(), 2) and bias = fill(NullLayer(), 2) while keeping variant = FullyGated().
This means that it is only needed to set inner_layer = fill(NullLayer(), 2) and reservoir = fill(NullReservoir(), 2) while keeping variant = FullyGated().
This variation can be obtained by setting variation=Minimal(). The inner_layer, reservoir and bias kwargs this time are not vectors, but must be defined like, for example inner_layer = DenseLayer() or reservoir = SparseDenseReservoir().
To showcase the use of the GRU() method this section will only illustrate the standard FullyGated() version. The full script for this example with the data can be found here.
The data used for this example is the Santa Fe laser dataset [7] retrieved from here. The data is split to account for a next step prediction.
This variation can be obtained by setting variation=Minimal(). The inner_layer, reservoir and bias kwargs this time are not vectors, but must be defined like, for example inner_layer = DenseLayer() or reservoir = SparseDenseReservoir().
To showcase the use of the GRU() method, this section will only illustrate the standard FullyGated() version. The full script for this example with the data can be found here.
The data used for this example is the Santa Fe laser dataset [7] retrieved from here. The data is split to account for a next step prediction.
The default inner reservoir and input layer for the GRU are the same defaults for the reservoir and input_layer of the ESN. One can use the explicit call if they choose so.
The default inner reservoir and input layer for the GRU are the same defaults for the reservoir and input_layer of the ESN. One can use the explicit call if they choose to.
It is interesting to see a comparison of the GRU driven ESN and the standard RNN driven ESN. Using the same parameters defined before it is possible to do the following
using StatsBase
+ size=(1080, 720))
It is interesting to see a comparison of the GRU driven ESN and the standard RNN driven ESN. Using the same parameters defined before it is possible to do the following
1Lun, Shu-Xian, et al. "A novel model of leaky integrator echo state network for time-series prediction." Neurocomputing 159 (2015): 58-66.
2Cho, Kyunghyun, et al. “Learning phrase representations using RNN encoder-decoder for statistical machine translation.” arXiv preprint arXiv:1406.1078 (2014).
3Wang, Xinjie, Yaochu Jin, and Kuangrong Hao. "A Gated Recurrent Unit based Echo State Network." 2020 International Joint Conference on Neural Networks (IJCNN). IEEE, 2020.
4Di Sarli, Daniele, Claudio Gallicchio, and Alessio Micheli. "Gated Echo State Networks: a preliminary study." 2020 International Conference on INnovations in Intelligent SysTems and Applications (INISTA). IEEE, 2020.
5Dey, Rahul, and Fathi M. Salem. "Gate-variants of gated recurrent unit (GRU) neural networks." 2017 IEEE 60th international midwest symposium on circuits and systems (MWSCAS). IEEE, 2017.
6Zhou, Guo-Bing, et al. "Minimal gated unit for recurrent neural networks." International Journal of Automation and Computing 13.3 (2016): 226-234.
7Hübner, Uwe, Nimmi B. Abraham, and Carlos O. Weiss. "Dimensions and entropies of chaotic intensity pulsations in a single-mode far-infrared NH 3 laser." Physical Review A 40.11 (1989): 6354.
Settings
This document was generated with Documenter.jl version 0.27.25 on Thursday 21 September 2023. Using Julia version 1.9.3.
+20.96408330872688
1Lun, Shu-Xian, et al. "A novel model of leaky integrator echo state network for time-series prediction." Neurocomputing 159 (2015): 58-66.
2Cho, Kyunghyun, et al. “Learning phrase representations using RNN encoder-decoder for statistical machine translation.” arXiv preprint arXiv:1406.1078 (2014).
3Wang, Xinjie, Yaochu Jin, and Kuangrong Hao. "A Gated Recurrent Unit based Echo State Network." 2020 International Joint Conference on Neural Networks (IJCNN). IEEE, 2020.
4Di Sarli, Daniele, Claudio Gallicchio, and Alessio Micheli. "Gated Echo State Networks: a preliminary study." 2020 International Conference on INnovations in Intelligent SysTems and Applications (INISTA). IEEE, 2020.
5Dey, Rahul, and Fathi M. Salem. "Gate-variants of gated recurrent unit (GRU) neural networks." 2017 IEEE 60th international midwest symposium on circuits and systems (MWSCAS). IEEE, 2017.
6Zhou, Guo-Bing, et al. "Minimal gated unit for recurrent neural networks." International Journal of Automation and Computing 13.3 (2016): 226-234.
7Hübner, Uwe, Nimmi B. Abraham, and Carlos O. Weiss. "Dimensions and entropies of chaotic intensity pulsations in a single-mode far-infrared NH 3 laser." Physical Review A 40.11 (1989): 6354.
Settings
This document was generated with Documenter.jl version 1.0.1 on Sunday 24 September 2023. Using Julia version 1.9.3.
diff --git a/dev/esn_tutorials/different_training/index.html b/dev/esn_tutorials/different_training/index.html
index 57d5c7e9..409dabad 100644
--- a/dev/esn_tutorials/different_training/index.html
+++ b/dev/esn_tutorials/different_training/index.html
@@ -1,6 +1,2 @@
-Using Different Training Methods · ReservoirComputing.jl
Following the idea of giving physical information to machine learning models the hybrid echo state networks [1] try to achieve this results by feeding model data into the ESN. In this example it is explained how to create and leverage such models in ReservoirComputing.jl. The full script for this example is available here. This example was run on Julia v1.7.2.
Following the idea of giving physical information to machine learning models, the hybrid echo state networks [1] try to achieve this results by feeding model data into the ESN. In this example, it is explained how to create and leverage such models in ReservoirComputing.jl. The full script for this example is available here. This example was run on Julia v1.7.2.
To feed the data to the ESN, it is necessary to create a suitable function.
function prior_model_data_generator(u0, tspan, tsteps, model = lorenz)
prob = ODEProblem(lorenz, u0, tspan)
sol = Array(solve(prob, saveat = tsteps))
return sol
-end
prior_model_data_generator (generic function with 2 methods)
Given initial condition, time span and time steps this function returns the data for the chosen model. Now, using the Hybrid method it is possible to input all this information to the model
using ReservoirComputing, Random
+end
prior_model_data_generator (generic function with 2 methods)
Given the initial condition, time span, and time steps, this function returns the data for the chosen model. Now, using the Hybrid method, it is possible to input all this information to the model.
1Pathak, Jaideep, et al. "Hybrid forecasting of chaotic processes: Using machine learning in conjunction with a knowledge-based model." Chaos: An Interdisciplinary Journal of Nonlinear Science 28.4 (2018): 041101.
Settings
This document was generated with Documenter.jl version 0.27.25 on Thursday 21 September 2023. Using Julia version 1.9.3.
1Pathak, Jaideep, et al. "Hybrid forecasting of chaotic processes: Using machine learning in conjunction with a knowledge-based model." Chaos: An Interdisciplinary Journal of Nonlinear Science 28.4 (2018): 041101.
Settings
This document was generated with Documenter.jl version 1.0.1 on Sunday 24 September 2023. Using Julia version 1.9.3.
diff --git a/dev/esn_tutorials/lorenz_basic/32790bda.svg b/dev/esn_tutorials/lorenz_basic/32790bda.svg
new file mode 100644
index 00000000..b352cb05
--- /dev/null
+++ b/dev/esn_tutorials/lorenz_basic/32790bda.svg
@@ -0,0 +1,92 @@
+
+
diff --git a/dev/esn_tutorials/lorenz_basic/index.html b/dev/esn_tutorials/lorenz_basic/index.html
index eafe2c54..471f6ed7 100644
--- a/dev/esn_tutorials/lorenz_basic/index.html
+++ b/dev/esn_tutorials/lorenz_basic/index.html
@@ -1,9 +1,5 @@
-Lorenz System Forecasting · ReservoirComputing.jl
This example expands on the readme Lorenz system forecasting to better showcase how to use methods and functions provided in the library for Echo State Networks. Here the prediction method used is Generative, for a more detailed explanation of the differences between Generative and Predictive please refer to the other examples given in the documentation. The full script for this example is available here. This example was run on Julia v1.7.2.
Starting off the workflow the first step is to obtain the data. Leveraging OrdinaryDiffEq it is possible to derive the Lorenz system data in the following way:
using OrdinaryDiffEq
+Lorenz System Forecasting · ReservoirComputing.jl
This example expands on the readme Lorenz system forecasting to better showcase how to use methods and functions provided in the library for Echo State Networks. Here the prediction method used is Generative, for a more detailed explanation of the differences between Generative and Predictive please refer to the other examples given in the documentation. The full script for this example is available here. This example was run on Julia v1.7.2.
Starting off the workflow, the first step is to obtain the data. Leveraging OrdinaryDiffEq it is possible to derive the Lorenz system data in the following way:
using OrdinaryDiffEq
#define lorenz system
function lorenz!(du,u,p,t)
@@ -57,7 +53,7 @@
[-6.496311609072913, -0.7054966719917979, 31.41147109034457]
[-5.412475842259617, -0.3802159215457954, 29.84101721524591]
[-4.486956988649644, -0.26529770021033905, 28.320947150047015]
- [-3.722463627032463, -0.2914952957436106, 26.87140831430775]
After obtaining the data it is necessary to determine the kind of prediction for the model. Since this example is going to use the Generative prediction type, this means that the target data is foing to be the next step of the input data. In addition it is important to notice that the Lorenz system just obtained presents a transient period that is not representative of the general behavior of the system. This can easily be discarded setting a shift parameter.
#determine shift length, training length and prediction length
+ [-3.722463627032463, -0.2914952957436106, 26.87140831430775]
After obtaining the data, it is necessary to determine the kind of prediction for the model. Since this example will use the Generative prediction type, this means that the target data will be the next step of the input data. In addition, it is important to notice that the Lorenz system just obtained presents a transient period that is not representative of the general behavior of the system. This can easily be discarded by setting a shift parameter.
It is important to notice that the data needs to be formatted in a matrix with the features as rows and time steps as columns like it is done in this example. This is needed even if the time series consists of single values.
Once the data is ready it is possible to define the parameters for the ESN and the ESN struct itself. In this example the values from [1] are loosely followed as general guidelines.
using ReservoirComputing
+ 25.6556 24.4858 23.3794 22.3394 24.3255 25.4374 26.6729
It is important to notice that the data needs to be formatted in a matrix with the features as rows and time steps as columns as in this example. This is needed even if the time series consists of single values.
Once the data is ready, it is possible to define the parameters for the ESN and the ESN struct itself. In this example, the values from [1] are loosely followed as general guidelines.
Most of the parameters here chosen mirror the default ones, so a direct call is not necessary. The readme example is identical to this one, except for the explicit call. Going line by line to see what is happening starting from res_size: this value determines the dimensions of the reservoir matrix. In this case a size of 300 has been chosen, so the reservoir matrix is going to be 300 x 300. This is not always the case, since some input layer constructions can modify the dimensions of the reservoir, but in that case everything is taken care of internally.
The res_radius determines the scaling of the spectral radius of the reservoir matrix; a proper scaling is necessary to assure the Echo State Property. The default value in the RandSparseReservoir() method is 1.0 in accordance to the most followed guidelines found in the literature (see [2] and references therein). The sparsity of the reservoir matrix in this case is obtained by choosing a degree of connections and dividing that by the reservoir size. Of course it is also possible to simply choose any value between 0.0 and 1.0 to test behaviors for different sparsity values. In this example the call to the parameters inside RandSparseReservoir() was done explicitly to showcase the meaning of each of them, but it is also possible to simply pass the values directly like so RandSparseReservoir(1.2, 6/300).
The value of input_scaling determines the upper and lower bounds of the uniform distribution of the weights in the WeightedLayer(). Like before this value can be passed either as argument or keyword argument WeightedLayer(0.1). The value of 0.1 represents the default. The default input layer is the DenseLayer, a fully connected layer. The details of the weighted version can be found in [3], for this example this version returns the best results.
The reservoir driver represents the dynamics of the reservoir. In the standard ESN definition these dynamics are obtained through a Recurrent Neural Network (RNN), and this is reflected by calling the RNN driver for the ESN struct. This option is set as the default and unless there is the need to change parameters it is not needed. The full equation is the following:
where $α$ represents the leaky coefficient and tanh can be any activation function. Also $\textbf{x}$ represent the state vector, $\textbf{u}$ the input data and $\textbf{W}, \textbf{W}_{\text{in}}$ are the reservoir matrix and input matrix respectively. The default call to the RNN in the library is the following RNN(;activation_function=tanh, leaky_coefficient=1.0), where the meaning of the parameters is clear from the equation above. Instead og the hyperbolic tangent any activation function can be used, either leveraging external lybraries such as NNlib or creating a custom one.
The final calls are modifications to the states in training or prediction. The default calls, depicted in the example, do not make any modifications to the states. This is the safest bet is one is not sure on how these work. The nla_type applies a non linear algorithm to the states, while the states_type can expand them concatenating them with the input data, or padding them concatenating a constant value to all the states. More in depth descriptions about these parameters are given in other examples in the documentation.
Now that the ESN has been created and all the parameters have been explained it is time to proceed with the training. The full call of the readme example follows this general idea:
#define training method
+ states_type = StandardStates())
Most of the parameters chosen here mirror the default ones, so a direct call is not necessary. The readme example is identical to this one, except for the explicit call. Going line by line to see what is happening, starting from res_size: this value determines the dimensions of the reservoir matrix. In this case, a size of 300 has been chosen, so the reservoir matrix will be 300 x 300. This is not always the case, since some input layer constructions can modify the dimensions of the reservoir, but in that case, everything is taken care of internally.
The res_radius determines the scaling of the spectral radius of the reservoir matrix; a proper scaling is necessary to assure the Echo State Property. The default value in the RandSparseReservoir() method is 1.0 in accordance with the most commonly followed guidelines found in the literature (see [2] and references therein). The sparsity of the reservoir matrix in this case is obtained by choosing a degree of connections and dividing that by the reservoir size. Of course, it is also possible to simply choose any value between 0.0 and 1.0 to test behaviors for different sparsity values. In this example, the call to the parameters inside RandSparseReservoir() was done explicitly to showcase the meaning of each of them, but it is also possible to simply pass the values directly, like so RandSparseReservoir(1.2, 6/300).
The value of input_scaling determines the upper and lower bounds of the uniform distribution of the weights in the WeightedLayer(). Like before, this value can be passed either as an argument or as a keyword argument WeightedLayer(0.1). The value of 0.1 represents the default. The default input layer is the DenseLayer, a fully connected layer. The details of the weighted version can be found in [3], for this example, this version returns the best results.
The reservoir driver represents the dynamics of the reservoir. In the standard ESN definition, these dynamics are obtained through a Recurrent Neural Network (RNN), and this is reflected by calling the RNN driver for the ESN struct. This option is set as the default, and unless there is the need to change parameters, it is not needed. The full equation is the following:
where $α$ represents the leaky coefficient, and tanh can be any activation function. Also, $\textbf{x}$ represents the state vector, $\textbf{u}$ the input data, and $\textbf{W}, \textbf{W}_{\text{in}}$ are the reservoir matrix and input matrix, respectively. The default call to the RNN in the library is the following RNN(;activation_function=tanh, leaky_coefficient=1.0), where the meaning of the parameters is clear from the equation above. Instead of the hyperbolic tangent, any activation function can be used, either leveraging external libraries such as NNlib or creating a custom one.
The final calls are modifications to the states in training or prediction. The default calls, depicted in the example, do not make any modifications to the states. This is the safest bet if one is not sure how these work. The nla_type applies a non-linear algorithm to the states, while the states_type can expand them by concatenating them with the input data, or padding them by concatenating a constant value to all the states. More in depth descriptions of these parameters are given in other examples in the documentation.
Now that the ESN has been created and all the parameters have been explained, it is time to proceed with the training. The full call of the readme example follows this general idea:
The training returns an OutputLayer struct containing the trained output matrix and other informations needed for the prediction. The necessary elements in the train() call are the ESN struct created in the previous step and the target_data, that in this case is the one step ahead evolution of the Lorenz system. The training method chosen in this example is the standard one, so an equivalent way of calling the train function here is output_layer = train(esn, target_data) like the readme basic version. Likewise the default value for the ridge regression parameter is set to zero, so the actual default training is Ordinary Least Squares regression. Other training methods are available and will be explained in following examples.
Once the OutputLayer has been obtained the prediction can be done following this procedure:
The training returns an OutputLayer struct containing the trained output matrix and other needed for the prediction. The necessary elements in the train() call are the ESN struct created in the previous step and the target_data, which in this case is the one step ahead evolution of the Lorenz system. The training method chosen in this example is the standard one, so an equivalent way of calling the train function here is output_layer = train(esn, target_data) like the readme basic version. Likewise, the default value for the ridge regression parameter is set to zero, so the actual default training is Ordinary Least Squares regression. Other training methods are available and will be explained in the following examples.
Once the OutputLayer has been obtained, the prediction can be done following this procedure:
both the training method and the output layer are needed in this call. The number of steps for the prediction must be specified to the Generative method. The output results are given in a matrix.
Saving the states during prediction
While the states are saved in the ESN struct for the training, for the prediction they are not saved by default. To inspect the states it is necessary to pass the boolean keyword argument save_states to the prediction call, in this example using esn(... ; save_states=true). This returns a tuple (output, states) where size(states) = res_size, prediction_len
To inspect the results they can easily be plotted using an external library. In this case Plots is adopted:
both the training method and the output layer are needed in this call. The number of steps for the prediction must be specified in the Generative method. The output results are given in a matrix.
Saving the states during prediction
While the states are saved in the ESN struct for the training, for the prediction they are not saved by default. To inspect the states, it is necessary to pass the boolean keyword argument save_states to the prediction call, in this example using esn(... ; save_states=true). This returns a tuple (output, states) where size(states) = res_size, prediction_len
To inspect the results, they can easily be plotted using an external library. In this case, Plots is adopted:
1Pathak, Jaideep, et al. "Using machine learning to replicate chaotic attractors and calculate Lyapunov exponents from data." Chaos: An Interdisciplinary Journal of Nonlinear Science 27.12 (2017): 121102.
2Lukoševičius, Mantas. "A practical guide to applying echo state networks." Neural networks: Tricks of the trade. Springer, Berlin, Heidelberg, 2012. 659-686.
3Lu, Zhixin, et al. "Reservoir observers: Model-free inference of unmeasured variables in chaotic systems." Chaos: An Interdisciplinary Journal of Nonlinear Science 27.4 (2017): 041102.
Settings
This document was generated with Documenter.jl version 0.27.25 on Thursday 21 September 2023. Using Julia version 1.9.3.
1Pathak, Jaideep, et al. "Using machine learning to replicate chaotic attractors and calculate Lyapunov exponents from data." Chaos: An Interdisciplinary Journal of Nonlinear Science 27.12 (2017): 121102.
2Lukoševičius, Mantas. "A practical guide to applying echo state networks." Neural networks: Tricks of the trade. Springer, Berlin, Heidelberg, 2012. 659-686.
3Lu, Zhixin, et al. "Reservoir observers: Model-free inference of unmeasured variables in chaotic systems." Chaos: An Interdisciplinary Journal of Nonlinear Science 27.4 (2017): 041102.
Settings
This document was generated with Documenter.jl version 1.0.1 on Sunday 24 September 2023. Using Julia version 1.9.3.
diff --git a/dev/general/different_training/index.html b/dev/general/different_training/index.html
index d4526f52..71eae169 100644
--- a/dev/general/different_training/index.html
+++ b/dev/general/different_training/index.html
@@ -1,21 +1,17 @@
-Changing Training Algorithms · ReservoirComputing.jl
Notably Echo State Networks have been trained with Ridge Regression algorithms, but the range of useful algorithms to use is much greater. In this section of the documentation it is possible to explore how to use other training methods to obtain the readout layer. All the methods implemented in ReservoirComputing.jl can be used for all models in the library, not only ESNs. The general workflow illustrated in this section will be based on a dummy RC model my_model = MyModel(...) that need training in order to obtain the readout layer. The training is done following:
In this section it is possible to explore how to properly build the training_algo and all the possible choices available. In the example section of the documentation it will be provided copy-pastable code to better explore the training algorithms and their impact over the model.
The library includes a standard implementation of ridge regression, callable using StandardRidge(regularization_coeff) where the default value for the regularization coefficent is set to zero. This is also the default model called when no model is specified in train(). This makes the default call for traning train(my_model, train_data) use Ordinary Least Squares (OLS) for regression.
Leveraging MLJLinearModels it is possible to expand the choices of linear models used for the training. The wrappers provided are structured in the following way:
struct LinearModel
+Changing Training Algorithms · ReservoirComputing.jl
Notably Echo State Networks have been trained with Ridge Regression algorithms, but the range of useful algorithms to use is much greater. In this section of the documentation, it is possible to explore how to use other training methods to obtain the readout layer. All the methods implemented in ReservoirComputing.jl can be used for all models in the library, not only ESNs. The general workflow illustrated in this section will be based on a dummy RC model my_model = MyModel(...) that needs training to obtain the readout layer. The training is done as follows:
In this section, it is possible to explore how to properly build the training_algo and all the possible choices available. In the example section of the documentation it will be provided copy-pasteable code to better explore the training algorithms and their impact on the model.
The library includes a standard implementation of ridge regression, callable using StandardRidge(regularization_coeff) where the default value for the regularization coefficient is set to zero. This is also the default model called when no model is specified in train(). This makes the default call for training train(my_model, train_data) use Ordinary Least Squares (OLS) for regression.
Leveraging MLJLinearModels it is possible to expand the choices of linear models used for the training. The wrappers provided are structured in the following way:
to call the ridge regression using the MLJLinearModels APIs one can use LinearModel(;regression=LinearRegression). It is also possible to use a specific solver, by calling LinearModel(regression=LinearRegression, solver=Analytical()). For all the available solvers please reref to the MLJLinearModels documentation. To change the regularization coefficient in the ridge example, using for example lambda = 0.1, it is needed to pass it in the regression_kwargs like so LinearModel(;regression=LinearRegression, solver=Analytical(), regression_kwargs=(lambda=lambda)). The nomenclature of the coefficients must follow the MLJLinearModels APIs, using lambda, gamma for LassoRegression and delta, lambda, gamma for HuberRegression. Again, please check the relevant documentation if in doubt. When using MLJLinearModels based regressors do remember to specify using MLJLinearModels.
Another way to obtain the readout layer is possible using Gaussian regression. This is provided through a wrapper of GaussianProcesses structured in the following way:
struct GaussianProcess
+end
to call the ridge regression using the MLJLinearModels APIs, one can use LinearModel(;regression=LinearRegression). It is also possible to use a specific solver, by calling LinearModel(regression=LinearRegression, solver=Analytical()). For all the available solvers, please refer to the MLJLinearModels documentation. To change the regularization coefficient in the ridge example, using for example lambda = 0.1, it is needed to pass it in the regression_kwargs like so LinearModel(;regression=LinearRegression, solver=Analytical(), regression_kwargs=(lambda=lambda)). The nomenclature of the coefficients must follow the MLJLinearModels APIs, using lambda, gamma for LassoRegression and delta, lambda, gamma for HuberRegression. Again, please check the relevant documentation if in doubt. When using MLJLinearModels based regressors, do remember to specify using MLJLinearModels.
Another way to obtain the readout layer is possible using Gaussian regression. This is provided through a wrapper of GaussianProcesses structured in the following way:
struct GaussianProcess
mean
kernel
lognoise
optimize
optimizer
-end
While it is necessary to specify a mean and a kernel, the other defaults are lognoise=-2, optimize=false, optimizer=Optim.LBFGS(). For the choice of means and kernels please refer to the proper documentation, here and here respectively.
Building on the simple example given in the GaussianProcesses documentation it is possible to build an intuition of how to use this algorithms for training ReservoirComputing.jl models.
mZero = MeanZero() #Zero mean function
+end
While it is necessary to specify a mean and a kernel, the other defaults are lognoise=-2, optimize=false, optimizer=Optim.LBFGS(). For the choice of means and kernels, please refer to the proper documentation, here and here, respectively.
Building on the simple example given in the GaussianProcesses documentation, it is possible to build an intuition of how to use these algorithms for training ReservoirComputing.jl models.
mZero = MeanZero() #Zero mean function
kern = SE(0.0,0.0) #Squared exponential kernel (note that hyperparameters are on the log scale)
logObsNoise = -1.0
-gp = GaussianProcess(mZero, kern, lognoise=logObsNoise)
Like in the previous case, if one uses GaussianProcesses based regressors it is necessary to specify using GaussianProcesses. Additionally, if the optimizer chosen is from an external package, i.e. Optim, that package need to be used in the script as well adding using Optim.
Contrary to the LinearModels and GaussianProcesses, no wrappers are needed for support vector regression. By using LIBSVM.jl, LIBSVM wrappers in Julia, it is possible to call both epsilonSVR() or nuSVR() directly in train(). For the full range of kernel provided and the parameters to call we refer the user to the official documentation. Like before, if one intends to use LIBSVM regressors it is necessary to specify using LIBSVM.
Settings
This document was generated with Documenter.jl version 0.27.25 on Thursday 21 September 2023. Using Julia version 1.9.3.
Like in the previous case, if one uses GaussianProcesses based regressors, it is necessary to specify using GaussianProcesses. Additionally, if the optimizer chosen is from an external package, i.e. Optim, that package needs to be used in the script as well by adding using Optim.
Contrary to the LinearModels and GaussianProcesses, no wrappers are needed for support vector regression. By using LIBSVM.jl, LIBSVM wrappers in Julia, it is possible to call both epsilonSVR() or nuSVR() directly in train(). For the full range of kernels provided and the parameters to call, we refer the user to the official documentation. Like before, if one intends to use LIBSVM regressors, it is necessary to specify using LIBSVM.
Settings
This document was generated with Documenter.jl version 1.0.1 on Sunday 24 September 2023. Using Julia version 1.9.3.
diff --git a/dev/general/predictive_generative/index.html b/dev/general/predictive_generative/index.html
index 8b4af5e4..89378cfa 100644
--- a/dev/general/predictive_generative/index.html
+++ b/dev/general/predictive_generative/index.html
@@ -1,6 +1,2 @@
-Generative vs Predictive · ReservoirComputing.jl
The library provides two different methods for prediction denoted as Predictive() and Generative(), following the two major applications of Reservoir Computing models found in the literature. Both these methods are given as argument for the trained model. While copy-pastable example will be provided further on in the documentation it is better the clarify the difference early on to focus more on the library implementation going forward.
In the first method, the user user can use Reservoir Computing models in a similar fashion as standard Machine Learning models. This means using a set of features as input and a set of labels as outputs. In this case the features and labels can be vectors of different dimensions, as $X=\{x_1,...,x_n\} \ x_i \in \mathbb{R}^{N}$ and $Y=\{y_1,...,y_n\} \ y_i \in \mathbb{R}^{M}$ where $X$ is the feature set and $Y$ the label set. Given the difference in dimensionality for the prediction call it will be needed to feed to the function the feature set to be labeled, for example calling Predictive(X) using the set given in this example.
!this allows for one step ahaed or h steps ahaed prediction
The generative method allows the user to extend the forecasting capabilities of the model, letting the predicted results to be fed back in the model to generate the next prediction. By doing so the model is able to run autonomously, without any feature dataset as input. The call for this model needs only the number of steps that the user intend to forecast, for example calling Generative(100) to generate one hundred time steps.
Settings
This document was generated with Documenter.jl version 0.27.25 on Thursday 21 September 2023. Using Julia version 1.9.3.
The library provides two different methods for prediction, denoted as Predictive() and Generative(), following the two major applications of Reservoir Computing models found in the literature. Both of these methods are given as arguments for the trained model. While copy-pasteable example swill be provided further on in the documentation, it is better to clarify the difference early on to focus more on the library implementation going forward.
In the first method, the user can use Reservoir Computing models similarly as standard Machine Learning models. This means using a set of features as input and a set of labels as outputs. In this case, the features and labels can be vectors of different dimensions, as $X=\{x_1,...,x_n\} \ x_i \in \mathbb{R}^{N}$ and $Y=\{y_1,...,y_n\} \ y_i \in \mathbb{R}^{M}$ where $X$ is the feature set and $Y$ the label set. Given the difference in dimensionality for the prediction call, it will be needed to feed to the function the feature set to be labeled, for example by calling Predictive(X) using the set given in this example.
!this allows for one step ahead or h steps ahead prediction.
The generative method allows the user to extend the forecasting capabilities of the model, letting the predicted results to be fed back into the model to generate the next prediction. By doing so, the model can run autonomously, without any feature dataset as input. The call for this model needs only the number of steps that the user intends to forecast, for example calling Generative(100) to generate one hundred time steps.
Settings
This document was generated with Documenter.jl version 1.0.1 on Sunday 24 September 2023. Using Julia version 1.9.3.
diff --git a/dev/general/states_variation/index.html b/dev/general/states_variation/index.html
index 474f3e5e..39eff551 100644
--- a/dev/general/states_variation/index.html
+++ b/dev/general/states_variation/index.html
@@ -1,9 +1,5 @@
-Altering States · ReservoirComputing.jl
In every ReservoirComputing model is posible to perform some alteration on the states in the training stage. Depending on the chosen modification this can improve the results for the prediction. Or more simply they can be used to reproduce results in the literature. The alterations are divided in two possibilities: the first concerns padding or extending the states, the second concerns non linear algorithms performed over the states.
Extending the states means appending to them the corresponding input values. If $\textbf{x}(t)$ is the reservoir state at time t corresponding to the input $\textbf{u}(t)$ the extended state will be represented as $[\textbf{x}(t); \textbf{u}(t)]$ where $[;]$ is intended as vertical concatenation. This procedure is, for example, used in Jaeger's Scholarpedia description of Echo State Networks. The extension of the states can be obtained in every ReservoirComputing.jl model by using the keyword argument states_type and calling the method ExtendedStates(). No argument is needed.
Padding the states is appending a constant value, 1.0 for example, to each state. Using the notation introduced before we can define the padded states as $[\textbf{x}(t); 1.0]$. This approach is detailed in the seminal guide to Echo State Networks by Mantas Lukoševičius. By using the keyword argument states_type the user can call the method PaddedStates(padding) where padding represents the value that will be concatenated to the states. As default the value is set to unity, so the majority of times calling PaddedStates() will suffice.
Altough not found easily in the literature, it is also possible to pad the extended states by using the method PaddedExtendedStates(padding) that has unity as padding default as well.
Of course it is also possible to not apport any of these changes to the states by calling StandardStates(). This is also the default choice for the states.
First introduced in [1] and expanded in [2] these are nonlinear combinations of the columns of the matrix states. There are three such algorithms implemented. Using the keyword argument nla_type it is possible to choose in every model in ReservoirComputing.jl the specific non linear algorithm to use. The defualt value is set to NLADefault(), where no non linear algorithm takes place.
Following the nomenclature used in [2] the algorithms can be called as NLAT1(), NLAT2() and NLAT3(). To better explain what they do, let $\textbf{x}_{i, j}$ be elements of the states matrix, with $i=1,...,T \ j=1,...,N$ where $T$ is the length of the training and $N$ is the reservoir size.
In every ReservoirComputing model is possible to perform some alteration on the states in the training stage. Depending on the chosen modification, this can improve the results of the prediction. Or more simply, they can be used to reproduce results in the literature. The alterations are divided into two possibilities: the first concerns padding or extending the states, and the second concerns non-linear algorithms performed over the states.
Extending the states means appending to them the corresponding input values. If $\textbf{x}(t)$ is the reservoir state at time t corresponding to the input $\textbf{u}(t)$ the extended state will be represented as $[\textbf{x}(t); \textbf{u}(t)]$ where $[;]$ is intended as vertical concatenation. This procedure is, for example, used in Jaeger's Scholarpedia description of Echo State Networks. The extension of the states can be obtained in every ReservoirComputing.jl model by using the keyword argument states_type and calling the method ExtendedStates(). No argument is needed.
Padding the states means appending a constant value, 1.0 for example, to each state. Using the notation introduced before, we can define the padded states as $[\textbf{x}(t); 1.0]$. This approach is detailed in the seminal guide to Echo State Networks by Mantas Lukoševičius. By using the keyword argument states_type the user can call the method PaddedStates(padding) where padding represents the value that will be concatenated to the states. As default, the value is set to unity, so the majority of the time, calling PaddedStates() will suffice.
Though not easily found in the literature, it is also possible to pad the extended states by using the method PaddedExtendedStates(padding) that has unity as padding default as well.
Of course, it is also possible to not apport any of these changes to the states by calling StandardStates(). This is also the default choice for the states.
First introduced in [1] and expanded in [2] these are nonlinear combinations of the columns of the matrix states. There are three such algorithms implemented. Using the keyword argument nla_type it is possible to choose in every model in ReservoirComputing.jl the specific non-linear algorithm to use. The default value is set to NLADefault(), where no non-linear algorithm takes place.
Following the nomenclature used in [2], the algorithms can be called as NLAT1(), NLAT2() and NLAT3(). To better explain what they do, let $\textbf{x}_{i, j}$ be elements of the state matrix, with $i=1,...,T \ j=1,...,N$ where $T$ is the length of the training and $N$ is the reservoir size.
\[\tilde{\textbf{x}}_{i,j} = \textbf{x}_{i,j-1} \times \textbf{x}_{i,j-2} \ \ \text{if \textit{j} > 1 is odd} \\
\tilde{\textbf{x}}_{i,j} = \textbf{x}_{i,j} \ \ \text{if \textit{j} is 1 or even}\]
NLAT3
\[\tilde{\textbf{x}}_{i,j} = \textbf{x}_{i,j-1} \times \textbf{x}_{i,j+1} \ \ \text{if \textit{j} > 1 is odd} \\
-\tilde{\textbf{x}}_{i,j} = \textbf{x}_{i,j} \ \ \text{if \textit{j} is 1 or even}\]
1Pathak, Jaideep, et al. "Using machine learning to replicate chaotic attractors and calculate Lyapunov exponents from data." Chaos: An Interdisciplinary Journal of Nonlinear Science 27.12 (2017): 121102.
2Chattopadhyay, Ashesh, Pedram Hassanzadeh, and Devika Subramanian. "Data-driven predictions of a multiscale Lorenz 96 chaotic system using machine-learning methods: reservoir computing, artificial neural network, and long short-term memory network." Nonlinear Processes in Geophysics 27.3 (2020): 373-389.
Settings
This document was generated with Documenter.jl version 0.27.25 on Thursday 21 September 2023. Using Julia version 1.9.3.
+\tilde{\textbf{x}}_{i,j} = \textbf{x}_{i,j} \ \ \text{if \textit{j} is 1 or even}\]
1Pathak, Jaideep, et al. "Using machine learning to replicate chaotic attractors and calculate Lyapunov exponents from data." Chaos: An Interdisciplinary Journal of Nonlinear Science 27.12 (2017): 121102.
2Chattopadhyay, Ashesh, Pedram Hassanzadeh, and Devika Subramanian. "Data-driven predictions of a multiscale Lorenz 96 chaotic system using machine-learning methods: reservoir computing, artificial neural network, and long short-term memory network." Nonlinear Processes in Geophysics 27.3 (2020): 373-389.
Settings
This document was generated with Documenter.jl version 1.0.1 on Sunday 24 September 2023. Using Julia version 1.9.3.
ReservoirComputing.jl provides an efficient, modular and easy to use implementation of Reservoir Computing models such as Echo State Networks (ESNs). Reservoir Computing (RC) is an umbrella term used to describe a family of models such as ESNs and Liquid State Machines (LSMs). The key concept is to expand the input data into a higher dimension and use regression in order to train the model; in some ways Reservoir Computers can be considered similar to kernel methods.
Introductory material
This library assumes some basic knowledge of Reservoir Computing. For a good introduction, we suggest the following papers: the first two are the seminal papers about ESN and LSM, the others are in-depth review papers that should cover all the needed information. For the majority of the algorithms implemented in this library we cited in the documentation the original work introducing them. If you ever are in doubt about about a method or a function just type ? function in the Julia REPL to read the relevant notes.
Jaeger, Herbert: The “echo state” approach to analyzing and training recurrent neural networks-with an erratum note.
Maass W, Natschläger T, Markram H: Real-time computing without stable states: a new framework for neural computation based on perturbations.
Lukoševičius, Mantas: A practical guide to applying echo state networks." Neural networks: Tricks of the trade.
Lukoševičius, Mantas, and Herbert Jaeger: Reservoir computing approaches to recurrent neural network training.
Performance tip
For faster computations on the CPU it is suggested to add using MKL to the script. For clarity's sake this library will not be indicated under every example in the documentation.
LIBSVM.AbstractSVR: a direct call of LIBSVM regression methods.
Also provided are two different ways of doing predictions using RC:
Generative: the algorithm uses the prediction of the model in the previous step to continue the prediction. It only needs the number of steps as input.
Predictive: standard Machine Learning type of prediction. Given the features the RC model will return the label/prediction.
It is possible to modify the RC obtained states in the training and prediction step using the following:
StandardStates: default choice, no changes will be made to the states.
ExtendedStates: the states are extended using a vertical concatenation with the input data.
PaddedStates: the states are padded using a vertical concatenation with the choosing padding value
PaddedExtendedStates: a combination of the first two. First the states are extended and then padded.
In addition another modification is possible through the choice of non linear algorithms:
NLADefault: default choice, no changes will be made to the states.
The package provides also an implementation of Reservoir Computing models based on one dimensional Cellular Automata through the RECA call. For the moment the only input encoding available (an input encoding plays a similar role to the input matrix for ESNs) is a random mapping, called through RandomMapping.
All the training methods described above can be used, as well as all the modifications to the states. Both prediction methods are also possible in theory, although in the literature only Predictive tasks have been explored.
Contributions are very welcomed! Some interesting variation of RC models are posted in the issues, but everyone is free to just post relevant papers that could fit the scope of the library. Help with the documentation, providing new examples or application cases is also really important and appreciated. Everything that can make the package a little better is a great contribution, no matter how small. The API section of the documentation provides a more in depth look into how things work and are connected, so that is a good place to start exploring more the library. For every doubt that cannot be expressed in issues please feel free to contact any of the lead developers on Slack or by email.
ReservoirComputing.jl provides an efficient, modular, and easy to use implementation of Reservoir Computing models such as Echo State Networks (ESNs). Reservoir Computing (RC) is an umbrella term used to describe a family of models such as ESNs and Liquid State Machines (LSMs). The key concept is to expand the input data into a higher dimension and use regression to train the model; in some ways, Reservoir Computers can be considered similar to kernel methods.
Introductory material
This library assumes some basic knowledge of Reservoir Computing. For a good introduction, we suggest the following papers: the first two are the seminal papers about ESN and LSM, the others are in-depth review papers that should cover all the needed information. For the majority of the algorithms implemented in this library, we cited in the documentation the original work introducing them. If you ever have doubts about a method or a function, just type ? function in the Julia REPL to read the relevant notes.
Jaeger, Herbert: The “echo state” approach to analyzing and training recurrent neural networks-with an erratum note.
Maass W, Natschläger T, Markram H: Real-time computing without stable states: a new framework for neural computation based on perturbations.
Lukoševičius, Mantas: A practical guide to applying echo state networks." Neural networks: Tricks of the trade.
Lukoševičius, Mantas, and Herbert Jaeger: Reservoir computing approaches to recurrent neural network training.
Performance tip
For faster computations on the CPU, it is suggested to add using MKL to the script. For clarity's sake, this library will not be indicated under every example in the documentation.
LIBSVM.AbstractSVR: a direct call of LIBSVM regression methods.
Also provided are two different ways of making predictions using RC:
Generative: the algorithm uses the prediction of the model in the previous step to continue the prediction. It only needs the number of steps as input.
Predictive: standard Machine Learning type of prediction. Given the features, the RC model will return the label/prediction.
It is possible to modify the RC obtained states in the training and prediction steps using the following:
StandardStates: default choice, no changes will be made to the states.
ExtendedStates: the states are extended using a vertical concatenation, with the input data.
PaddedStates: the states are padded using a vertical concatenation with the chosen padding value.
PaddedExtendedStates: a combination of the first two. First, the states are extended and then padded.
In addition, another modification is possible through the choice of non-linear algorithms:
NLADefault: default choice, no changes will be made to the states.
The package provides also an implementation of Reservoir Computing models based on one dimensional Cellular Automata through the RECA call. For the moment, the only input encoding available (an input encoding plays a similar role to the input matrix for ESNs) is a random mapping, called through RandomMapping.
All the training methods described above can be used, as can all the modifications to the states. Both prediction methods are also possible in theory, although in the literature only Predictive tasks have been explored.
If you use this library in your work, please cite:
@article{JMLR:v23:22-0611,
author = {Francesco Martinuzzi and Chris Rackauckas and Anas Abdelrehim and Miguel D. Mahecha and Karin Mora},
title = {ReservoirComputing.jl: An Efficient and Modular Library for Reservoir Computing Models},
journal = {Journal of Machine Learning Research},
@@ -12,19 +9,18 @@
number = {288},
pages = {1--8},
url = {http://jmlr.org/papers/v23/22-0611.html}
-}
Reservoir Computing based on Elementary Cellular Automata (ECA) has been recently introduced. Dubbed as ReCA [1][2] it proposed the advantage of storing the reservoir states as binary data. Less parameter tuning represents another advantage of this model. The architecture implemented in ReservoirComputing.jl follows [3] which build over the original implementation, improving the results. It is strongly suggested to go through the paper to get a solid understanding of the model before delving into experimentation with the code.
To showcase how to use this models this page illustrates the performance of ReCA in the 5 bit memory task [4]. The script for the example and companion data can be found here.
Reservoir Computing based on Elementary Cellular Automata (ECA) has been recently introduced. Dubbed as ReCA [1][2] it proposed the advantage of storing the reservoir states as binary data. Less parameter tuning represents another advantage of this model. The architecture implemented in ReservoirComputing.jl follows [3] which builds on top of the original implementation, improving the results. It is strongly suggested to go through the paper to get a solid understanding of the model before delving into experimentation with the code.
To showcase how to use these models, this page illustrates the performance of ReCA in the 5 bit memory task [4]. The script for the example and companion data can be found here.
To use a ReCA model it is necessary to define the rule one intends to use. To do so ReservoirComputing.jl leverages CellularAutomata.jl that needs to be called as well to define the RECA struct:
To use a ReCA model, it is necessary to define the rule one intends to use. To do so, ReservoirComputing.jl leverages CellularAutomata.jl that needs to be called as well to define the RECA struct:
using ReservoirComputing, CellularAutomata
-ca = DCA(90)
The prediction in this case will be a Predictive() with the input data equal to the training data. In addition, to test the 5 bit memory task, a conversion from Float to Bool is necessary (at the moment we are aware of a bug that doesn't allow to input boolean data to the RECA models):
The prediction in this case will be a Predictive() with the input data equal to the training data. In addition, to test the 5 bit memory task, a conversion from Float to Bool is necessary (at the moment, we are aware of a bug that doesn't allow boolean input data to the RECA models):
This document was generated with Documenter.jl version 0.27.25 on Thursday 21 September 2023. Using Julia version 1.9.3.
diff --git a/dev/search_index.js b/dev/search_index.js
index 781845e6..edc0c787 100644
--- a/dev/search_index.js
+++ b/dev/search_index.js
@@ -1,3 +1,3 @@
var documenterSearchIndex = {"docs":
-[{"location":"general/predictive_generative/#Generative-vs-Predictive","page":"Generative vs Predictive","title":"Generative vs Predictive","text":"","category":"section"},{"location":"general/predictive_generative/","page":"Generative vs Predictive","title":"Generative vs Predictive","text":"The library provides two different methods for prediction denoted as Predictive() and Generative(), following the two major applications of Reservoir Computing models found in the literature. Both these methods are given as argument for the trained model. While copy-pastable example will be provided further on in the documentation it is better the clarify the difference early on to focus more on the library implementation going forward.","category":"page"},{"location":"general/predictive_generative/#Predictive","page":"Generative vs Predictive","title":"Predictive","text":"","category":"section"},{"location":"general/predictive_generative/","page":"Generative vs Predictive","title":"Generative vs Predictive","text":"In the first method, the user user can use Reservoir Computing models in a similar fashion as standard Machine Learning models. This means using a set of features as input and a set of labels as outputs. In this case the features and labels can be vectors of different dimensions, as X=x_1x_n x_i in mathbbR^N and Y=y_1y_n y_i in mathbbR^M where X is the feature set and Y the label set. Given the difference in dimensionality for the prediction call it will be needed to feed to the function the feature set to be labeled, for example calling Predictive(X) using the set given in this example.","category":"page"},{"location":"general/predictive_generative/","page":"Generative vs Predictive","title":"Generative vs Predictive","text":"!this allows for one step ahaed or h steps ahaed prediction","category":"page"},{"location":"general/predictive_generative/#Generative","page":"Generative vs Predictive","title":"Generative","text":"","category":"section"},{"location":"general/predictive_generative/","page":"Generative vs Predictive","title":"Generative vs Predictive","text":"The generative method allows the user to extend the forecasting capabilities of the model, letting the predicted results to be fed back in the model to generate the next prediction. By doing so the model is able to run autonomously, without any feature dataset as input. The call for this model needs only the number of steps that the user intend to forecast, for example calling Generative(100) to generate one hundred time steps.","category":"page"},{"location":"reca_tutorials/reca/#Reservoir-Computing-using-Cellular-Automata","page":"Reservoir Computing with Cellular Automata","title":"Reservoir Computing using Cellular Automata","text":"","category":"section"},{"location":"reca_tutorials/reca/","page":"Reservoir Computing with Cellular Automata","title":"Reservoir Computing with Cellular Automata","text":"Reservoir Computing based on Elementary Cellular Automata (ECA) has been recently introduced. Dubbed as ReCA [1][2] it proposed the advantage of storing the reservoir states as binary data. Less parameter tuning represents another advantage of this model. The architecture implemented in ReservoirComputing.jl follows [3] which build over the original implementation, improving the results. It is strongly suggested to go through the paper to get a solid understanding of the model before delving into experimentation with the code.","category":"page"},{"location":"reca_tutorials/reca/","page":"Reservoir Computing with Cellular Automata","title":"Reservoir Computing with Cellular Automata","text":"To showcase how to use this models this page illustrates the performance of ReCA in the 5 bit memory task [4]. The script for the example and companion data can be found here.","category":"page"},{"location":"reca_tutorials/reca/#bit-memory-task","page":"Reservoir Computing with Cellular Automata","title":"5 bit memory task","text":"","category":"section"},{"location":"reca_tutorials/reca/","page":"Reservoir Computing with Cellular Automata","title":"Reservoir Computing with Cellular Automata","text":"The data can be read as follows:","category":"page"},{"location":"reca_tutorials/reca/","page":"Reservoir Computing with Cellular Automata","title":"Reservoir Computing with Cellular Automata","text":"using DelimitedFiles\n\ninput = readdlm(\"./5bitinput.txt\", ',', Float32)\noutput = readdlm(\"./5bitoutput.txt\", ',', Float32)","category":"page"},{"location":"reca_tutorials/reca/","page":"Reservoir Computing with Cellular Automata","title":"Reservoir Computing with Cellular Automata","text":"To use a ReCA model it is necessary to define the rule one intends to use. To do so ReservoirComputing.jl leverages CellularAutomata.jl that needs to be called as well to define the RECA struct:","category":"page"},{"location":"reca_tutorials/reca/","page":"Reservoir Computing with Cellular Automata","title":"Reservoir Computing with Cellular Automata","text":"using ReservoirComputing, CellularAutomata\n\nca = DCA(90)","category":"page"},{"location":"reca_tutorials/reca/","page":"Reservoir Computing with Cellular Automata","title":"Reservoir Computing with Cellular Automata","text":"To define the ReCA model it suffices to call:","category":"page"},{"location":"reca_tutorials/reca/","page":"Reservoir Computing with Cellular Automata","title":"Reservoir Computing with Cellular Automata","text":"reca = RECA(input, ca; \n generations = 16,\n input_encoding = RandomMapping(16, 40))","category":"page"},{"location":"reca_tutorials/reca/","page":"Reservoir Computing with Cellular Automata","title":"Reservoir Computing with Cellular Automata","text":"After the training can be performed with the chosen method. ","category":"page"},{"location":"reca_tutorials/reca/","page":"Reservoir Computing with Cellular Automata","title":"Reservoir Computing with Cellular Automata","text":"output_layer = train(reca, output, StandardRidge(0.00001))","category":"page"},{"location":"reca_tutorials/reca/","page":"Reservoir Computing with Cellular Automata","title":"Reservoir Computing with Cellular Automata","text":"The prediction in this case will be a Predictive() with the input data equal to the training data. In addition, to test the 5 bit memory task, a conversion from Float to Bool is necessary (at the moment we are aware of a bug that doesn't allow to input boolean data to the RECA models):","category":"page"},{"location":"reca_tutorials/reca/","page":"Reservoir Computing with Cellular Automata","title":"Reservoir Computing with Cellular Automata","text":"prediction = reca(Predictive(input), output_layer)\nfinal_pred = convert(AbstractArray{Float32}, prediction .> 0.5)\n\nfinal_pred == output","category":"page"},{"location":"reca_tutorials/reca/","page":"Reservoir Computing with Cellular Automata","title":"Reservoir Computing with Cellular Automata","text":"[1]: Yilmaz, Ozgur. \"Reservoir computing using cellular automata.\" arXiv preprint arXiv:1410.0162 (2014).","category":"page"},{"location":"reca_tutorials/reca/","page":"Reservoir Computing with Cellular Automata","title":"Reservoir Computing with Cellular Automata","text":"[2]: Margem, Mrwan, and Ozgür Yilmaz. \"An experimental study on cellular automata reservoir in pathological sequence learning tasks.\" (2017).","category":"page"},{"location":"reca_tutorials/reca/","page":"Reservoir Computing with Cellular Automata","title":"Reservoir Computing with Cellular Automata","text":"[3]: Nichele, Stefano, and Andreas Molund. \"Deep reservoir computing using cellular automata.\" arXiv preprint arXiv:1703.02806 (2017).","category":"page"},{"location":"reca_tutorials/reca/","page":"Reservoir Computing with Cellular Automata","title":"Reservoir Computing with Cellular Automata","text":"[4]: Hochreiter, Sepp, and Jürgen Schmidhuber. \"Long short-term memory.\" Neural computation 9.8 (1997): 1735-1780.","category":"page"},{"location":"esn_tutorials/different_drivers/#Using-Different-Reservoir-Drivers","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"","category":"section"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"While the original implementation of the Echo State Network implemented the model using the equations of Recurrent Neural Networks to obtain non linearity in the reservoir, other variations have been proposed in recent years. More specifically the different drivers implemented in ReservoirComputing.jl are the multiple activation function RNN MRNN() and the Gated Recurrent Unit GRU(). To change them it suffice to give the chosen method to the ESN keyword argument reservoir_driver. In this section some example of their usage will be given, as well as a quick introduction to their equations.","category":"page"},{"location":"esn_tutorials/different_drivers/#Multiple-Activation-Function-RNN","page":"Using Different Reservoir Drivers","title":"Multiple Activation Function RNN","text":"","category":"section"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"Based on the double activation function ESN (DAFESN) proposed in [1], the Multiple Activation Function ESN expands the idea and allows a custom number of activation functions to be used in the reservoir dynamics. This can be thought as a linear combination of multiple activation functions with corresponding parameters.","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"mathbfx(t+1) = (1-alpha)mathbfx(t) + lambda_1 f_1(mathbfWmathbfx(t)+mathbfW_inmathbfu(t)) + dots + lambda_D f_D(mathbfWmathbfx(t)+mathbfW_inmathbfu(t))","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"where D is the number of activation function and respective parameters chosen.","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"The method to call to use the mutliple activation function ESN is MRNN(activation_function, leaky_coefficient, scaling_factor). The arguments can be used as both args or kwargs. activation_function and scaling_factor have to be vectors (or tuples) containing the chosen activation functions and respective scaling factors (f_1f_D and lambda_1lambda_D following the nomenclature introduced above). The leaky_coefficient represents alpha and it is a single value. ","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"Starting the example, the data used is based on the following function based on the DAFESN paper [1]. A full script of the example is available here. This example was run on Julia v1.7.2.","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"u(t) = sin(t)+sin(0.51*t)+sin(0.22*t)+sin(0.1002*t)+sin(0.05343*t)","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"For this example the type of prediction will be one step ahead. The metric used to assure a good prediction is going to be the normalized root-mean-square deviation rmsd from StatsBase. Like in the other examples first it is needed to gather the data:","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"train_len = 3000\npredict_len = 2000\nshift = 1\n\ndata = u.(collect(0.0:0.01:500))\ntraining_input = reduce(hcat, data[shift:shift+train_len-1])\ntraining_target = reduce(hcat, data[shift+1:shift+train_len])\ntesting_input = reduce(hcat, data[shift+train_len:shift+train_len+predict_len-1])\ntesting_target = reduce(hcat, data[shift+train_len+1:shift+train_len+predict_len])","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"In order to follow the paper more closely it is necessary to define a couple of activation functions. The numbering of them follows the ones in the paper. Of course one can also use any function, custom defined, available in the base language or any activation function from NNlib.","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"f2(x) = (1-exp(-x))/(2*(1+exp(-x)))\nf3(x) = (2/pi)*atan((pi/2)*x)\nf4(x) = x/sqrt(1+x*x)","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"It is now possible to build different drivers, using the parameters suggested by the paper. Also in this instance the numbering follows the test cases of the paper. In the end a simple for loop is implemented to compare the different drivers and activation functions.","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"using ReservoirComputing, Random, StatsBase\n\n#fix seed for reproducibility\nRandom.seed!(42)\n\n#baseline case with RNN() driver. Parameter given as args\nbase_case = RNN(tanh, 0.85)\n\n#MRNN() test cases\n#Parameter given as kwargs\ncase3 = MRNN(activation_function=[tanh, f2], \n leaky_coefficient=0.85, \n scaling_factor=[0.5, 0.3])\n\n#Parameter given as kwargs\ncase4 = MRNN(activation_function=[tanh, f3], \n leaky_coefficient=0.9, \n scaling_factor=[0.45, 0.35])\n\n#Parameter given as args\ncase5 = MRNN([tanh, f4], 0.9, [0.43, 0.13])\n\n#tests\ntest_cases = [base_case, case3, case4, case5]\nfor case in test_cases\n esn = ESN(training_input,\n input_layer = WeightedLayer(scaling=0.3),\n reservoir = RandSparseReservoir(100, radius=0.4),\n reservoir_driver = case,\n states_type = ExtendedStates())\n wout = train(esn, training_target, StandardRidge(10e-6))\n output = esn(Predictive(testing_input), wout)\n println(rmsd(testing_target, output, normalize=true))\nend","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"In this example it is also possible to observe the input of parameters to the methods RNN() MRNN() both by argument and by keyword argument.","category":"page"},{"location":"esn_tutorials/different_drivers/#Gated-Recurrent-Unit","page":"Using Different Reservoir Drivers","title":"Gated Recurrent Unit","text":"","category":"section"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"Gated Recurrent Units (GRUs) [2] have been proposed in more recent years with the intent of limiting notable problems of RNNs, like the vanishing gradient. This change in the underlying equations can be easily transported in the Reservoir Computing paradigm, switching the RNN equations in the reservoir with the GRU equations. This approach has been explored in [3] and [4]. Different variations of GRU have been proposed [5][6]; this section is subdivided into different sections that go in detail about the governing equations and the implementation of them into ReservoirComputing.jl. Like before, to access the GRU reservoir driver it suffice to change the reservoir_diver keyword argument for ESN with GRU(). All the variations that are going to be presented can be used in this package by leveraging the keyword argument variant in the method GRU() and specifying the chosen variant: FullyGated() or Minimal(). Other variations are possible modifying the inner layers and reservoirs. The default is set to the standard version FullyGated(). The first section will go in more detail about the default of the GRU() method, and the following ones will refer to it to minimize repetitions. This example was run on Julia v1.7.2.","category":"page"},{"location":"esn_tutorials/different_drivers/#Standard-GRU","page":"Using Different Reservoir Drivers","title":"Standard GRU","text":"","category":"section"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"The equations for the standard GRU are as follows:","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"mathbfr(t) = sigma (mathbfW^r_textinmathbfu(t)+mathbfW^rmathbfx(t-1)+mathbfb_r) \nmathbfz(t) = sigma (mathbfW^z_textinmathbfu(t)+mathbfW^zmathbfx(t-1)+mathbfb_z) \ntildemathbfx(t) = texttanh(mathbfW_inmathbfu(t)+mathbfW(mathbfr(t) odot mathbfx(t-1))+mathbfb) \nmathbfx(t) = mathbfz(t) odot mathbfx(t-1)+(1-mathbfz(t)) odot tildemathbfx(t)","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"Going over the GRU keyword argument it will be explained how to feed the desired input to the model. ","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"activation_function is a vector with default values [NNlib.sigmoid, NNlib.sigmoid, tanh]. This argument controls the activation functions of the GRU, going from top to bottom. Changing the first element corresponds in changing the activation function for mathbfr(t) and so on.\ninner_layer is a vector with default values fill(DenseLayer(), 2). This keyword argument controls the mathbfW_textins going from top to bottom like before.\nreservoir is a vector with default value fill(RandSparseReservoir(), 2). In a similar fashion to inner_layer, this keyword argument controls the reservoir matrix construction in a top to bottom order.\nbias is again a vector with default value fill(DenseLayer(), 2). It is meant to control the mathbfbs, going as usual from top to bottom.\nvariant as already illustrated controls the GRU variant. The default value is set to FullyGated().","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"It is important to notice that inner_layer and reservoir control every layer except mathbfW_in and mathbfW and mathbfb. These arguments are given as input to the ESN() call as input_layer, reservoir and bias. ","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"The following sections are going to illustrate the variations of the GRU architecture and how to obtain them in ReservoirComputing.jl","category":"page"},{"location":"esn_tutorials/different_drivers/#Type-1","page":"Using Different Reservoir Drivers","title":"Type 1","text":"","category":"section"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"The first variation of the GRU is dependent only on the previous hidden state and the bias:","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"mathbfr(t) = sigma (mathbfW^rmathbfx(t-1)+mathbfb_r) \nmathbfz(t) = sigma (mathbfW^zmathbfx(t-1)+mathbfb_z) ","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"To obtain this variation it will suffice to set inner_layer = fill(NullLayer(), 2) and leaving the variant = FullyGated().","category":"page"},{"location":"esn_tutorials/different_drivers/#Type-2","page":"Using Different Reservoir Drivers","title":"Type 2","text":"","category":"section"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"The second variation only depends on the previous hidden state:","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"mathbfr(t) = sigma (mathbfW^rmathbfx(t-1)) \nmathbfz(t) = sigma (mathbfW^zmathbfx(t-1)) ","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"Similarly to before, to obtain this variation it is only needed to set inner_layer = fill(NullLayer(), 2) and bias = fill(NullLayer(), 2) while keeping variant = FullyGated().","category":"page"},{"location":"esn_tutorials/different_drivers/#Type-3","page":"Using Different Reservoir Drivers","title":"Type 3","text":"","category":"section"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"The final variation before the minimal one depends only on the biases","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"mathbfr(t) = sigma (mathbfb_r) \nmathbfz(t) = sigma (mathbfb_z) ","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"This means that it is only needed to set inner_layer = fill(NullLayer(), 2) and reservoir = fill(NullReservoir(), 2) while keeping variant = FullyGated().","category":"page"},{"location":"esn_tutorials/different_drivers/#Minimal","page":"Using Different Reservoir Drivers","title":"Minimal","text":"","category":"section"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"The minimal GRU variation merges two gates into one:","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"mathbff(t) = sigma (mathbfW^f_textinmathbfu(t)+mathbfW^fmathbfx(t-1)+mathbfb_f) \ntildemathbfx(t) = texttanh(mathbfW_inmathbfu(t)+mathbfW(mathbff(t) odot mathbfx(t-1))+mathbfb) \nmathbfx(t) = (1-mathbff(t)) odot mathbfx(t-1) + mathbff(t) odot tildemathbfx(t)","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"This variation can be obtained by setting variation=Minimal(). The inner_layer, reservoir and bias kwargs this time are not vectors, but must be defined like, for example inner_layer = DenseLayer() or reservoir = SparseDenseReservoir().","category":"page"},{"location":"esn_tutorials/different_drivers/#Examples","page":"Using Different Reservoir Drivers","title":"Examples","text":"","category":"section"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"To showcase the use of the GRU() method this section will only illustrate the standard FullyGated() version. The full script for this example with the data can be found here. ","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"The data used for this example is the Santa Fe laser dataset [7] retrieved from here. The data is split to account for a next step prediction.","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"using DelimitedFiles\n\ndata = reduce(hcat, readdlm(\"./data/santafe_laser.txt\"))\n\ntrain_len = 5000\npredict_len = 2000\n\ntraining_input = data[:, 1:train_len]\ntraining_target = data[:, 2:train_len+1]\ntesting_input = data[:,train_len+1:train_len+predict_len]\ntesting_target = data[:,train_len+2:train_len+predict_len+1]","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"The construction of the ESN proceeds as usual. ","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"using ReservoirComputing, Random\n\nres_size = 300\nres_radius = 1.4\n\nRandom.seed!(42)\nesn = ESN(training_input; \n reservoir = RandSparseReservoir(res_size, radius=res_radius),\n reservoir_driver = GRU())","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"The default inner reservoir and input layer for the GRU are the same defaults for the reservoir and input_layer of the ESN. One can use the explicit call if they choose so.","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"gru = GRU(reservoir=[RandSparseReservoir(res_size), \n RandSparseReservoir(res_size)],\n inner_layer=[DenseLayer(), DenseLayer()])\nesn = ESN(training_input; \n reservoir = RandSparseReservoir(res_size, radius=res_radius),\n reservoir_driver = gru)","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"The training and prediction can proceed as usual:","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"training_method = StandardRidge(0.0)\noutput_layer = train(esn, training_target, training_method)\noutput = esn(Predictive(testing_input), output_layer)","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"The results can be plotted using Plots.jl","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"using Plots\n\nplot([testing_target' output'], label=[\"actual\" \"predicted\"], \n plot_title=\"Santa Fe Laser\",\n titlefontsize=20,\n legendfontsize=12,\n linewidth=2.5,\n xtickfontsize = 12,\n ytickfontsize = 12,\n size=(1080, 720))","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"It is interesting to see a comparison of the GRU driven ESN and the standard RNN driven ESN. Using the same parameters defined before it is possible to do the following","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"using StatsBase\n\nesn_rnn = ESN(training_input; \n reservoir = RandSparseReservoir(res_size, radius=res_radius),\n reservoir_driver = RNN())\n\noutput_layer = train(esn_rnn, training_target, training_method)\noutput_rnn = esn_rnn(Predictive(testing_input), output_layer)\n\nprintln(msd(testing_target, output))\nprintln(msd(testing_target, output_rnn))","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"[1]: Lun, Shu-Xian, et al. \"A novel model of leaky integrator echo state network for time-series prediction.\" Neurocomputing 159 (2015): 58-66.","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"[2]: Cho, Kyunghyun, et al. “Learning phrase representations using RNN encoder-decoder for statistical machine translation.” arXiv preprint arXiv:1406.1078 (2014).","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"[3]: Wang, Xinjie, Yaochu Jin, and Kuangrong Hao. \"A Gated Recurrent Unit based Echo State Network.\" 2020 International Joint Conference on Neural Networks (IJCNN). IEEE, 2020.","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"[4]: Di Sarli, Daniele, Claudio Gallicchio, and Alessio Micheli. \"Gated Echo State Networks: a preliminary study.\" 2020 International Conference on INnovations in Intelligent SysTems and Applications (INISTA). IEEE, 2020.","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"[5]: Dey, Rahul, and Fathi M. Salem. \"Gate-variants of gated recurrent unit (GRU) neural networks.\" 2017 IEEE 60th international midwest symposium on circuits and systems (MWSCAS). IEEE, 2017.","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"[6]: Zhou, Guo-Bing, et al. \"Minimal gated unit for recurrent neural networks.\" International Journal of Automation and Computing 13.3 (2016): 226-234.","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"[7]: Hübner, Uwe, Nimmi B. Abraham, and Carlos O. Weiss. \"Dimensions and entropies of chaotic intensity pulsations in a single-mode far-infrared NH 3 laser.\" Physical Review A 40.11 (1989): 6354.","category":"page"},{"location":"esn_tutorials/lorenz_basic/#Lorenz-System-Forecasting","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"","category":"section"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"This example expands on the readme Lorenz system forecasting to better showcase how to use methods and functions provided in the library for Echo State Networks. Here the prediction method used is Generative, for a more detailed explanation of the differences between Generative and Predictive please refer to the other examples given in the documentation. The full script for this example is available here. This example was run on Julia v1.7.2.","category":"page"},{"location":"esn_tutorials/lorenz_basic/#Generating-the-data","page":"Lorenz System Forecasting","title":"Generating the data","text":"","category":"section"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"Starting off the workflow the first step is to obtain the data. Leveraging OrdinaryDiffEq it is possible to derive the Lorenz system data in the following way:","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"using OrdinaryDiffEq\n\n#define lorenz system\nfunction lorenz!(du,u,p,t)\n du[1] = 10.0*(u[2]-u[1])\n du[2] = u[1]*(28.0-u[3]) - u[2]\n du[3] = u[1]*u[2] - (8/3)*u[3]\nend\n\n#solve and take data\nprob = ODEProblem(lorenz!, [1.0,0.0,0.0], (0.0,200.0))\ndata = solve(prob, ABM54(), dt=0.02)","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"After obtaining the data it is necessary to determine the kind of prediction for the model. Since this example is going to use the Generative prediction type, this means that the target data is foing to be the next step of the input data. In addition it is important to notice that the Lorenz system just obtained presents a transient period that is not representative of the general behavior of the system. This can easily be discarded setting a shift parameter.","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"#determine shift length, training length and prediction length\nshift = 300\ntrain_len = 5000\npredict_len = 1250\n\n#split the data accordingly\ninput_data = data[:, shift:shift+train_len-1]\ntarget_data = data[:, shift+1:shift+train_len]\ntest_data = data[:,shift+train_len+1:shift+train_len+predict_len]","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"It is important to notice that the data needs to be formatted in a matrix with the features as rows and time steps as columns like it is done in this example. This is needed even if the time series consists of single values. ","category":"page"},{"location":"esn_tutorials/lorenz_basic/#Building-the-Echo-State-Network","page":"Lorenz System Forecasting","title":"Building the Echo State Network","text":"","category":"section"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"Once the data is ready it is possible to define the parameters for the ESN and the ESN struct itself. In this example the values from [1] are loosely followed as general guidelines.","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"using ReservoirComputing\n\n#define ESN parameters\nres_size = 300\nres_radius = 1.2\nres_sparsity = 6/300\ninput_scaling = 0.1\n\n#build ESN struct\nesn = ESN(input_data; \n variation = Default(),\n reservoir = RandSparseReservoir(res_size, radius=res_radius, sparsity=res_sparsity),\n input_layer = WeightedLayer(scaling=input_scaling),\n reservoir_driver = RNN(),\n nla_type = NLADefault(),\n states_type = StandardStates())","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"Most of the parameters here chosen mirror the default ones, so a direct call is not necessary. The readme example is identical to this one, except for the explicit call. Going line by line to see what is happening starting from res_size: this value determines the dimensions of the reservoir matrix. In this case a size of 300 has been chosen, so the reservoir matrix is going to be 300 x 300. This is not always the case, since some input layer constructions can modify the dimensions of the reservoir, but in that case everything is taken care of internally. ","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"The res_radius determines the scaling of the spectral radius of the reservoir matrix; a proper scaling is necessary to assure the Echo State Property. The default value in the RandSparseReservoir() method is 1.0 in accordance to the most followed guidelines found in the literature (see [2] and references therein). The sparsity of the reservoir matrix in this case is obtained by choosing a degree of connections and dividing that by the reservoir size. Of course it is also possible to simply choose any value between 0.0 and 1.0 to test behaviors for different sparsity values. In this example the call to the parameters inside RandSparseReservoir() was done explicitly to showcase the meaning of each of them, but it is also possible to simply pass the values directly like so RandSparseReservoir(1.2, 6/300).","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"The value of input_scaling determines the upper and lower bounds of the uniform distribution of the weights in the WeightedLayer(). Like before this value can be passed either as argument or keyword argument WeightedLayer(0.1). The value of 0.1 represents the default. The default input layer is the DenseLayer, a fully connected layer. The details of the weighted version can be found in [3], for this example this version returns the best results.","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"The reservoir driver represents the dynamics of the reservoir. In the standard ESN definition these dynamics are obtained through a Recurrent Neural Network (RNN), and this is reflected by calling the RNN driver for the ESN struct. This option is set as the default and unless there is the need to change parameters it is not needed. The full equation is the following:","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"textbfx(t+1) = (1-alpha)textbfx(t) + alpha cdot texttanh(textbfWtextbfx(t)+textbfW_textintextbfu(t))","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"where α represents the leaky coefficient and tanh can be any activation function. Also textbfx represent the state vector, textbfu the input data and textbfW textbfW_textin are the reservoir matrix and input matrix respectively. The default call to the RNN in the library is the following RNN(;activation_function=tanh, leaky_coefficient=1.0), where the meaning of the parameters is clear from the equation above. Instead og the hyperbolic tangent any activation function can be used, either leveraging external lybraries such as NNlib or creating a custom one. ","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"The final calls are modifications to the states in training or prediction. The default calls, depicted in the example, do not make any modifications to the states. This is the safest bet is one is not sure on how these work. The nla_type applies a non linear algorithm to the states, while the states_type can expand them concatenating them with the input data, or padding them concatenating a constant value to all the states. More in depth descriptions about these parameters are given in other examples in the documentation.","category":"page"},{"location":"esn_tutorials/lorenz_basic/#Training-and-Prediction","page":"Lorenz System Forecasting","title":"Training and Prediction","text":"","category":"section"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"Now that the ESN has been created and all the parameters have been explained it is time to proceed with the training. The full call of the readme example follows this general idea:","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"#define training method\ntraining_method = StandardRidge(0.0)\n\n#obtain output layer\noutput_layer = train(esn, target_data, training_method)","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"The training returns an OutputLayer struct containing the trained output matrix and other informations needed for the prediction. The necessary elements in the train() call are the ESN struct created in the previous step and the target_data, that in this case is the one step ahead evolution of the Lorenz system. The training method chosen in this example is the standard one, so an equivalent way of calling the train function here is output_layer = train(esn, target_data) like the readme basic version. Likewise the default value for the ridge regression parameter is set to zero, so the actual default training is Ordinary Least Squares regression. Other training methods are available and will be explained in following examples. ","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"Once the OutputLayer has been obtained the prediction can be done following this procedure:","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"output = esn(Generative(predict_len), output_layer)","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"both the training method and the output layer are needed in this call. The number of steps for the prediction must be specified to the Generative method. The output results are given in a matrix. ","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"info: Saving the states during prediction\nWhile the states are saved in the ESN struct for the training, for the prediction they are not saved by default. To inspect the states it is necessary to pass the boolean keyword argument save_states to the prediction call, in this example using esn(... ; save_states=true). This returns a tuple (output, states) where size(states) = res_size, prediction_len","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"To inspect the results they can easily be plotted using an external library. In this case Plots is adopted:","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"using Plots, Plots.PlotMeasures\n\nts = 0.0:0.02:200.0\nlorenz_maxlyap = 0.9056\npredict_ts = ts[shift+train_len+1:shift+train_len+predict_len]\nlyap_time = (predict_ts .- predict_ts[1])*(1/lorenz_maxlyap)\n\np1 = plot(lyap_time, [test_data[1,:] output[1,:]], label = [\"actual\" \"predicted\"], \n ylabel = \"x(t)\", linewidth=2.5, xticks=false, yticks = -15:15:15);\np2 = plot(lyap_time, [test_data[2,:] output[2,:]], label = [\"actual\" \"predicted\"], \n ylabel = \"y(t)\", linewidth=2.5, xticks=false, yticks = -20:20:20);\np3 = plot(lyap_time, [test_data[3,:] output[3,:]], label = [\"actual\" \"predicted\"], \n ylabel = \"z(t)\", linewidth=2.5, xlabel = \"max(λ)*t\", yticks = 10:15:40);\n\n\nplot(p1, p2, p3, plot_title = \"Lorenz System Coordinates\", \n layout=(3,1), xtickfontsize = 12, ytickfontsize = 12, xguidefontsize=15, yguidefontsize=15,\n legendfontsize=12, titlefontsize=20)","category":"page"},{"location":"esn_tutorials/lorenz_basic/#Bibliography","page":"Lorenz System Forecasting","title":"Bibliography","text":"","category":"section"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"[1]: Pathak, Jaideep, et al. \"Using machine learning to replicate chaotic attractors and calculate Lyapunov exponents from data.\" Chaos: An Interdisciplinary Journal of Nonlinear Science 27.12 (2017): 121102.","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"[2]: Lukoševičius, Mantas. \"A practical guide to applying echo state networks.\" Neural networks: Tricks of the trade. Springer, Berlin, Heidelberg, 2012. 659-686.","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"[3]: Lu, Zhixin, et al. \"Reservoir observers: Model-free inference of unmeasured variables in chaotic systems.\" Chaos: An Interdisciplinary Journal of Nonlinear Science 27.4 (2017): 041102.","category":"page"},{"location":"esn_tutorials/hybrid/#Hybrid-Echo-State-Networks","page":"Hybrid Echo State Networks","title":"Hybrid Echo State Networks","text":"","category":"section"},{"location":"esn_tutorials/hybrid/","page":"Hybrid Echo State Networks","title":"Hybrid Echo State Networks","text":"Following the idea of giving physical information to machine learning models the hybrid echo state networks [1] try to achieve this results by feeding model data into the ESN. In this example it is explained how to create and leverage such models in ReservoirComputing.jl. The full script for this example is available here. This example was run on Julia v1.7.2.","category":"page"},{"location":"esn_tutorials/hybrid/#Generating-the-data","page":"Hybrid Echo State Networks","title":"Generating the data","text":"","category":"section"},{"location":"esn_tutorials/hybrid/","page":"Hybrid Echo State Networks","title":"Hybrid Echo State Networks","text":"For this example we are going to forecast the Lorenz system. As usual the data is generated leveraging DifferentialEquations.jl:","category":"page"},{"location":"esn_tutorials/hybrid/","page":"Hybrid Echo State Networks","title":"Hybrid Echo State Networks","text":"using DifferentialEquations\n\nu0 = [1.0,0.0,0.0] \ntspan = (0.0,1000.0) \ndatasize = 100000\ntsteps = range(tspan[1], tspan[2], length = datasize) \n\nfunction lorenz(du,u,p,t)\n p = [10.0,28.0,8/3]\n du[1] = p[1]*(u[2]-u[1])\n du[2] = u[1]*(p[2]-u[3]) - u[2]\n du[3] = u[1]*u[2] - p[3]*u[3]\nend\n\node_prob = ODEProblem(lorenz, u0, tspan)\node_sol = solve(ode_prob, saveat = tsteps)\node_data =Array(ode_sol)\n\ntrain_len = 10000\n\ninput_data = ode_data[:, 1:train_len]\ntarget_data = ode_data[:, 2:train_len+1]\ntest_data = ode_data[:, train_len+1:end][:, 1:1000]\n\npredict_len = size(test_data, 2)\ntspan_train = (tspan[1], ode_sol.t[train_len])","category":"page"},{"location":"esn_tutorials/hybrid/#Building-the-Hybrid-Echo-State-Network","page":"Hybrid Echo State Networks","title":"Building the Hybrid Echo State Network","text":"","category":"section"},{"location":"esn_tutorials/hybrid/","page":"Hybrid Echo State Networks","title":"Hybrid Echo State Networks","text":"In order to feed the data to the ESN it is necessary to create a suitable function.","category":"page"},{"location":"esn_tutorials/hybrid/","page":"Hybrid Echo State Networks","title":"Hybrid Echo State Networks","text":"function prior_model_data_generator(u0, tspan, tsteps, model = lorenz)\n prob = ODEProblem(lorenz, u0, tspan) \n sol = Array(solve(prob, saveat = tsteps))\n return sol\nend","category":"page"},{"location":"esn_tutorials/hybrid/","page":"Hybrid Echo State Networks","title":"Hybrid Echo State Networks","text":"Given initial condition, time span and time steps this function returns the data for the chosen model. Now, using the Hybrid method it is possible to input all this information to the model","category":"page"},{"location":"esn_tutorials/hybrid/","page":"Hybrid Echo State Networks","title":"Hybrid Echo State Networks","text":"using ReservoirComputing, Random\nRandom.seed!(42)\n\nhybrid = Hybrid(prior_model_data_generator, u0, tspan_train, train_len)\n\nesn = ESN(input_data,\n reservoir = RandSparseReservoir(300),\n variation = hybrid)","category":"page"},{"location":"esn_tutorials/hybrid/#Training-and-Prediction","page":"Hybrid Echo State Networks","title":"Training and Prediction","text":"","category":"section"},{"location":"esn_tutorials/hybrid/","page":"Hybrid Echo State Networks","title":"Hybrid Echo State Networks","text":"The training and prediction of the Hybrid ESN can proceed as usual:","category":"page"},{"location":"esn_tutorials/hybrid/","page":"Hybrid Echo State Networks","title":"Hybrid Echo State Networks","text":"output_layer = train(esn, target_data, StandardRidge(0.3))\noutput = esn(Generative(predict_len), output_layer)","category":"page"},{"location":"esn_tutorials/hybrid/","page":"Hybrid Echo State Networks","title":"Hybrid Echo State Networks","text":"It is now possible to plot the results, leveraging Plots.jl:","category":"page"},{"location":"esn_tutorials/hybrid/","page":"Hybrid Echo State Networks","title":"Hybrid Echo State Networks","text":"using Plots\nlorenz_maxlyap = 0.9056\npredict_ts = tsteps[train_len+1:train_len+predict_len]\nlyap_time = (predict_ts .- predict_ts[1])*(1/lorenz_maxlyap)\n\np1 = plot(lyap_time, [test_data[1,:] output[1,:]], label = [\"actual\" \"predicted\"], \n ylabel = \"x(t)\", linewidth=2.5, xticks=false, yticks = -15:15:15);\np2 = plot(lyap_time, [test_data[2,:] output[2,:]], label = [\"actual\" \"predicted\"], \n ylabel = \"y(t)\", linewidth=2.5, xticks=false, yticks = -20:20:20);\np3 = plot(lyap_time, [test_data[3,:] output[3,:]], label = [\"actual\" \"predicted\"], \n ylabel = \"z(t)\", linewidth=2.5, xlabel = \"max(λ)*t\", yticks = 10:15:40);\n\n\nplot(p1, p2, p3, plot_title = \"Lorenz System Coordinates\", \n layout=(3,1), xtickfontsize = 12, ytickfontsize = 12, xguidefontsize=15, yguidefontsize=15,\n legendfontsize=12, titlefontsize=20)","category":"page"},{"location":"esn_tutorials/hybrid/#Bibliography","page":"Hybrid Echo State Networks","title":"Bibliography","text":"","category":"section"},{"location":"esn_tutorials/hybrid/","page":"Hybrid Echo State Networks","title":"Hybrid Echo State Networks","text":"[1]: Pathak, Jaideep, et al. \"Hybrid forecasting of chaotic processes: Using machine learning in conjunction with a knowledge-based model.\" Chaos: An Interdisciplinary Journal of Nonlinear Science 28.4 (2018): 041101.","category":"page"},{"location":"esn_tutorials/deep_esn/#Deep-Echo-State-Networks","page":"Deep Echo State Networks","title":"Deep Echo State Networks","text":"","category":"section"},{"location":"esn_tutorials/deep_esn/","page":"Deep Echo State Networks","title":"Deep Echo State Networks","text":"Deep Echo State Network architectures started to gain some traction recently. In this guide we illustrate how it is possible to use ReservoirComputing.jl to build a deep ESN. ","category":"page"},{"location":"esn_tutorials/deep_esn/","page":"Deep Echo State Networks","title":"Deep Echo State Networks","text":"The network implemented in this library is taken from [1]. It works by stacking reservoirs on top of each other, feeding the output on one in the next. The states are obtained by merging all the inner states of the stacked reservoirs. For a more in depth explanation refer to the paper linked above. The full script for this example can be found here. This example was run on Julia v1.7.2.","category":"page"},{"location":"esn_tutorials/deep_esn/#Lorenz-Example","page":"Deep Echo State Networks","title":"Lorenz Example","text":"","category":"section"},{"location":"esn_tutorials/deep_esn/","page":"Deep Echo State Networks","title":"Deep Echo State Networks","text":"For this example we are going to reuse the Lorenz data used in the Lorenz System Forecasting example.","category":"page"},{"location":"esn_tutorials/deep_esn/","page":"Deep Echo State Networks","title":"Deep Echo State Networks","text":"using OrdinaryDiffEq\n\n#define lorenz system\nfunction lorenz!(du,u,p,t)\n du[1] = 10.0*(u[2]-u[1])\n du[2] = u[1]*(28.0-u[3]) - u[2]\n du[3] = u[1]*u[2] - (8/3)*u[3]\nend\n\n#solve and take data\nprob = ODEProblem(lorenz!, [1.0,0.0,0.0], (0.0,200.0))\ndata = solve(prob, ABM54(), dt=0.02)\n\n#determine shift length, training length and prediction length\nshift = 300\ntrain_len = 5000\npredict_len = 1250\n\n#split the data accordingly\ninput_data = data[:, shift:shift+train_len-1]\ntarget_data = data[:, shift+1:shift+train_len]\ntest_data = data[:,shift+train_len+1:shift+train_len+predict_len]","category":"page"},{"location":"esn_tutorials/deep_esn/","page":"Deep Echo State Networks","title":"Deep Echo State Networks","text":"Again, it is important to notice that the data needs to be formatted in a matrix with the features as rows and time steps as columns like it is done in this example. This is needed even if the time series consists of single values. ","category":"page"},{"location":"esn_tutorials/deep_esn/","page":"Deep Echo State Networks","title":"Deep Echo State Networks","text":"The construction of the ESN is also really similar. The only difference is that the reservoir can be fed as an array of reservoirs. ","category":"page"},{"location":"esn_tutorials/deep_esn/","page":"Deep Echo State Networks","title":"Deep Echo State Networks","text":"using ReservoirComputing\n\nreservoirs = [RandSparseReservoir(99, radius=1.1, sparsity=0.1),\n RandSparseReservoir(100, radius=1.2, sparsity=0.1),\n RandSparseReservoir(200, radius=1.4, sparsity=0.1)]\n\nesn = ESN(input_data; \n variation = Default(),\n reservoir = reservoirs,\n input_layer = DenseLayer(),\n reservoir_driver = RNN(),\n nla_type = NLADefault(),\n states_type = StandardStates())","category":"page"},{"location":"esn_tutorials/deep_esn/","page":"Deep Echo State Networks","title":"Deep Echo State Networks","text":"As it is possible to see, different sizes can be chosen for the different reservoirs. The input layer and bias can also be given as vectors, but of course they have to be of the same size of the reservoirs vector. If they are not passed as a vector, the value passed is going to be used for all the layers in the deep ESN.","category":"page"},{"location":"esn_tutorials/deep_esn/","page":"Deep Echo State Networks","title":"Deep Echo State Networks","text":"In addition to using the provided functions for the construction of the layers the user can also choose to build their own matrix, or array of matrices, and feed that into the ESN in the same way.","category":"page"},{"location":"esn_tutorials/deep_esn/","page":"Deep Echo State Networks","title":"Deep Echo State Networks","text":"The training and prediction follows the usual framework:","category":"page"},{"location":"esn_tutorials/deep_esn/","page":"Deep Echo State Networks","title":"Deep Echo State Networks","text":"training_method = StandardRidge(0.0) \noutput_layer = train(esn, target_data, training_method)\n\noutput = esn(Generative(predict_len), output_layer)","category":"page"},{"location":"esn_tutorials/deep_esn/","page":"Deep Echo State Networks","title":"Deep Echo State Networks","text":"Plotting the results:","category":"page"},{"location":"esn_tutorials/deep_esn/","page":"Deep Echo State Networks","title":"Deep Echo State Networks","text":"using Plots\n\nts = 0.0:0.02:200.0\nlorenz_maxlyap = 0.9056\npredict_ts = ts[shift+train_len+1:shift+train_len+predict_len]\nlyap_time = (predict_ts .- predict_ts[1])*(1/lorenz_maxlyap)\n\np1 = plot(lyap_time, [test_data[1,:] output[1,:]], label = [\"actual\" \"predicted\"], \n ylabel = \"x(t)\", linewidth=2.5, xticks=false, yticks = -15:15:15);\np2 = plot(lyap_time, [test_data[2,:] output[2,:]], label = [\"actual\" \"predicted\"], \n ylabel = \"y(t)\", linewidth=2.5, xticks=false, yticks = -20:20:20);\np3 = plot(lyap_time, [test_data[3,:] output[3,:]], label = [\"actual\" \"predicted\"], \n ylabel = \"z(t)\", linewidth=2.5, xlabel = \"max(λ)*t\", yticks = 10:15:40);\n\n\nplot(p1, p2, p3, plot_title = \"Lorenz System Coordinates\", \n layout=(3,1), xtickfontsize = 12, ytickfontsize = 12, xguidefontsize=15, yguidefontsize=15,\n legendfontsize=12, titlefontsize=20)","category":"page"},{"location":"esn_tutorials/deep_esn/","page":"Deep Echo State Networks","title":"Deep Echo State Networks","text":"Note that there is a known bug at the moment with using WeightedLayer as the input layer with the deep ESN. We are in the process of investigating and solving it. The leak coefficient for the reservoirs has to always be the same with the current implementation. This is also something we are actively looking into expanding.","category":"page"},{"location":"esn_tutorials/deep_esn/#Documentation","page":"Deep Echo State Networks","title":"Documentation","text":"","category":"section"},{"location":"esn_tutorials/deep_esn/","page":"Deep Echo State Networks","title":"Deep Echo State Networks","text":"[1]: Gallicchio, Claudio, and Alessio Micheli. \"Deep echo state network (deepesn): A brief survey.\" arXiv preprint arXiv:1712.04323 (2017).","category":"page"},{"location":"general/states_variation/#Altering-States","page":"Altering States","title":"Altering States","text":"","category":"section"},{"location":"general/states_variation/","page":"Altering States","title":"Altering States","text":"In every ReservoirComputing model is posible to perform some alteration on the states in the training stage. Depending on the chosen modification this can improve the results for the prediction. Or more simply they can be used to reproduce results in the literature. The alterations are divided in two possibilities: the first concerns padding or extending the states, the second concerns non linear algorithms performed over the states.","category":"page"},{"location":"general/states_variation/#Padding-and-Extending-States","page":"Altering States","title":"Padding and Extending States","text":"","category":"section"},{"location":"general/states_variation/","page":"Altering States","title":"Altering States","text":"Extending the states means appending to them the corresponding input values. If textbfx(t) is the reservoir state at time t corresponding to the input textbfu(t) the extended state will be represented as textbfx(t) textbfu(t) where is intended as vertical concatenation. This procedure is, for example, used in Jaeger's Scholarpedia description of Echo State Networks. The extension of the states can be obtained in every ReservoirComputing.jl model by using the keyword argument states_type and calling the method ExtendedStates(). No argument is needed.","category":"page"},{"location":"general/states_variation/","page":"Altering States","title":"Altering States","text":"Padding the states is appending a constant value, 1.0 for example, to each state. Using the notation introduced before we can define the padded states as textbfx(t) 10. This approach is detailed in the seminal guide to Echo State Networks by Mantas Lukoševičius. By using the keyword argument states_type the user can call the method PaddedStates(padding) where padding represents the value that will be concatenated to the states. As default the value is set to unity, so the majority of times calling PaddedStates() will suffice.","category":"page"},{"location":"general/states_variation/","page":"Altering States","title":"Altering States","text":"Altough not found easily in the literature, it is also possible to pad the extended states by using the method PaddedExtendedStates(padding) that has unity as padding default as well.","category":"page"},{"location":"general/states_variation/","page":"Altering States","title":"Altering States","text":"Of course it is also possible to not apport any of these changes to the states by calling StandardStates(). This is also the default choice for the states.","category":"page"},{"location":"general/states_variation/#Non-Linear-Algorithms","page":"Altering States","title":"Non Linear Algorithms","text":"","category":"section"},{"location":"general/states_variation/","page":"Altering States","title":"Altering States","text":"First introduced in [1] and expanded in [2] these are nonlinear combinations of the columns of the matrix states. There are three such algorithms implemented. Using the keyword argument nla_type it is possible to choose in every model in ReservoirComputing.jl the specific non linear algorithm to use. The defualt value is set to NLADefault(), where no non linear algorithm takes place.","category":"page"},{"location":"general/states_variation/","page":"Altering States","title":"Altering States","text":"Following the nomenclature used in [2] the algorithms can be called as NLAT1(), NLAT2() and NLAT3(). To better explain what they do, let textbfx_i j be elements of the states matrix, with i=1T j=1N where T is the length of the training and N is the reservoir size. ","category":"page"},{"location":"general/states_variation/","page":"Altering States","title":"Altering States","text":"NLAT1","category":"page"},{"location":"general/states_variation/","page":"Altering States","title":"Altering States","text":"tildetextbfx_ij = textbfx_ij times textbfx_ij textif textitj is odd \ntildetextbfx_ij = textbfx_ij textif textitj is even","category":"page"},{"location":"general/states_variation/","page":"Altering States","title":"Altering States","text":"NLAT2","category":"page"},{"location":"general/states_variation/","page":"Altering States","title":"Altering States","text":"tildetextbfx_ij = textbfx_ij-1 times textbfx_ij-2 textif textitj 1 is odd \ntildetextbfx_ij = textbfx_ij textif textitj is 1 or even","category":"page"},{"location":"general/states_variation/","page":"Altering States","title":"Altering States","text":"NLAT3","category":"page"},{"location":"general/states_variation/","page":"Altering States","title":"Altering States","text":"tildetextbfx_ij = textbfx_ij-1 times textbfx_ij+1 textif textitj 1 is odd \ntildetextbfx_ij = textbfx_ij textif textitj is 1 or even","category":"page"},{"location":"general/states_variation/","page":"Altering States","title":"Altering States","text":"[1]: Pathak, Jaideep, et al. \"Using machine learning to replicate chaotic attractors and calculate Lyapunov exponents from data.\" Chaos: An Interdisciplinary Journal of Nonlinear Science 27.12 (2017): 121102.","category":"page"},{"location":"general/states_variation/","page":"Altering States","title":"Altering States","text":"[2]: Chattopadhyay, Ashesh, Pedram Hassanzadeh, and Devika Subramanian. \"Data-driven predictions of a multiscale Lorenz 96 chaotic system using machine-learning methods: reservoir computing, artificial neural network, and long short-term memory network.\" Nonlinear Processes in Geophysics 27.3 (2020): 373-389.","category":"page"},{"location":"esn_tutorials/different_training/#Using-Different-Training-Methods","page":"Using Different Training Methods","title":"Using Different Training Methods","text":"","category":"section"},{"location":"esn_tutorials/different_training/#Linear-Methods","page":"Using Different Training Methods","title":"Linear Methods","text":"","category":"section"},{"location":"esn_tutorials/different_training/#Echo-State-Gaussian-Processes","page":"Using Different Training Methods","title":"Echo State Gaussian Processes","text":"","category":"section"},{"location":"esn_tutorials/different_training/#Support-Vector-Echo-State-Machines","page":"Using Different Training Methods","title":"Support Vector Echo State Machines","text":"","category":"section"},{"location":"api/predict/#Prediction-Types","page":"Prediction Types","title":"Prediction Types","text":"","category":"section"},{"location":"api/predict/","page":"Prediction Types","title":"Prediction Types","text":" Generative\n Predictive","category":"page"},{"location":"api/predict/#ReservoirComputing.Generative","page":"Prediction Types","title":"ReservoirComputing.Generative","text":"Generative(prediction_len)\n\nThis prediction methodology allows the models to produce an autonomous prediction, feeding the prediction into itself to generate the next step. The only parameter needed is the number of steps for the prediction.\n\n\n\n\n\n","category":"type"},{"location":"api/predict/#ReservoirComputing.Predictive","page":"Prediction Types","title":"ReservoirComputing.Predictive","text":"Predictive(prediction_data)\n\nGiven a set of labels as prediction_data this method of prediction will return the correspinding labels in a standard Machine Learning fashion.\n\n\n\n\n\n","category":"type"},{"location":"api/reca/#Reservoir-Computing-with-Cellular-Automata","page":"ReCA","title":"Reservoir Computing with Cellular Automata","text":"","category":"section"},{"location":"api/reca/","page":"ReCA","title":"ReCA","text":" RECA","category":"page"},{"location":"api/reca/#ReservoirComputing.RECA","page":"ReCA","title":"ReservoirComputing.RECA","text":"RECA(train_data,\n automata;\n generations = 8,\n input_encoding=RandomMapping(),\n nla_type = NLADefault(),\n states_type = StandardStates())\n\n[1] Yilmaz, Ozgur. “Reservoir computing using cellular automata.” arXiv preprint arXiv:1410.0162 (2014).\n\n[2] Nichele, Stefano, and Andreas Molund. “Deep reservoir computing using cellular automata.” arXiv preprint arXiv:1703.02806 (2017).\n\n\n\n\n\n","category":"type"},{"location":"api/reca/","page":"ReCA","title":"ReCA","text":"The input encodings are the equivalent of the input matrices of the ESNs. These are the available encodings:","category":"page"},{"location":"api/reca/","page":"ReCA","title":"ReCA","text":" RandomMapping","category":"page"},{"location":"api/reca/#ReservoirComputing.RandomMapping","page":"ReCA","title":"ReservoirComputing.RandomMapping","text":"RandomMapping(permutations, expansion_size)\nRandomMapping(permutations; expansion_size=40)\nRandomMapping(;permutations=8, expansion_size=40)\n\nRandom mapping of the input data directly in the reservoir. The expansion_size determines the dimension of the single reservoir, and permutations determines the number of total reservoirs that will be connected, each with a different mapping. The detail of this implementation can be found in [1].\n\n[1] Nichele, Stefano, and Andreas Molund. “Deep reservoir computing using cellular automata.” arXiv preprint arXiv:1703.02806 (2017).\n\n\n\n\n\n","category":"type"},{"location":"api/reca/","page":"ReCA","title":"ReCA","text":"The training and prediction follow the same workflow of the ESN. It is important to note that at the moment we were not able to find any paper using these models with a Generative approach for the prediction, so full support is given only to the Predictive method.","category":"page"},{"location":"esn_tutorials/change_layers/#Using-Different-Layers","page":"Using Different Layers","title":"Using Different Layers","text":"","category":"section"},{"location":"esn_tutorials/change_layers/","page":"Using Different Layers","title":"Using Different Layers","text":"A great deal of efforts in the ESNs field are devoted to finding an ideal construction for the reservoir matrices. With a simple interface using ReservoirComputing.jl is possible to leverage the currently implemented matrix constructions methods for both the reservoir and the input layer. In this page it is showcased how it is possible to change both of these layers.","category":"page"},{"location":"esn_tutorials/change_layers/","page":"Using Different Layers","title":"Using Different Layers","text":"The input_init keyword argument provided with the ESN constructor allows for changing the input layer. The layers provided in ReservoirComputing.jl are the following:","category":"page"},{"location":"esn_tutorials/change_layers/","page":"Using Different Layers","title":"Using Different Layers","text":"WeightedLayer(scaling)\nDenseLayer(scaling)\nSparseLayer(scaling, sparsity)\nMinimumLayer(weight, sampling)\nInformedLayer(model_in_size; scaling=0.1, gamma=0.5)","category":"page"},{"location":"esn_tutorials/change_layers/","page":"Using Different Layers","title":"Using Different Layers","text":"In addition the user can define a custom layer following this workflow:","category":"page"},{"location":"esn_tutorials/change_layers/","page":"Using Different Layers","title":"Using Different Layers","text":"#creation of the new struct for the layer\nstruct MyNewLayer <: AbstractLayer\n #the layer params go here\nend\n\n#dispatch over the function to actually build the layer matrix\nfunction create_layer(input_layer::MyNewLayer, res_size, in_size)\n #the new algorithm to build the input layer goes here\nend","category":"page"},{"location":"esn_tutorials/change_layers/","page":"Using Different Layers","title":"Using Different Layers","text":"Similarly the reservoir_init keyword argument provides the possibility to change the construction for the reservoir matrix. The available reservoir are:","category":"page"},{"location":"esn_tutorials/change_layers/","page":"Using Different Layers","title":"Using Different Layers","text":"RandSparseReservoir(res_size, radius, sparsity)\nPseudoSVDReservoir(res_size, max_value, sparsity, sorted, reverse_sort)\nDelayLineReservoir(res_size, weight)\nDelayLineBackwardReservoir(res_size, weight, fb_weight)\nSimpleCycleReservoir(res_size, weight)\nCycleJumpsReservoir(res_size, cycle_weight, jump_weight, jump_size)","category":"page"},{"location":"esn_tutorials/change_layers/","page":"Using Different Layers","title":"Using Different Layers","text":"And, like before, it is possible to build a custom reservoir by following this workflow:","category":"page"},{"location":"esn_tutorials/change_layers/","page":"Using Different Layers","title":"Using Different Layers","text":"#creation of the new struct for the reservoir\nstruct MyNewReservoir <: AbstractReservoir\n #the reservoir params go here\nend\n\n#dispatch over the function to build the reservoir matrix\nfunction create_reservoir(reservoir::AbstractReservoir, res_size)\n #the new algorithm to build the reservoir matrix goes here\nend","category":"page"},{"location":"esn_tutorials/change_layers/#Example-of-minimally-complex-ESN","page":"Using Different Layers","title":"Example of minimally complex ESN","text":"","category":"section"},{"location":"esn_tutorials/change_layers/","page":"Using Different Layers","title":"Using Different Layers","text":"Using [1] and [2] as references this section will provide an example on how to change both the input layer and the reservoir for ESNs. The full script for this example can be found here. This example was run on Julia v1.7.2.","category":"page"},{"location":"esn_tutorials/change_layers/","page":"Using Different Layers","title":"Using Different Layers","text":"The task for this example will be the one step ahead prediction of the Henon map. To obtain the data one can leverage the package DynamicalSystems.jl. The data is scaled to be between -1 and 1.","category":"page"},{"location":"esn_tutorials/change_layers/","page":"Using Different Layers","title":"Using Different Layers","text":"using PredefinedDynamicalSystems\ntrain_len = 3000\npredict_len = 2000\n\nds = PredefinedDynamicalSystems.henon()\ntraj, time = trajectory(ds, 7000)\ndata = Matrix(traj)'\ndata = (data .-0.5) .* 2\nshift = 200\n\ntraining_input = data[:, shift:shift+train_len-1]\ntraining_target = data[:, shift+1:shift+train_len]\ntesting_input = data[:,shift+train_len:shift+train_len+predict_len-1]\ntesting_target = data[:,shift+train_len+1:shift+train_len+predict_len]","category":"page"},{"location":"esn_tutorials/change_layers/","page":"Using Different Layers","title":"Using Different Layers","text":"Now it is possible to define the input layers and reservoirs we want to compare and run the comparison in a simple for loop. The accuracy will be tested using the mean squared deviation msd from StatsBase.","category":"page"},{"location":"esn_tutorials/change_layers/","page":"Using Different Layers","title":"Using Different Layers","text":"using ReservoirComputing, StatsBase\n\nres_size = 300\ninput_layer = [MinimumLayer(0.85, IrrationalSample()), MinimumLayer(0.95, IrrationalSample())]\nreservoirs = [SimpleCycleReservoir(res_size, 0.7), \n CycleJumpsReservoir(res_size, cycle_weight=0.7, jump_weight=0.2, jump_size=5)]\n\nfor i=1:length(reservoirs)\n esn = ESN(training_input;\n input_layer = input_layer[i],\n reservoir = reservoirs[i])\n wout = train(esn, training_target, StandardRidge(0.001))\n output = esn(Predictive(testing_input), wout)\n println(msd(testing_target, output))\nend","category":"page"},{"location":"esn_tutorials/change_layers/","page":"Using Different Layers","title":"Using Different Layers","text":"As it is possible to see, changing layers in ESN models is straightforward. Be sure to check the API documentation for a full list of reservoir and layers.","category":"page"},{"location":"esn_tutorials/change_layers/#Bibliography","page":"Using Different Layers","title":"Bibliography","text":"","category":"section"},{"location":"esn_tutorials/change_layers/","page":"Using Different Layers","title":"Using Different Layers","text":"[1]: Rodan, Ali, and Peter Tiňo. “Simple deterministically constructed cycle reservoirs with regular jumps.” Neural computation 24.7 (2012): 1822-1852.","category":"page"},{"location":"esn_tutorials/change_layers/","page":"Using Different Layers","title":"Using Different Layers","text":"[2]: Rodan, Ali, and Peter Tiňo. “Minimum complexity echo state network.” IEEE transactions on neural networks 22.1 (2010): 131-144.","category":"page"},{"location":"general/different_training/#Changing-Training-Algorithms","page":"Changing Training Algorithms","title":"Changing Training Algorithms","text":"","category":"section"},{"location":"general/different_training/","page":"Changing Training Algorithms","title":"Changing Training Algorithms","text":"Notably Echo State Networks have been trained with Ridge Regression algorithms, but the range of useful algorithms to use is much greater. In this section of the documentation it is possible to explore how to use other training methods to obtain the readout layer. All the methods implemented in ReservoirComputing.jl can be used for all models in the library, not only ESNs. The general workflow illustrated in this section will be based on a dummy RC model my_model = MyModel(...) that need training in order to obtain the readout layer. The training is done following:","category":"page"},{"location":"general/different_training/","page":"Changing Training Algorithms","title":"Changing Training Algorithms","text":"training_algo = TrainingAlgo()\nreadout_layer = train(my_model, train_data, training_algo)","category":"page"},{"location":"general/different_training/","page":"Changing Training Algorithms","title":"Changing Training Algorithms","text":"In this section it is possible to explore how to properly build the training_algo and all the possible choices available. In the example section of the documentation it will be provided copy-pastable code to better explore the training algorithms and their impact over the model.","category":"page"},{"location":"general/different_training/#Linear-Models","page":"Changing Training Algorithms","title":"Linear Models","text":"","category":"section"},{"location":"general/different_training/","page":"Changing Training Algorithms","title":"Changing Training Algorithms","text":"The library includes a standard implementation of ridge regression, callable using StandardRidge(regularization_coeff) where the default value for the regularization coefficent is set to zero. This is also the default model called when no model is specified in train(). This makes the default call for traning train(my_model, train_data) use Ordinary Least Squares (OLS) for regression.","category":"page"},{"location":"general/different_training/","page":"Changing Training Algorithms","title":"Changing Training Algorithms","text":"Leveraging MLJLinearModels it is possible to expand the choices of linear models used for the training. The wrappers provided are structured in the following way:","category":"page"},{"location":"general/different_training/","page":"Changing Training Algorithms","title":"Changing Training Algorithms","text":"struct LinearModel\n regression\n solver\n regression_kwargs\nend","category":"page"},{"location":"general/different_training/","page":"Changing Training Algorithms","title":"Changing Training Algorithms","text":"to call the ridge regression using the MLJLinearModels APIs one can use LinearModel(;regression=LinearRegression). It is also possible to use a specific solver, by calling LinearModel(regression=LinearRegression, solver=Analytical()). For all the available solvers please reref to the MLJLinearModels documentation. To change the regularization coefficient in the ridge example, using for example lambda = 0.1, it is needed to pass it in the regression_kwargs like so LinearModel(;regression=LinearRegression, solver=Analytical(), regression_kwargs=(lambda=lambda)). The nomenclature of the coefficients must follow the MLJLinearModels APIs, using lambda, gamma for LassoRegression and delta, lambda, gamma for HuberRegression. Again, please check the relevant documentation if in doubt. When using MLJLinearModels based regressors do remember to specify using MLJLinearModels.","category":"page"},{"location":"general/different_training/#Gaussian-Processes","page":"Changing Training Algorithms","title":"Gaussian Processes","text":"","category":"section"},{"location":"general/different_training/","page":"Changing Training Algorithms","title":"Changing Training Algorithms","text":"Another way to obtain the readout layer is possible using Gaussian regression. This is provided through a wrapper of GaussianProcesses structured in the following way:","category":"page"},{"location":"general/different_training/","page":"Changing Training Algorithms","title":"Changing Training Algorithms","text":"struct GaussianProcess\n mean\n kernel\n lognoise\n optimize\n optimizer\nend","category":"page"},{"location":"general/different_training/","page":"Changing Training Algorithms","title":"Changing Training Algorithms","text":"While it is necessary to specify a mean and a kernel, the other defaults are lognoise=-2, optimize=false, optimizer=Optim.LBFGS(). For the choice of means and kernels please refer to the proper documentation, here and here respectively. ","category":"page"},{"location":"general/different_training/","page":"Changing Training Algorithms","title":"Changing Training Algorithms","text":"Building on the simple example given in the GaussianProcesses documentation it is possible to build an intuition of how to use this algorithms for training ReservoirComputing.jl models.","category":"page"},{"location":"general/different_training/","page":"Changing Training Algorithms","title":"Changing Training Algorithms","text":"mZero = MeanZero() #Zero mean function\nkern = SE(0.0,0.0) #Squared exponential kernel (note that hyperparameters are on the log scale)\nlogObsNoise = -1.0\n\ngp = GaussianProcess(mZero, kern, lognoise=logObsNoise)","category":"page"},{"location":"general/different_training/","page":"Changing Training Algorithms","title":"Changing Training Algorithms","text":"Like in the previous case, if one uses GaussianProcesses based regressors it is necessary to specify using GaussianProcesses. Additionally, if the optimizer chosen is from an external package, i.e. Optim, that package need to be used in the script as well adding using Optim.","category":"page"},{"location":"general/different_training/#Support-Vector-Regression","page":"Changing Training Algorithms","title":"Support Vector Regression","text":"","category":"section"},{"location":"general/different_training/","page":"Changing Training Algorithms","title":"Changing Training Algorithms","text":"Contrary to the LinearModels and GaussianProcesses, no wrappers are needed for support vector regression. By using LIBSVM.jl, LIBSVM wrappers in Julia, it is possible to call both epsilonSVR() or nuSVR() directly in train(). For the full range of kernel provided and the parameters to call we refer the user to the official documentation. Like before, if one intends to use LIBSVM regressors it is necessary to specify using LIBSVM.","category":"page"},{"location":"api/states/#States-Modifications","page":"States Modifications","title":"States Modifications","text":"","category":"section"},{"location":"api/states/#Padding-and-Estension","page":"States Modifications","title":"Padding and Estension","text":"","category":"section"},{"location":"api/states/","page":"States Modifications","title":"States Modifications","text":" StandardStates\n ExtendedStates\n PaddedStates\n PaddedExtendedStates","category":"page"},{"location":"api/states/#ReservoirComputing.StandardStates","page":"States Modifications","title":"ReservoirComputing.StandardStates","text":"StandardStates()\n\nNo modification of the states takes place, default option.\n\n\n\n\n\n","category":"type"},{"location":"api/states/#ReservoirComputing.ExtendedStates","page":"States Modifications","title":"ReservoirComputing.ExtendedStates","text":"ExtendedStates()\n\nThe states are extended with the input data, for the training section, and the prediction data, during the prediction section. This is obtained with a vertical concatenation of the data and the states.\n\n\n\n\n\n","category":"type"},{"location":"api/states/#ReservoirComputing.PaddedStates","page":"States Modifications","title":"ReservoirComputing.PaddedStates","text":"PaddedStates(padding)\nPaddedStates(;padding=1.0)\n\nThe states are padded with a chosen value. Usually this value is set to one. The padding is obtained through a vertical concatenation of the padding value and the states.\n\n\n\n\n\n","category":"type"},{"location":"api/states/#ReservoirComputing.PaddedExtendedStates","page":"States Modifications","title":"ReservoirComputing.PaddedExtendedStates","text":"PaddedExtendedStates(padding)\nPaddedExtendedStates(;padding=1.0)\n\nThe states are extended with the training data or predicted data and subsequently padded with a chosen value. Usually the padding value is set to one. The padding and the extension are obtained through a vertical concatenation of the padding value, the data and the states.\n\n\n\n\n\n","category":"type"},{"location":"api/states/#Non-Linear-Transformations","page":"States Modifications","title":"Non Linear Transformations","text":"","category":"section"},{"location":"api/states/","page":"States Modifications","title":"States Modifications","text":" NLADefault\n NLAT1\n NLAT2\n NLAT3","category":"page"},{"location":"api/states/#ReservoirComputing.NLADefault","page":"States Modifications","title":"ReservoirComputing.NLADefault","text":"NLADefault()\n\nReturns the array untouched, default option.\n\n\n\n\n\n","category":"type"},{"location":"api/states/#ReservoirComputing.NLAT1","page":"States Modifications","title":"ReservoirComputing.NLAT1","text":"NLAT1()\n\nApplies the $ \\text{T}_1 $ transformation algorithm, as defined in [1] and [2].\n\n[1] Chattopadhyay, Ashesh, et al. \"Data-driven prediction of a multi-scale Lorenz 96 chaotic system using a hierarchy of deep learning methods: Reservoir computing, ANN, and RNN-LSTM.\" (2019).\n\n[2] Pathak, Jaideep, et al. \"Model-free prediction of large spatiotemporally chaotic systems from data: A reservoir computing approach.\" Physical review letters 120.2 (2018): 024102.\n\n\n\n\n\n","category":"type"},{"location":"api/states/#ReservoirComputing.NLAT2","page":"States Modifications","title":"ReservoirComputing.NLAT2","text":"NLAT2()\n\nApply the $ \\text{T}_2 $ transformation algorithm, as defined in [1].\n\n[1] Chattopadhyay, Ashesh, et al. \"Data-driven prediction of a multi-scale Lorenz 96 chaotic system using a hierarchy of deep learning methods: Reservoir computing, ANN, and RNN-LSTM.\" (2019).\n\n\n\n\n\n","category":"type"},{"location":"api/states/#ReservoirComputing.NLAT3","page":"States Modifications","title":"ReservoirComputing.NLAT3","text":"NLAT3()\n\nApply the $ \\text{T}_3 $ transformation algorithm, as defined in [1].\n\n[1] Chattopadhyay, Ashesh, et al. \"Data-driven prediction of a multi-scale Lorenz 96 chaotic system using a hierarchy of deep learning methods: Reservoir computing, ANN, and RNN-LSTM.\" (2019).\n\n\n\n\n\n","category":"type"},{"location":"api/esn_drivers/#ESN-Drivers","page":"ESN Drivers","title":"ESN Drivers","text":"","category":"section"},{"location":"api/esn_drivers/","page":"ESN Drivers","title":"ESN Drivers","text":" RNN\n MRNN\n GRU","category":"page"},{"location":"api/esn_drivers/#ReservoirComputing.RNN","page":"ESN Drivers","title":"ReservoirComputing.RNN","text":"RNN(activation_function, leaky_coefficient)\nRNN(;activation_function=tanh, leaky_coefficient=1.0)\n\nReturns a Recurrent Neural Network initializer for the ESN. This is the default choice.\n\n\n\n\n\n","category":"type"},{"location":"api/esn_drivers/#ReservoirComputing.MRNN","page":"ESN Drivers","title":"ReservoirComputing.MRNN","text":"MRNN(activation_function, leaky_coefficient, scaling_factor)\nMRNN(;activation_function=[tanh, sigmoid], leaky_coefficient=1.0, \n scaling_factor=fill(leaky_coefficient, length(activation_function)))\n\nReturns a Multiple RNN initializer, where multiple function are combined in a linear combination with chosen parameters scaling_factor. The activation_function and scaling_factor arguments must vectors of the same size. Multiple combinations are possible, the implementation is based upon a double activation function idea, found in [1].\n\n[1] Lun, Shu-Xian, et al. \"A novel model of leaky integrator echo state network for time-series prediction.\" Neurocomputing 159 (2015): 58-66.\n\n\n\n\n\n","category":"type"},{"location":"api/esn_drivers/#ReservoirComputing.GRU","page":"ESN Drivers","title":"ReservoirComputing.GRU","text":"GRU(;activation_function=[NNlib.sigmoid, NNlib.sigmoid, tanh],\n inner_layer = fill(DenseLayer(), 2),\n reservoir = fill(RandSparseReservoir(), 2),\n bias = fill(DenseLayer(), 2),\n variant = FullyGated())\n\nReturns a Gated Recurrent Unit [1] reservoir driver.\n\n[1] Cho, Kyunghyun, et al. “Learning phrase representations using RNN encoder-decoder for statistical machine translation.” arXiv preprint arXiv:1406.1078 (2014).\n\n\n\n\n\n","category":"type"},{"location":"api/esn_drivers/","page":"ESN Drivers","title":"ESN Drivers","text":"The GRU driver also provides the user the choice of the possible variant:","category":"page"},{"location":"api/esn_drivers/","page":"ESN Drivers","title":"ESN Drivers","text":" FullyGated\n Minimal","category":"page"},{"location":"api/esn_drivers/#ReservoirComputing.FullyGated","page":"ESN Drivers","title":"ReservoirComputing.FullyGated","text":"FullyGated()\n\nReturns a standard Gated Recurrent Unit ESN initializer, as described in [1].\n\n[1] Cho, Kyunghyun, et al. “Learning phrase representations using RNN encoder-decoder for statistical machine translation.” arXiv preprint arXiv:1406.1078 (2014).\n\n\n\n\n\n","category":"type"},{"location":"api/esn_drivers/#ReservoirComputing.Minimal","page":"ESN Drivers","title":"ReservoirComputing.Minimal","text":"Minimal()\n\nReturns a minimal GRU ESN initializer as described in [1].\n\n[1] Zhou, Guo-Bing, et al. \"Minimal gated unit for recurrent neural networks.\" International Journal of Automation and Computing 13.3 (2016): 226-234.\n\n\n\n\n\n","category":"type"},{"location":"api/esn_drivers/","page":"ESN Drivers","title":"ESN Drivers","text":"Please refer to the original papers for more detail about these architectures.","category":"page"},{"location":"api/training/#Training-Algorithms","page":"Training Algorithms","title":"Training Algorithms","text":"","category":"section"},{"location":"api/training/#Linear-Models","page":"Training Algorithms","title":"Linear Models","text":"","category":"section"},{"location":"api/training/","page":"Training Algorithms","title":"Training Algorithms","text":" StandardRidge\n LinearModel","category":"page"},{"location":"api/training/#ReservoirComputing.StandardRidge","page":"Training Algorithms","title":"ReservoirComputing.StandardRidge","text":"StandardRidge(regularization_coeff)\nStandardRidge(;regularization_coeff=0.0)\n\nRidge regression training for all the models in the library. The regularization_coeff is the regularization, it can be passed as an arg or kwarg.\n\n\n\n\n\n","category":"type"},{"location":"api/training/#ReservoirComputing.LinearModel","page":"Training Algorithms","title":"ReservoirComputing.LinearModel","text":"LinearModel(;regression=LinearRegression, \n solver=Analytical(), \n regression_kwargs=(;))\n\nLinear regression training based on MLJLinearModels for all the models in the library. All the parameters have to be passed into regression_kwargs, apart from the solver choice. MLJLinearModels.jl needs to be called in order to use these models.\n\n\n\n\n\n","category":"type"},{"location":"api/training/#Gaussian-Regression","page":"Training Algorithms","title":"Gaussian Regression","text":"","category":"section"},{"location":"api/training/","page":"Training Algorithms","title":"Training Algorithms","text":"Currently (v0.9) unavailable.","category":"page"},{"location":"api/training/#Support-Vector-Regression","page":"Training Algorithms","title":"Support Vector Regression","text":"","category":"section"},{"location":"api/training/","page":"Training Algorithms","title":"Training Algorithms","text":"Support vector Regression is possible using a direct call to LIBSVM regression methods. Instead of a wrapper please refer to the use of LIBSVM.AbstractSVR in the original library.","category":"page"},{"location":"api/esn_layers/#ESN-Layers","page":"ESN Layers","title":"ESN Layers","text":"","category":"section"},{"location":"api/esn_layers/#Input-Layers","page":"ESN Layers","title":"Input Layers","text":"","category":"section"},{"location":"api/esn_layers/","page":"ESN Layers","title":"ESN Layers","text":" WeightedLayer\n DenseLayer\n SparseLayer\n InformedLayer\n MinimumLayer\n NullLayer","category":"page"},{"location":"api/esn_layers/#ReservoirComputing.WeightedLayer","page":"ESN Layers","title":"ReservoirComputing.WeightedLayer","text":"WeightedInput(scaling)\nWeightedInput(;scaling=0.1)\n\nReturns a weighted layer initializer object, that will produce a weighted input matrix with a with random non-zero elements drawn from [-scaling, scaling], as described in [1]. The scaling factor can be given as arg or kwarg.\n\n[1] Lu, Zhixin, et al. \"Reservoir observers: Model-free inference of unmeasured variables in chaotic systems.\" Chaos: An Interdisciplinary Journal of Nonlinear Science 27.4 (2017): 041102.\n\n\n\n\n\n","category":"type"},{"location":"api/esn_layers/#ReservoirComputing.DenseLayer","page":"ESN Layers","title":"ReservoirComputing.DenseLayer","text":"DenseLayer(scaling)\nDenseLayer(;scaling=0.1)\n\nReturns a fully connected layer initializer object, that will produce a weighted input matrix with a with random non-zero elements drawn from [-scaling, scaling]. The scaling factor can be given as arg or kwarg. This is the default choice in the ESN construction.\n\n\n\n\n\n","category":"type"},{"location":"api/esn_layers/#ReservoirComputing.SparseLayer","page":"ESN Layers","title":"ReservoirComputing.SparseLayer","text":"SparseLayer(scaling, sparsity)\nSparseLayer(scaling; sparsity=0.1)\nSparseLayer(;scaling=0.1, sparsity=0.1)\n\nReturns a sparsely connected layer initializer object, that will produce a random sparse input matrix with random non-zero elements drawn from [-scaling, scaling] and given sparsity. The scaling and sparsity factors can be given as args or kwargs.\n\n\n\n\n\n","category":"type"},{"location":"api/esn_layers/#ReservoirComputing.InformedLayer","page":"ESN Layers","title":"ReservoirComputing.InformedLayer","text":"InformedLayer(model_in_size; scaling=0.1, gamma=0.5)\n\nReturns a weighted input layer matrix, with random non-zero elements drawn from [-scaling, scaling], where some γ of reservoir nodes are connected exclusively to the raw inputs, and the rest to the outputs of the prior knowledge model, as described in [1].\n\n[1] Jaideep Pathak et al. \"Hybrid Forecasting of Chaotic Processes: Using Machine Learning in Conjunction with a Knowledge-Based Model\" (2018)\n\n\n\n\n\n","category":"type"},{"location":"api/esn_layers/#ReservoirComputing.MinimumLayer","page":"ESN Layers","title":"ReservoirComputing.MinimumLayer","text":"MinimumLayer(weight, sampling)\nMinimumLayer(weight; sampling=BernoulliSample(0.5))\nMinimumLayer(;weight=0.1, sampling=BernoulliSample(0.5))\n\nReturns a fully connected layer initializer object. The matrix constructed with this initializer presents the same absolute weight value, decided by the weight factor. The sign of each entry is decided by the sampling struct. Construction detailed in [1] and [2].\n\n[1] Rodan, Ali, and Peter Tino. \"Minimum complexity echo state network.\" IEEE transactions on neural networks 22.1 (2010): 131-144. [2] Rodan, Ali, and Peter Tiňo. \"Simple deterministically constructed cycle reservoirs with regular jumps.\" Neural computation 24.7 (2012): 1822-1852.\n\n\n\n\n\n","category":"type"},{"location":"api/esn_layers/#ReservoirComputing.NullLayer","page":"ESN Layers","title":"ReservoirComputing.NullLayer","text":"NullLayer(model_in_size; scaling=0.1, gamma=0.5)\n\nReturns a vector of zeros.\n\n\n\n\n\n","category":"type"},{"location":"api/esn_layers/","page":"ESN Layers","title":"ESN Layers","text":"The sign in the MinimumLayer are chosen based on the following methods:","category":"page"},{"location":"api/esn_layers/","page":"ESN Layers","title":"ESN Layers","text":" BernoulliSample\n IrrationalSample","category":"page"},{"location":"api/esn_layers/#ReservoirComputing.BernoulliSample","page":"ESN Layers","title":"ReservoirComputing.BernoulliSample","text":"BernoulliSample(p)\nBernoulliSample(;p=0.5)\n\nReturns a Bernoulli sign constructor for the MinimumLayer call. The p factor determines the probability of the result as in the Distributions call. The value can be passed as an arg or kwarg. This sign weight determination for input layers is introduced in [1].\n\n[1] Rodan, Ali, and Peter Tino. \"Minimum complexity echo state network.\" IEEE transactions on neural networks 22.1 (2010): 131-144.\n\n\n\n\n\n","category":"type"},{"location":"api/esn_layers/#ReservoirComputing.IrrationalSample","page":"ESN Layers","title":"ReservoirComputing.IrrationalSample","text":"IrrationalSample(irrational, start)\nIrrationalSample(;irrational=pi, start=1)\n\nReturns an irrational sign contructor for the '''MinimumLayer''' call. The values can be passed as args or kwargs. The sign of the weight are decided from the decimal expansion of the given irrational. The first start decimal digits are thresholded at 4.5, then the n-th input sign will be + and - respectively.\n\n[1] Rodan, Ali, and Peter Tiňo. \"Simple deterministically constructed cycle reservoirs with regular jumps.\" Neural computation 24.7 (2012): 1822-1852.\n\n\n\n\n\n","category":"type"},{"location":"api/esn_layers/","page":"ESN Layers","title":"ESN Layers","text":"To derive the matrix one can call the following function:","category":"page"},{"location":"api/esn_layers/","page":"ESN Layers","title":"ESN Layers","text":" create_layer","category":"page"},{"location":"api/esn_layers/#ReservoirComputing.create_layer","page":"ESN Layers","title":"ReservoirComputing.create_layer","text":"create_layer(input_layer::AbstractLayer, res_size, in_size)\n\nReturns a res_size times in_size matrix layer, built accordingly to the input_layer constructor.\n\n\n\n\n\n","category":"function"},{"location":"api/esn_layers/","page":"ESN Layers","title":"ESN Layers","text":"To create new input layers it suffice to define a new struct containing the needed parameters of the new input layer. This struct wiil need to be an AbstractLayer, so the create_layer function can be dispatched over it. The workflow should follow this snippet:","category":"page"},{"location":"api/esn_layers/","page":"ESN Layers","title":"ESN Layers","text":"#creation of the new struct for the layer\nstruct MyNewLayer <: AbstractLayer\n #the layer params go here\nend\n\n#dispatch over the function to actually build the layer matrix\nfunction create_layer(input_layer::MyNewLayer, res_size, in_size)\n #the new algorithm to build the input layer goes here\nend","category":"page"},{"location":"api/esn_layers/#Reservoirs","page":"ESN Layers","title":"Reservoirs","text":"","category":"section"},{"location":"api/esn_layers/","page":"ESN Layers","title":"ESN Layers","text":" RandSparseReservoir\n PseudoSVDReservoir\n DelayLineReservoir\n DelayLineBackwardReservoir\n SimpleCycleReservoir\n CycleJumpsReservoir\n NullReservoir","category":"page"},{"location":"api/esn_layers/#ReservoirComputing.RandSparseReservoir","page":"ESN Layers","title":"ReservoirComputing.RandSparseReservoir","text":"RandSparseReservoir(res_size, radius, sparsity)\nRandSparseReservoir(res_size; radius=1.0, sparsity=0.1)\n\nReturns a random sparse reservoir initializer, that will return a matrix with given sparsity and scaled spectral radius according to radius. This is the default choice in the ESN construction.\n\n\n\n\n\n","category":"type"},{"location":"api/esn_layers/#ReservoirComputing.PseudoSVDReservoir","page":"ESN Layers","title":"ReservoirComputing.PseudoSVDReservoir","text":"PseudoSVDReservoir(max_value, sparsity, sorted, reverse_sort)\nPseudoSVDReservoir(max_value, sparsity; sorted=true, reverse_sort=false)\n\nReturns an initializer to build a sparse reservoir matrix, with given sparsity created using SVD as described in [1]. \n\n[1] Yang, Cuili, et al. \"Design of polynomial echo state networks for time series prediction.\" Neurocomputing 290 (2018): 148-160.\n\n\n\n\n\n","category":"type"},{"location":"api/esn_layers/#ReservoirComputing.DelayLineReservoir","page":"ESN Layers","title":"ReservoirComputing.DelayLineReservoir","text":"DelayLineReservoir(res_size, weight)\nDelayLineReservoir(res_size; weight=0.1)\n\nReturns a Delay Line Reservoir matrix constructor to obtain a deterministi reservoir as described in [1]. The weight can be passed as arg or kwarg and it determines the absolute value of all the connections in the reservoir.\n\n[1] Rodan, Ali, and Peter Tino. \"Minimum complexity echo state network.\" IEEE transactions on neural networks 22.1 (2010): 131-144.\n\n\n\n\n\n","category":"type"},{"location":"api/esn_layers/#ReservoirComputing.DelayLineBackwardReservoir","page":"ESN Layers","title":"ReservoirComputing.DelayLineBackwardReservoir","text":"DelayLineBackwardReservoir(res_size, weight, fb_weight)\nDelayLineBackwardReservoir(res_size; weight=0.1, fb_weight=0.2)\n\nReturns a Delay Line Reservoir constructor to create a matrix with Backward connections as described in [1]. The weight and fb_weight can be passed as either args or kwargs, and they determine the only absolute values of the connections in the reservoir.\n\n[1] Rodan, Ali, and Peter Tino. \"Minimum complexity echo state network.\" IEEE transactions on neural networks 22.1 (2010): 131-144.\n\n\n\n\n\n","category":"type"},{"location":"api/esn_layers/#ReservoirComputing.SimpleCycleReservoir","page":"ESN Layers","title":"ReservoirComputing.SimpleCycleReservoir","text":"SimpleCycleReservoir(res_size, weight)\nSimpleCycleReservoir(res_size; weight=0.1)\n\nReturns a Simple Cycle Reservoir Reservoir constructor to biuld a reservoir matrix as described in [1]. The weight can be passed as arg or kwarg and it determines the absolute value of all the connections in the reservoir.\n\n[1] Rodan, Ali, and Peter Tino. \"Minimum complexity echo state network.\" IEEE transactions on neural networks 22.1 (2010): 131-144.\n\n\n\n\n\n","category":"type"},{"location":"api/esn_layers/#ReservoirComputing.CycleJumpsReservoir","page":"ESN Layers","title":"ReservoirComputing.CycleJumpsReservoir","text":"CycleJumpsReservoir(res_size; cycle_weight=0.1, jump_weight=0.1, jump_size=3)\nCycleJumpsReservoir(res_size, cycle_weight, jump_weight, jump_size)\n\nReturn a Cycle Reservoir with Jumps constructor to create a reservoir matrix as described in [1]. The weight and jump_weight can be passed as args or kwargs and they determine the absolute values of all the connections in the reservoir. The jump_size can also be passed either as arg and kwarg and it detemines the jumps between jump_weights.\n\n[1] Rodan, Ali, and Peter Tiňo. \"Simple deterministically constructed cycle reservoirs with regular jumps.\" Neural computation 24.7 (2012): 1822-1852.\n\n\n\n\n\n","category":"type"},{"location":"api/esn_layers/#ReservoirComputing.NullReservoir","page":"ESN Layers","title":"ReservoirComputing.NullReservoir","text":"NullReservoir()\n\nReturn a constructor for a matrix zeros(res_size, res_size)\n\n\n\n\n\n","category":"type"},{"location":"api/esn_layers/","page":"ESN Layers","title":"ESN Layers","text":"Like for the input layers, to actually build the matrix of the reservoir one can call the following function:","category":"page"},{"location":"api/esn_layers/","page":"ESN Layers","title":"ESN Layers","text":" create_reservoir","category":"page"},{"location":"api/esn_layers/#ReservoirComputing.create_reservoir","page":"ESN Layers","title":"ReservoirComputing.create_reservoir","text":"create_reservoir(reservoir::AbstractReservoir, res_size)\ncreate_reservoir(reservoir, args...)\n\nGiven an ``AbstractReservoir constructor and the reservoir size it returns the corresponding matrix. Alternatively it accepts a given matrix.\n\n\n\n\n\n","category":"function"},{"location":"api/esn_layers/","page":"ESN Layers","title":"ESN Layers","text":"To create a new reservoir the procedure is imilar to the one for the input layers. First the definition of the new struct of type AbstractReservoir with the reservoir parameters is needed. Then the dispatch over the create_reservoir function makes the model actually build the reservoir matrix. An example of the workflow is given in the following snippet:","category":"page"},{"location":"api/esn_layers/","page":"ESN Layers","title":"ESN Layers","text":"#creation of the new struct for the reservoir\nstruct MyNewReservoir <: AbstractReservoir\n #the reservoir params go here\nend\n\n#dispatch over the function to build the reservoir matrix\nfunction create_reservoir(reservoir::AbstractReservoir, res_size)\n #the new algorithm to build the reservoir matrix goes here\nend","category":"page"},{"location":"#ReservoirComputing.jl","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"","category":"section"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"ReservoirComputing.jl provides an efficient, modular and easy to use implementation of Reservoir Computing models such as Echo State Networks (ESNs). Reservoir Computing (RC) is an umbrella term used to describe a family of models such as ESNs and Liquid State Machines (LSMs). The key concept is to expand the input data into a higher dimension and use regression in order to train the model; in some ways Reservoir Computers can be considered similar to kernel methods. ","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"info: Introductory material\nThis library assumes some basic knowledge of Reservoir Computing. For a good introduction, we suggest the following papers: the first two are the seminal papers about ESN and LSM, the others are in-depth review papers that should cover all the needed information. For the majority of the algorithms implemented in this library we cited in the documentation the original work introducing them. If you ever are in doubt about about a method or a function just type ? function in the Julia REPL to read the relevant notes.Jaeger, Herbert: The “echo state” approach to analyzing and training recurrent neural networks-with an erratum note.\nMaass W, Natschläger T, Markram H: Real-time computing without stable states: a new framework for neural computation based on perturbations.\nLukoševičius, Mantas: A practical guide to applying echo state networks.\" Neural networks: Tricks of the trade.\nLukoševičius, Mantas, and Herbert Jaeger: Reservoir computing approaches to recurrent neural network training.","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"info: Performance tip\nFor faster computations on the CPU it is suggested to add using MKL to the script. For clarity's sake this library will not be indicated under every example in the documentation.","category":"page"},{"location":"#Installation","page":"ReservoirComputing.jl","title":"Installation","text":"","category":"section"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"ReservoirComputing.jl is registered in the General Julia Registry, so the installation of the package follows the usual procedure:","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"import Pkg; Pkg.add(\"ReservoirComputing\")","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"The support for this library is for Julia v1.6 or greater.","category":"page"},{"location":"#Features-Overview","page":"ReservoirComputing.jl","title":"Features Overview","text":"","category":"section"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"This library provides multiple ways of training the chosen RC model. More specifically the available algorithms are:","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"StandardRidge: a naive implementation of Ridge Regression. The default choice for training.\nLinearModel: a wrap around MLJLinearModels.\nLIBSVM.AbstractSVR: a direct call of LIBSVM regression methods.","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"Also provided are two different ways of doing predictions using RC:","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"Generative: the algorithm uses the prediction of the model in the previous step to continue the prediction. It only needs the number of steps as input.\nPredictive: standard Machine Learning type of prediction. Given the features the RC model will return the label/prediction.","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"It is possible to modify the RC obtained states in the training and prediction step using the following:","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"StandardStates: default choice, no changes will be made to the states.\nExtendedStates: the states are extended using a vertical concatenation with the input data.\nPaddedStates: the states are padded using a vertical concatenation with the choosing padding value\nPaddedExtendedStates: a combination of the first two. First the states are extended and then padded.","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"In addition another modification is possible through the choice of non linear algorithms:","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"NLADefault: default choice, no changes will be made to the states.\nNLAT1\nNLAT2\nNLAT3","category":"page"},{"location":"#Echo-State-Networks","page":"ReservoirComputing.jl","title":"Echo State Networks","text":"","category":"section"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"Regarding ESNs in the library are implemented the following input layers:","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"WeightedLayer: weighted layer matrix with weights sampled from a uniform distribution.\nDenseLayer: dense layer matrix with weights sampled from a uniform distribution.\nSparseLayer: sparse layer matrix with weights sampled from a uniform distribution.\nMinimumLayer: matrix with constant weights and weight sign decided following one of the two:\nBernoulliSample\nIrrationalSample\nInformedLayer: special kin of weighted layer matrix for Hybrid ESNs.","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"The package also contains multiple implementation of Reservoirs:","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"RandSparseReservoir: random sparse matrix with scaling of spectral radius\nPseudoSVDReservoir: Pseudo SVD construction of a random sparse matrix\nDelayLineReservoir: minimal matrix with chosen weights\nDelayLineBackwardReservoir: minimal matrix with chosen weights\nSimpleCycleReservoir: minimal matrix with chosen weights\nCycleJumpsReservoir: minimal matrix with chosen weights","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"In addition multiple ways of driving the reservoir states are also provided:","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"RNN: standard Recurrent Neural Network driver.\nMRNN: Multiple RNN driver, it consists on a linear combination of RNNs\nGRU: gated Recurrent Unit driver, with all the possible GRU variants available:\nFullyGated\nVariant1\nVariant2\nVariant3\nMinimal","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"An hybrid version of the model is also available through Hybrid","category":"page"},{"location":"#Reservoir-Computing-with-Cellular-Automata","page":"ReservoirComputing.jl","title":"Reservoir Computing with Cellular Automata","text":"","category":"section"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"The package provides also an implementation of Reservoir Computing models based on one dimensional Cellular Automata through the RECA call. For the moment the only input encoding available (an input encoding plays a similar role to the input matrix for ESNs) is a random mapping, called through RandomMapping. ","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"All the training methods described above can be used, as well as all the modifications to the states. Both prediction methods are also possible in theory, although in the literature only Predictive tasks have been explored.","category":"page"},{"location":"#Contributing","page":"ReservoirComputing.jl","title":"Contributing","text":"","category":"section"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"Contributions are very welcomed! Some interesting variation of RC models are posted in the issues, but everyone is free to just post relevant papers that could fit the scope of the library. Help with the documentation, providing new examples or application cases is also really important and appreciated. Everything that can make the package a little better is a great contribution, no matter how small. The API section of the documentation provides a more in depth look into how things work and are connected, so that is a good place to start exploring more the library. For every doubt that cannot be expressed in issues please feel free to contact any of the lead developers on Slack or by email.","category":"page"},{"location":"#Citing","page":"ReservoirComputing.jl","title":"Citing","text":"","category":"section"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"If you use this library in your work, please cite:","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"@article{JMLR:v23:22-0611,\n author = {Francesco Martinuzzi and Chris Rackauckas and Anas Abdelrehim and Miguel D. Mahecha and Karin Mora},\n title = {ReservoirComputing.jl: An Efficient and Modular Library for Reservoir Computing Models},\n journal = {Journal of Machine Learning Research},\n year = {2022},\n volume = {23},\n number = {288},\n pages = {1--8},\n url = {http://jmlr.org/papers/v23/22-0611.html}\n}","category":"page"},{"location":"#Reproducibility","page":"ReservoirComputing.jl","title":"Reproducibility","text":"","category":"section"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"The documentation of this SciML package was built using these direct dependencies,","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"using Pkg # hide\nPkg.status() # hide","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"and using this machine and Julia version.","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"using InteractiveUtils # hide\nversioninfo() # hide","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"A more complete overview of all dependencies and their versions is also provided.","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"using Pkg # hide\nPkg.status(;mode = PKGMODE_MANIFEST) # hide","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"You can also download the \nmanifest file and the\nproject file.","category":"page"},{"location":"api/esn/#Echo-State-Networks","page":"Echo State Networks","title":"Echo State Networks","text":"","category":"section"},{"location":"api/esn/","page":"Echo State Networks","title":"Echo State Networks","text":" ESN","category":"page"},{"location":"api/esn/#ReservoirComputing.ESN","page":"Echo State Networks","title":"ReservoirComputing.ESN","text":"ESN(train_data;\n variation = Default(),\n input_layer = DenseLayer(),\n reservoir = RandSparseReservoir(),\n bias = NullLayer(),\n reservoir_driver = RNN(),\n nla_type = NLADefault(),\n states_type = StandardStates())\n(esn::ESN)(prediction::AbstractPrediction,\n output_layer::AbstractOutputLayer;\n initial_conditions=output_layer.last_value,\n last_state=esn.states[:, end])\n\nConstructor for the Echo State Network model. It requires the reservoir size as the input and the data for the training. It returns a struct ready to be trained with the states already harvested. \n\nAfter the training this struct can be used for the prediction following the second function call. This will take as input a prediction type and the output layer from the training. The initial_conditions and last_state parameters can be left as they are, unless there is a specific reason to change them. All the components are detailed in the API documentation. More examples are given in the general documentation.\n\n\n\n\n\n","category":"type"},{"location":"api/esn/","page":"Echo State Networks","title":"Echo State Networks","text":"In addition to all the components that can be explored in the documentation a couple need a separate introduction. The variation arguments can be","category":"page"},{"location":"api/esn/","page":"Echo State Networks","title":"Echo State Networks","text":" Default\n Hybrid","category":"page"},{"location":"api/esn/#ReservoirComputing.Default","page":"Echo State Networks","title":"ReservoirComputing.Default","text":"Default()\n\nSets the type of the ESN as the standard model. No parameters are needed.\n\n\n\n\n\n","category":"type"},{"location":"api/esn/#ReservoirComputing.Hybrid","page":"Echo State Networks","title":"ReservoirComputing.Hybrid","text":"Hybrid(prior_model, u0, tspan, datasize)\n\nGiven the model parameters returns an Hybrid variation of the ESN. This entails a different training and prediction. Construction based on [1].\n\n[1] Jaideep Pathak et al. \"Hybrid Forecasting of Chaotic Processes: Using Machine Learning in Conjunction with a Knowledge-Based Model\" (2018)\n\n\n\n\n\n","category":"type"},{"location":"api/esn/","page":"Echo State Networks","title":"Echo State Networks","text":"These arguments detail more deep variation of the underlying model and they need a separate call. For the moment the most complex is the Hybrid call, but this can and will change in the future. All ESN models can be trained using the following call:","category":"page"},{"location":"api/esn/","page":"Echo State Networks","title":"Echo State Networks","text":" train","category":"page"},{"location":"api/esn/#ReservoirComputing.train","page":"Echo State Networks","title":"ReservoirComputing.train","text":"train(esn::AbstractEchoStateNetwork, target_data, training_method=StandardRidge(0.0))\n\nTraining of the built ESN over the target_data. The default training method is RidgeRegression. The output is an OutputLayer object to be fed at the esn call for the prediction.\n\n\n\n\n\n","category":"function"}]
+[{"location":"general/predictive_generative/#Generative-vs-Predictive","page":"Generative vs Predictive","title":"Generative vs Predictive","text":"","category":"section"},{"location":"general/predictive_generative/","page":"Generative vs Predictive","title":"Generative vs Predictive","text":"The library provides two different methods for prediction, denoted as Predictive() and Generative(), following the two major applications of Reservoir Computing models found in the literature. Both of these methods are given as arguments for the trained model. While copy-pasteable example swill be provided further on in the documentation, it is better to clarify the difference early on to focus more on the library implementation going forward.","category":"page"},{"location":"general/predictive_generative/#Predictive","page":"Generative vs Predictive","title":"Predictive","text":"","category":"section"},{"location":"general/predictive_generative/","page":"Generative vs Predictive","title":"Generative vs Predictive","text":"In the first method, the user can use Reservoir Computing models similarly as standard Machine Learning models. This means using a set of features as input and a set of labels as outputs. In this case, the features and labels can be vectors of different dimensions, as X=x_1x_n x_i in mathbbR^N and Y=y_1y_n y_i in mathbbR^M where X is the feature set and Y the label set. Given the difference in dimensionality for the prediction call, it will be needed to feed to the function the feature set to be labeled, for example by calling Predictive(X) using the set given in this example.","category":"page"},{"location":"general/predictive_generative/","page":"Generative vs Predictive","title":"Generative vs Predictive","text":"!this allows for one step ahead or h steps ahead prediction.","category":"page"},{"location":"general/predictive_generative/#Generative","page":"Generative vs Predictive","title":"Generative","text":"","category":"section"},{"location":"general/predictive_generative/","page":"Generative vs Predictive","title":"Generative vs Predictive","text":"The generative method allows the user to extend the forecasting capabilities of the model, letting the predicted results to be fed back into the model to generate the next prediction. By doing so, the model can run autonomously, without any feature dataset as input. The call for this model needs only the number of steps that the user intends to forecast, for example calling Generative(100) to generate one hundred time steps.","category":"page"},{"location":"reca_tutorials/reca/#Reservoir-Computing-using-Cellular-Automata","page":"Reservoir Computing with Cellular Automata","title":"Reservoir Computing using Cellular Automata","text":"","category":"section"},{"location":"reca_tutorials/reca/","page":"Reservoir Computing with Cellular Automata","title":"Reservoir Computing with Cellular Automata","text":"Reservoir Computing based on Elementary Cellular Automata (ECA) has been recently introduced. Dubbed as ReCA [1][2] it proposed the advantage of storing the reservoir states as binary data. Less parameter tuning represents another advantage of this model. The architecture implemented in ReservoirComputing.jl follows [3] which builds on top of the original implementation, improving the results. It is strongly suggested to go through the paper to get a solid understanding of the model before delving into experimentation with the code.","category":"page"},{"location":"reca_tutorials/reca/","page":"Reservoir Computing with Cellular Automata","title":"Reservoir Computing with Cellular Automata","text":"To showcase how to use these models, this page illustrates the performance of ReCA in the 5 bit memory task [4]. The script for the example and companion data can be found here.","category":"page"},{"location":"reca_tutorials/reca/#bit-memory-task","page":"Reservoir Computing with Cellular Automata","title":"5 bit memory task","text":"","category":"section"},{"location":"reca_tutorials/reca/","page":"Reservoir Computing with Cellular Automata","title":"Reservoir Computing with Cellular Automata","text":"The data can be read as follows:","category":"page"},{"location":"reca_tutorials/reca/","page":"Reservoir Computing with Cellular Automata","title":"Reservoir Computing with Cellular Automata","text":"using DelimitedFiles\n\ninput = readdlm(\"./5bitinput.txt\", ',', Float32)\noutput = readdlm(\"./5bitoutput.txt\", ',', Float32)","category":"page"},{"location":"reca_tutorials/reca/","page":"Reservoir Computing with Cellular Automata","title":"Reservoir Computing with Cellular Automata","text":"To use a ReCA model, it is necessary to define the rule one intends to use. To do so, ReservoirComputing.jl leverages CellularAutomata.jl that needs to be called as well to define the RECA struct:","category":"page"},{"location":"reca_tutorials/reca/","page":"Reservoir Computing with Cellular Automata","title":"Reservoir Computing with Cellular Automata","text":"using ReservoirComputing, CellularAutomata\n\nca = DCA(90)","category":"page"},{"location":"reca_tutorials/reca/","page":"Reservoir Computing with Cellular Automata","title":"Reservoir Computing with Cellular Automata","text":"To define the ReCA model, it suffices to call:","category":"page"},{"location":"reca_tutorials/reca/","page":"Reservoir Computing with Cellular Automata","title":"Reservoir Computing with Cellular Automata","text":"reca = RECA(input, ca; \n generations = 16,\n input_encoding = RandomMapping(16, 40))","category":"page"},{"location":"reca_tutorials/reca/","page":"Reservoir Computing with Cellular Automata","title":"Reservoir Computing with Cellular Automata","text":"After this, the training can be performed with the chosen method. ","category":"page"},{"location":"reca_tutorials/reca/","page":"Reservoir Computing with Cellular Automata","title":"Reservoir Computing with Cellular Automata","text":"output_layer = train(reca, output, StandardRidge(0.00001))","category":"page"},{"location":"reca_tutorials/reca/","page":"Reservoir Computing with Cellular Automata","title":"Reservoir Computing with Cellular Automata","text":"The prediction in this case will be a Predictive() with the input data equal to the training data. In addition, to test the 5 bit memory task, a conversion from Float to Bool is necessary (at the moment, we are aware of a bug that doesn't allow boolean input data to the RECA models):","category":"page"},{"location":"reca_tutorials/reca/","page":"Reservoir Computing with Cellular Automata","title":"Reservoir Computing with Cellular Automata","text":"prediction = reca(Predictive(input), output_layer)\nfinal_pred = convert(AbstractArray{Float32}, prediction .> 0.5)\n\nfinal_pred == output","category":"page"},{"location":"reca_tutorials/reca/","page":"Reservoir Computing with Cellular Automata","title":"Reservoir Computing with Cellular Automata","text":"[1]: Yilmaz, Ozgur. \"Reservoir computing using cellular automata.\" arXiv preprint arXiv:1410.0162 (2014).","category":"page"},{"location":"reca_tutorials/reca/","page":"Reservoir Computing with Cellular Automata","title":"Reservoir Computing with Cellular Automata","text":"[2]: Margem, Mrwan, and Ozgür Yilmaz. \"An experimental study on cellular automata reservoir in pathological sequence learning tasks.\" (2017).","category":"page"},{"location":"reca_tutorials/reca/","page":"Reservoir Computing with Cellular Automata","title":"Reservoir Computing with Cellular Automata","text":"[3]: Nichele, Stefano, and Andreas Molund. \"Deep reservoir computing using cellular automata.\" arXiv preprint arXiv:1703.02806 (2017).","category":"page"},{"location":"reca_tutorials/reca/","page":"Reservoir Computing with Cellular Automata","title":"Reservoir Computing with Cellular Automata","text":"[4]: Hochreiter, Sepp, and Jürgen Schmidhuber. \"Long short-term memory.\" Neural computation 9.8 (1997): 1735-1780.","category":"page"},{"location":"esn_tutorials/different_drivers/#Using-Different-Reservoir-Drivers","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"","category":"section"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"While the original implementation of the Echo State Network implemented the model using the equations of Recurrent Neural Networks to obtain non-linearity in the reservoir, other variations have been proposed in recent years. More specifically, the different drivers implemented in ReservoirComputing.jl are the multiple activation function RNN MRNN() and the Gated Recurrent Unit GRU(). To change them, it suffices to give the chosen method to the ESN keyword argument reservoir_driver. In this section, some examples, of their usage will be given, as well as a brief introduction to their equations.","category":"page"},{"location":"esn_tutorials/different_drivers/#Multiple-Activation-Function-RNN","page":"Using Different Reservoir Drivers","title":"Multiple Activation Function RNN","text":"","category":"section"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"Based on the double activation function ESN (DAFESN) proposed in [1], the Multiple Activation Function ESN expands the idea and allows a custom number of activation functions to be used in the reservoir dynamics. This can be thought of as a linear combination of multiple activation functions with corresponding parameters.","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"mathbfx(t+1) = (1-alpha)mathbfx(t) + lambda_1 f_1(mathbfWmathbfx(t)+mathbfW_inmathbfu(t)) + dots + lambda_D f_D(mathbfWmathbfx(t)+mathbfW_inmathbfu(t))","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"where D is the number of activation functions and respective parameters chosen.","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"The method to call to use the multiple activation function ESN is MRNN(activation_function, leaky_coefficient, scaling_factor). The arguments can be used as both args and kwargs. activation_function and scaling_factor have to be vectors (or tuples) containing the chosen activation functions and respective scaling factors (f_1f_D and lambda_1lambda_D following the nomenclature introduced above). The leaky_coefficient represents alpha and it is a single value. ","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"Starting with the example, the data used is based on the following function based on the DAFESN paper [1]. A full script of the example is available here. This example was run on Julia v1.7.2.","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"u(t) = sin(t)+sin(0.51*t)+sin(0.22*t)+sin(0.1002*t)+sin(0.05343*t)","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"For this example, the type of prediction will be one step ahead. The metric used to assure a good prediction will be the normalized root-mean-square deviation rmsd from StatsBase. Like in the other examples, first it is needed to gather the data:","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"train_len = 3000\npredict_len = 2000\nshift = 1\n\ndata = u.(collect(0.0:0.01:500))\ntraining_input = reduce(hcat, data[shift:shift+train_len-1])\ntraining_target = reduce(hcat, data[shift+1:shift+train_len])\ntesting_input = reduce(hcat, data[shift+train_len:shift+train_len+predict_len-1])\ntesting_target = reduce(hcat, data[shift+train_len+1:shift+train_len+predict_len])","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"To follow the paper more closely, it is necessary to define a couple of activation functions. The numbering of them follows the ones in the paper. Of course, one can also use any custom-defined function, available in the base language or any activation function from NNlib.","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"f2(x) = (1-exp(-x))/(2*(1+exp(-x)))\nf3(x) = (2/pi)*atan((pi/2)*x)\nf4(x) = x/sqrt(1+x*x)","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"It is now possible to build different drivers, using the parameters suggested by the paper. Also, in this instance, the numbering follows the test cases of the paper. In the end, a simple for loop is implemented to compare the different drivers and activation functions.","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"using ReservoirComputing, Random, StatsBase\n\n#fix seed for reproducibility\nRandom.seed!(42)\n\n#baseline case with RNN() driver. Parameter given as args\nbase_case = RNN(tanh, 0.85)\n\n#MRNN() test cases\n#Parameter given as kwargs\ncase3 = MRNN(activation_function=[tanh, f2], \n leaky_coefficient=0.85, \n scaling_factor=[0.5, 0.3])\n\n#Parameter given as kwargs\ncase4 = MRNN(activation_function=[tanh, f3], \n leaky_coefficient=0.9, \n scaling_factor=[0.45, 0.35])\n\n#Parameter given as args\ncase5 = MRNN([tanh, f4], 0.9, [0.43, 0.13])\n\n#tests\ntest_cases = [base_case, case3, case4, case5]\nfor case in test_cases\n esn = ESN(training_input,\n input_layer = WeightedLayer(scaling=0.3),\n reservoir = RandSparseReservoir(100, radius=0.4),\n reservoir_driver = case,\n states_type = ExtendedStates())\n wout = train(esn, training_target, StandardRidge(10e-6))\n output = esn(Predictive(testing_input), wout)\n println(rmsd(testing_target, output, normalize=true))\nend","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"In this example, it is also possible to observe the input of parameters to the methods RNN() MRNN(), both by argument and by keyword argument.","category":"page"},{"location":"esn_tutorials/different_drivers/#Gated-Recurrent-Unit","page":"Using Different Reservoir Drivers","title":"Gated Recurrent Unit","text":"","category":"section"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"Gated Recurrent Units (GRUs) [2] have been proposed in more recent years with the intent of limiting notable problems of RNNs, like the vanishing gradient. This change in the underlying equations can be easily transported into the Reservoir Computing paradigm, by switching the RNN equations in the reservoir with the GRU equations. This approach has been explored in [3] and [4]. Different variations of GRU have been proposed [5][6]; this section is subdivided into different sections that go into detail about the governing equations and the implementation of them into ReservoirComputing.jl. Like before, to access the GRU reservoir driver, it suffices to change the reservoir_diver keyword argument for ESN with GRU(). All the variations that will be presented can be used in this package by leveraging the keyword argument variant in the method GRU() and specifying the chosen variant: FullyGated() or Minimal(). Other variations are possible by modifying the inner layers and reservoirs. The default is set to the standard version FullyGated(). The first section will go into more detail about the default of the GRU() method, and the following ones will refer to it to minimize repetitions. This example was run on Julia v1.7.2.","category":"page"},{"location":"esn_tutorials/different_drivers/#Standard-GRU","page":"Using Different Reservoir Drivers","title":"Standard GRU","text":"","category":"section"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"The equations for the standard GRU are as follows:","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"mathbfr(t) = sigma (mathbfW^r_textinmathbfu(t)+mathbfW^rmathbfx(t-1)+mathbfb_r) \nmathbfz(t) = sigma (mathbfW^z_textinmathbfu(t)+mathbfW^zmathbfx(t-1)+mathbfb_z) \ntildemathbfx(t) = texttanh(mathbfW_inmathbfu(t)+mathbfW(mathbfr(t) odot mathbfx(t-1))+mathbfb) \nmathbfx(t) = mathbfz(t) odot mathbfx(t-1)+(1-mathbfz(t)) odot tildemathbfx(t)","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"Going over the GRU keyword argument, it will be explained how to feed the desired input to the model. ","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"activation_function is a vector with default values [NNlib.sigmoid, NNlib.sigmoid, tanh]. This argument controls the activation functions of the GRU, going from top to bottom. Changing the first element corresponds to changing the activation function for mathbfr(t) and so on.\ninner_layer is a vector with default values fill(DenseLayer(), 2). This keyword argument controls the mathbfW_textins going from top to bottom like before.\nreservoir is a vector with default value fill(RandSparseReservoir(), 2). In a similar fashion to inner_layer, this keyword argument controls the reservoir matrix construction in a top to bottom order.\nbias is again a vector with default value fill(DenseLayer(), 2). It is meant to control the mathbfbs, going as usual from top to bottom.\nvariant controls the GRU variant. The default value is set to FullyGated().","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"It is important to notice that inner_layer and reservoir control every layer except mathbfW_in and mathbfW and mathbfb. These arguments are given as input to the ESN() call as input_layer, reservoir and bias. ","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"The following sections are going to illustrate the variations of the GRU architecture and how to obtain them in ReservoirComputing.jl","category":"page"},{"location":"esn_tutorials/different_drivers/#Type-1","page":"Using Different Reservoir Drivers","title":"Type 1","text":"","category":"section"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"The first variation of the GRU is dependent only on the previous hidden state and the bias:","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"mathbfr(t) = sigma (mathbfW^rmathbfx(t-1)+mathbfb_r) \nmathbfz(t) = sigma (mathbfW^zmathbfx(t-1)+mathbfb_z) ","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"To obtain this variation, it will suffice to set inner_layer = fill(NullLayer(), 2) and leaving the variant = FullyGated().","category":"page"},{"location":"esn_tutorials/different_drivers/#Type-2","page":"Using Different Reservoir Drivers","title":"Type 2","text":"","category":"section"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"The second variation only depends on the previous hidden state:","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"mathbfr(t) = sigma (mathbfW^rmathbfx(t-1)) \nmathbfz(t) = sigma (mathbfW^zmathbfx(t-1)) ","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"Similarly to before, to obtain this variation, it is only required to set inner_layer = fill(NullLayer(), 2) and bias = fill(NullLayer(), 2) while keeping variant = FullyGated().","category":"page"},{"location":"esn_tutorials/different_drivers/#Type-3","page":"Using Different Reservoir Drivers","title":"Type 3","text":"","category":"section"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"The final variation, before the minimal one, depends only on the biases","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"mathbfr(t) = sigma (mathbfb_r) \nmathbfz(t) = sigma (mathbfb_z) ","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"This means that it is only needed to set inner_layer = fill(NullLayer(), 2) and reservoir = fill(NullReservoir(), 2) while keeping variant = FullyGated().","category":"page"},{"location":"esn_tutorials/different_drivers/#Minimal","page":"Using Different Reservoir Drivers","title":"Minimal","text":"","category":"section"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"The minimal GRU variation merges two gates into one:","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"mathbff(t) = sigma (mathbfW^f_textinmathbfu(t)+mathbfW^fmathbfx(t-1)+mathbfb_f) \ntildemathbfx(t) = texttanh(mathbfW_inmathbfu(t)+mathbfW(mathbff(t) odot mathbfx(t-1))+mathbfb) \nmathbfx(t) = (1-mathbff(t)) odot mathbfx(t-1) + mathbff(t) odot tildemathbfx(t)","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"This variation can be obtained by setting variation=Minimal(). The inner_layer, reservoir and bias kwargs this time are not vectors, but must be defined like, for example inner_layer = DenseLayer() or reservoir = SparseDenseReservoir().","category":"page"},{"location":"esn_tutorials/different_drivers/#Examples","page":"Using Different Reservoir Drivers","title":"Examples","text":"","category":"section"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"To showcase the use of the GRU() method, this section will only illustrate the standard FullyGated() version. The full script for this example with the data can be found here. ","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"The data used for this example is the Santa Fe laser dataset [7] retrieved from here. The data is split to account for a next step prediction.","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"using DelimitedFiles\n\ndata = reduce(hcat, readdlm(\"./data/santafe_laser.txt\"))\n\ntrain_len = 5000\npredict_len = 2000\n\ntraining_input = data[:, 1:train_len]\ntraining_target = data[:, 2:train_len+1]\ntesting_input = data[:,train_len+1:train_len+predict_len]\ntesting_target = data[:,train_len+2:train_len+predict_len+1]","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"The construction of the ESN proceeds as usual. ","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"using ReservoirComputing, Random\n\nres_size = 300\nres_radius = 1.4\n\nRandom.seed!(42)\nesn = ESN(training_input; \n reservoir = RandSparseReservoir(res_size, radius=res_radius),\n reservoir_driver = GRU())","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"The default inner reservoir and input layer for the GRU are the same defaults for the reservoir and input_layer of the ESN. One can use the explicit call if they choose to.","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"gru = GRU(reservoir=[RandSparseReservoir(res_size), \n RandSparseReservoir(res_size)],\n inner_layer=[DenseLayer(), DenseLayer()])\nesn = ESN(training_input; \n reservoir = RandSparseReservoir(res_size, radius=res_radius),\n reservoir_driver = gru)","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"The training and prediction can proceed as usual:","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"training_method = StandardRidge(0.0)\noutput_layer = train(esn, training_target, training_method)\noutput = esn(Predictive(testing_input), output_layer)","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"The results can be plotted using Plots.jl","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"using Plots\n\nplot([testing_target' output'], label=[\"actual\" \"predicted\"], \n plot_title=\"Santa Fe Laser\",\n titlefontsize=20,\n legendfontsize=12,\n linewidth=2.5,\n xtickfontsize = 12,\n ytickfontsize = 12,\n size=(1080, 720))","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"It is interesting to see a comparison of the GRU driven ESN and the standard RNN driven ESN. Using the same parameters defined before it is possible to do the following","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"using StatsBase\n\nesn_rnn = ESN(training_input; \n reservoir = RandSparseReservoir(res_size, radius=res_radius),\n reservoir_driver = RNN())\n\noutput_layer = train(esn_rnn, training_target, training_method)\noutput_rnn = esn_rnn(Predictive(testing_input), output_layer)\n\nprintln(msd(testing_target, output))\nprintln(msd(testing_target, output_rnn))","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"[1]: Lun, Shu-Xian, et al. \"A novel model of leaky integrator echo state network for time-series prediction.\" Neurocomputing 159 (2015): 58-66.","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"[2]: Cho, Kyunghyun, et al. “Learning phrase representations using RNN encoder-decoder for statistical machine translation.” arXiv preprint arXiv:1406.1078 (2014).","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"[3]: Wang, Xinjie, Yaochu Jin, and Kuangrong Hao. \"A Gated Recurrent Unit based Echo State Network.\" 2020 International Joint Conference on Neural Networks (IJCNN). IEEE, 2020.","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"[4]: Di Sarli, Daniele, Claudio Gallicchio, and Alessio Micheli. \"Gated Echo State Networks: a preliminary study.\" 2020 International Conference on INnovations in Intelligent SysTems and Applications (INISTA). IEEE, 2020.","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"[5]: Dey, Rahul, and Fathi M. Salem. \"Gate-variants of gated recurrent unit (GRU) neural networks.\" 2017 IEEE 60th international midwest symposium on circuits and systems (MWSCAS). IEEE, 2017.","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"[6]: Zhou, Guo-Bing, et al. \"Minimal gated unit for recurrent neural networks.\" International Journal of Automation and Computing 13.3 (2016): 226-234.","category":"page"},{"location":"esn_tutorials/different_drivers/","page":"Using Different Reservoir Drivers","title":"Using Different Reservoir Drivers","text":"[7]: Hübner, Uwe, Nimmi B. Abraham, and Carlos O. Weiss. \"Dimensions and entropies of chaotic intensity pulsations in a single-mode far-infrared NH 3 laser.\" Physical Review A 40.11 (1989): 6354.","category":"page"},{"location":"esn_tutorials/lorenz_basic/#Lorenz-System-Forecasting","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"","category":"section"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"This example expands on the readme Lorenz system forecasting to better showcase how to use methods and functions provided in the library for Echo State Networks. Here the prediction method used is Generative, for a more detailed explanation of the differences between Generative and Predictive please refer to the other examples given in the documentation. The full script for this example is available here. This example was run on Julia v1.7.2.","category":"page"},{"location":"esn_tutorials/lorenz_basic/#Generating-the-data","page":"Lorenz System Forecasting","title":"Generating the data","text":"","category":"section"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"Starting off the workflow, the first step is to obtain the data. Leveraging OrdinaryDiffEq it is possible to derive the Lorenz system data in the following way:","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"using OrdinaryDiffEq\n\n#define lorenz system\nfunction lorenz!(du,u,p,t)\n du[1] = 10.0*(u[2]-u[1])\n du[2] = u[1]*(28.0-u[3]) - u[2]\n du[3] = u[1]*u[2] - (8/3)*u[3]\nend\n\n#solve and take data\nprob = ODEProblem(lorenz!, [1.0,0.0,0.0], (0.0,200.0))\ndata = solve(prob, ABM54(), dt=0.02)","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"After obtaining the data, it is necessary to determine the kind of prediction for the model. Since this example will use the Generative prediction type, this means that the target data will be the next step of the input data. In addition, it is important to notice that the Lorenz system just obtained presents a transient period that is not representative of the general behavior of the system. This can easily be discarded by setting a shift parameter.","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"#determine shift length, training length and prediction length\nshift = 300\ntrain_len = 5000\npredict_len = 1250\n\n#split the data accordingly\ninput_data = data[:, shift:shift+train_len-1]\ntarget_data = data[:, shift+1:shift+train_len]\ntest_data = data[:,shift+train_len+1:shift+train_len+predict_len]","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"It is important to notice that the data needs to be formatted in a matrix with the features as rows and time steps as columns as in this example. This is needed even if the time series consists of single values. ","category":"page"},{"location":"esn_tutorials/lorenz_basic/#Building-the-Echo-State-Network","page":"Lorenz System Forecasting","title":"Building the Echo State Network","text":"","category":"section"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"Once the data is ready, it is possible to define the parameters for the ESN and the ESN struct itself. In this example, the values from [1] are loosely followed as general guidelines.","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"using ReservoirComputing\n\n#define ESN parameters\nres_size = 300\nres_radius = 1.2\nres_sparsity = 6/300\ninput_scaling = 0.1\n\n#build ESN struct\nesn = ESN(input_data; \n variation = Default(),\n reservoir = RandSparseReservoir(res_size, radius=res_radius, sparsity=res_sparsity),\n input_layer = WeightedLayer(scaling=input_scaling),\n reservoir_driver = RNN(),\n nla_type = NLADefault(),\n states_type = StandardStates())","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"Most of the parameters chosen here mirror the default ones, so a direct call is not necessary. The readme example is identical to this one, except for the explicit call. Going line by line to see what is happening, starting from res_size: this value determines the dimensions of the reservoir matrix. In this case, a size of 300 has been chosen, so the reservoir matrix will be 300 x 300. This is not always the case, since some input layer constructions can modify the dimensions of the reservoir, but in that case, everything is taken care of internally. ","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"The res_radius determines the scaling of the spectral radius of the reservoir matrix; a proper scaling is necessary to assure the Echo State Property. The default value in the RandSparseReservoir() method is 1.0 in accordance with the most commonly followed guidelines found in the literature (see [2] and references therein). The sparsity of the reservoir matrix in this case is obtained by choosing a degree of connections and dividing that by the reservoir size. Of course, it is also possible to simply choose any value between 0.0 and 1.0 to test behaviors for different sparsity values. In this example, the call to the parameters inside RandSparseReservoir() was done explicitly to showcase the meaning of each of them, but it is also possible to simply pass the values directly, like so RandSparseReservoir(1.2, 6/300).","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"The value of input_scaling determines the upper and lower bounds of the uniform distribution of the weights in the WeightedLayer(). Like before, this value can be passed either as an argument or as a keyword argument WeightedLayer(0.1). The value of 0.1 represents the default. The default input layer is the DenseLayer, a fully connected layer. The details of the weighted version can be found in [3], for this example, this version returns the best results.","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"The reservoir driver represents the dynamics of the reservoir. In the standard ESN definition, these dynamics are obtained through a Recurrent Neural Network (RNN), and this is reflected by calling the RNN driver for the ESN struct. This option is set as the default, and unless there is the need to change parameters, it is not needed. The full equation is the following:","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"textbfx(t+1) = (1-alpha)textbfx(t) + alpha cdot texttanh(textbfWtextbfx(t)+textbfW_textintextbfu(t))","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"where α represents the leaky coefficient, and tanh can be any activation function. Also, textbfx represents the state vector, textbfu the input data, and textbfW textbfW_textin are the reservoir matrix and input matrix, respectively. The default call to the RNN in the library is the following RNN(;activation_function=tanh, leaky_coefficient=1.0), where the meaning of the parameters is clear from the equation above. Instead of the hyperbolic tangent, any activation function can be used, either leveraging external libraries such as NNlib or creating a custom one. ","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"The final calls are modifications to the states in training or prediction. The default calls, depicted in the example, do not make any modifications to the states. This is the safest bet if one is not sure how these work. The nla_type applies a non-linear algorithm to the states, while the states_type can expand them by concatenating them with the input data, or padding them by concatenating a constant value to all the states. More in depth descriptions of these parameters are given in other examples in the documentation.","category":"page"},{"location":"esn_tutorials/lorenz_basic/#Training-and-Prediction","page":"Lorenz System Forecasting","title":"Training and Prediction","text":"","category":"section"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"Now that the ESN has been created and all the parameters have been explained, it is time to proceed with the training. The full call of the readme example follows this general idea:","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"#define training method\ntraining_method = StandardRidge(0.0)\n\n#obtain output layer\noutput_layer = train(esn, target_data, training_method)","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"The training returns an OutputLayer struct containing the trained output matrix and other needed for the prediction. The necessary elements in the train() call are the ESN struct created in the previous step and the target_data, which in this case is the one step ahead evolution of the Lorenz system. The training method chosen in this example is the standard one, so an equivalent way of calling the train function here is output_layer = train(esn, target_data) like the readme basic version. Likewise, the default value for the ridge regression parameter is set to zero, so the actual default training is Ordinary Least Squares regression. Other training methods are available and will be explained in the following examples. ","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"Once the OutputLayer has been obtained, the prediction can be done following this procedure:","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"output = esn(Generative(predict_len), output_layer)","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"both the training method and the output layer are needed in this call. The number of steps for the prediction must be specified in the Generative method. The output results are given in a matrix. ","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"info: Saving the states during prediction\nWhile the states are saved in the ESN struct for the training, for the prediction they are not saved by default. To inspect the states, it is necessary to pass the boolean keyword argument save_states to the prediction call, in this example using esn(... ; save_states=true). This returns a tuple (output, states) where size(states) = res_size, prediction_len","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"To inspect the results, they can easily be plotted using an external library. In this case, Plots is adopted:","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"using Plots, Plots.PlotMeasures\n\nts = 0.0:0.02:200.0\nlorenz_maxlyap = 0.9056\npredict_ts = ts[shift+train_len+1:shift+train_len+predict_len]\nlyap_time = (predict_ts .- predict_ts[1])*(1/lorenz_maxlyap)\n\np1 = plot(lyap_time, [test_data[1,:] output[1,:]], label = [\"actual\" \"predicted\"], \n ylabel = \"x(t)\", linewidth=2.5, xticks=false, yticks = -15:15:15);\np2 = plot(lyap_time, [test_data[2,:] output[2,:]], label = [\"actual\" \"predicted\"], \n ylabel = \"y(t)\", linewidth=2.5, xticks=false, yticks = -20:20:20);\np3 = plot(lyap_time, [test_data[3,:] output[3,:]], label = [\"actual\" \"predicted\"], \n ylabel = \"z(t)\", linewidth=2.5, xlabel = \"max(λ)*t\", yticks = 10:15:40);\n\n\nplot(p1, p2, p3, plot_title = \"Lorenz System Coordinates\", \n layout=(3,1), xtickfontsize = 12, ytickfontsize = 12, xguidefontsize=15, yguidefontsize=15,\n legendfontsize=12, titlefontsize=20)","category":"page"},{"location":"esn_tutorials/lorenz_basic/#Bibliography","page":"Lorenz System Forecasting","title":"Bibliography","text":"","category":"section"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"[1]: Pathak, Jaideep, et al. \"Using machine learning to replicate chaotic attractors and calculate Lyapunov exponents from data.\" Chaos: An Interdisciplinary Journal of Nonlinear Science 27.12 (2017): 121102.","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"[2]: Lukoševičius, Mantas. \"A practical guide to applying echo state networks.\" Neural networks: Tricks of the trade. Springer, Berlin, Heidelberg, 2012. 659-686.","category":"page"},{"location":"esn_tutorials/lorenz_basic/","page":"Lorenz System Forecasting","title":"Lorenz System Forecasting","text":"[3]: Lu, Zhixin, et al. \"Reservoir observers: Model-free inference of unmeasured variables in chaotic systems.\" Chaos: An Interdisciplinary Journal of Nonlinear Science 27.4 (2017): 041102.","category":"page"},{"location":"esn_tutorials/hybrid/#Hybrid-Echo-State-Networks","page":"Hybrid Echo State Networks","title":"Hybrid Echo State Networks","text":"","category":"section"},{"location":"esn_tutorials/hybrid/","page":"Hybrid Echo State Networks","title":"Hybrid Echo State Networks","text":"Following the idea of giving physical information to machine learning models, the hybrid echo state networks [1] try to achieve this results by feeding model data into the ESN. In this example, it is explained how to create and leverage such models in ReservoirComputing.jl. The full script for this example is available here. This example was run on Julia v1.7.2.","category":"page"},{"location":"esn_tutorials/hybrid/#Generating-the-data","page":"Hybrid Echo State Networks","title":"Generating the data","text":"","category":"section"},{"location":"esn_tutorials/hybrid/","page":"Hybrid Echo State Networks","title":"Hybrid Echo State Networks","text":"For this example, we are going to forecast the Lorenz system. As usual, the data is generated leveraging DifferentialEquations.jl:","category":"page"},{"location":"esn_tutorials/hybrid/","page":"Hybrid Echo State Networks","title":"Hybrid Echo State Networks","text":"using DifferentialEquations\n\nu0 = [1.0,0.0,0.0] \ntspan = (0.0,1000.0) \ndatasize = 100000\ntsteps = range(tspan[1], tspan[2], length = datasize) \n\nfunction lorenz(du,u,p,t)\n p = [10.0,28.0,8/3]\n du[1] = p[1]*(u[2]-u[1])\n du[2] = u[1]*(p[2]-u[3]) - u[2]\n du[3] = u[1]*u[2] - p[3]*u[3]\nend\n\node_prob = ODEProblem(lorenz, u0, tspan)\node_sol = solve(ode_prob, saveat = tsteps)\node_data =Array(ode_sol)\n\ntrain_len = 10000\n\ninput_data = ode_data[:, 1:train_len]\ntarget_data = ode_data[:, 2:train_len+1]\ntest_data = ode_data[:, train_len+1:end][:, 1:1000]\n\npredict_len = size(test_data, 2)\ntspan_train = (tspan[1], ode_sol.t[train_len])","category":"page"},{"location":"esn_tutorials/hybrid/#Building-the-Hybrid-Echo-State-Network","page":"Hybrid Echo State Networks","title":"Building the Hybrid Echo State Network","text":"","category":"section"},{"location":"esn_tutorials/hybrid/","page":"Hybrid Echo State Networks","title":"Hybrid Echo State Networks","text":"To feed the data to the ESN, it is necessary to create a suitable function.","category":"page"},{"location":"esn_tutorials/hybrid/","page":"Hybrid Echo State Networks","title":"Hybrid Echo State Networks","text":"function prior_model_data_generator(u0, tspan, tsteps, model = lorenz)\n prob = ODEProblem(lorenz, u0, tspan) \n sol = Array(solve(prob, saveat = tsteps))\n return sol\nend","category":"page"},{"location":"esn_tutorials/hybrid/","page":"Hybrid Echo State Networks","title":"Hybrid Echo State Networks","text":"Given the initial condition, time span, and time steps, this function returns the data for the chosen model. Now, using the Hybrid method, it is possible to input all this information to the model.","category":"page"},{"location":"esn_tutorials/hybrid/","page":"Hybrid Echo State Networks","title":"Hybrid Echo State Networks","text":"using ReservoirComputing, Random\nRandom.seed!(42)\n\nhybrid = Hybrid(prior_model_data_generator, u0, tspan_train, train_len)\n\nesn = ESN(input_data,\n reservoir = RandSparseReservoir(300),\n variation = hybrid)","category":"page"},{"location":"esn_tutorials/hybrid/#Training-and-Prediction","page":"Hybrid Echo State Networks","title":"Training and Prediction","text":"","category":"section"},{"location":"esn_tutorials/hybrid/","page":"Hybrid Echo State Networks","title":"Hybrid Echo State Networks","text":"The training and prediction of the Hybrid ESN can proceed as usual:","category":"page"},{"location":"esn_tutorials/hybrid/","page":"Hybrid Echo State Networks","title":"Hybrid Echo State Networks","text":"output_layer = train(esn, target_data, StandardRidge(0.3))\noutput = esn(Generative(predict_len), output_layer)","category":"page"},{"location":"esn_tutorials/hybrid/","page":"Hybrid Echo State Networks","title":"Hybrid Echo State Networks","text":"It is now possible to plot the results, leveraging Plots.jl:","category":"page"},{"location":"esn_tutorials/hybrid/","page":"Hybrid Echo State Networks","title":"Hybrid Echo State Networks","text":"using Plots\nlorenz_maxlyap = 0.9056\npredict_ts = tsteps[train_len+1:train_len+predict_len]\nlyap_time = (predict_ts .- predict_ts[1])*(1/lorenz_maxlyap)\n\np1 = plot(lyap_time, [test_data[1,:] output[1,:]], label = [\"actual\" \"predicted\"], \n ylabel = \"x(t)\", linewidth=2.5, xticks=false, yticks = -15:15:15);\np2 = plot(lyap_time, [test_data[2,:] output[2,:]], label = [\"actual\" \"predicted\"], \n ylabel = \"y(t)\", linewidth=2.5, xticks=false, yticks = -20:20:20);\np3 = plot(lyap_time, [test_data[3,:] output[3,:]], label = [\"actual\" \"predicted\"], \n ylabel = \"z(t)\", linewidth=2.5, xlabel = \"max(λ)*t\", yticks = 10:15:40);\n\n\nplot(p1, p2, p3, plot_title = \"Lorenz System Coordinates\", \n layout=(3,1), xtickfontsize = 12, ytickfontsize = 12, xguidefontsize=15, yguidefontsize=15,\n legendfontsize=12, titlefontsize=20)","category":"page"},{"location":"esn_tutorials/hybrid/#Bibliography","page":"Hybrid Echo State Networks","title":"Bibliography","text":"","category":"section"},{"location":"esn_tutorials/hybrid/","page":"Hybrid Echo State Networks","title":"Hybrid Echo State Networks","text":"[1]: Pathak, Jaideep, et al. \"Hybrid forecasting of chaotic processes: Using machine learning in conjunction with a knowledge-based model.\" Chaos: An Interdisciplinary Journal of Nonlinear Science 28.4 (2018): 041101.","category":"page"},{"location":"esn_tutorials/deep_esn/#Deep-Echo-State-Networks","page":"Deep Echo State Networks","title":"Deep Echo State Networks","text":"","category":"section"},{"location":"esn_tutorials/deep_esn/","page":"Deep Echo State Networks","title":"Deep Echo State Networks","text":"Deep Echo State Network architectures started to gain some traction recently. In this guide, we illustrate how it is possible to use ReservoirComputing.jl to build a deep ESN. ","category":"page"},{"location":"esn_tutorials/deep_esn/","page":"Deep Echo State Networks","title":"Deep Echo State Networks","text":"The network implemented in this library is taken from [1]. It works by stacking reservoirs on top of each other, feeding the output from one into the next. The states are obtained by merging all the inner states of the stacked reservoirs. For a more in-depth explanation, refer to the paper linked above. The full script for this example can be found here. This example was run on Julia v1.7.2.","category":"page"},{"location":"esn_tutorials/deep_esn/#Lorenz-Example","page":"Deep Echo State Networks","title":"Lorenz Example","text":"","category":"section"},{"location":"esn_tutorials/deep_esn/","page":"Deep Echo State Networks","title":"Deep Echo State Networks","text":"For this example, we are going to reuse the Lorenz data used in the Lorenz System Forecasting example.","category":"page"},{"location":"esn_tutorials/deep_esn/","page":"Deep Echo State Networks","title":"Deep Echo State Networks","text":"using OrdinaryDiffEq\n\n#define lorenz system\nfunction lorenz!(du,u,p,t)\n du[1] = 10.0*(u[2]-u[1])\n du[2] = u[1]*(28.0-u[3]) - u[2]\n du[3] = u[1]*u[2] - (8/3)*u[3]\nend\n\n#solve and take data\nprob = ODEProblem(lorenz!, [1.0,0.0,0.0], (0.0,200.0))\ndata = solve(prob, ABM54(), dt=0.02)\n\n#determine shift length, training length and prediction length\nshift = 300\ntrain_len = 5000\npredict_len = 1250\n\n#split the data accordingly\ninput_data = data[:, shift:shift+train_len-1]\ntarget_data = data[:, shift+1:shift+train_len]\ntest_data = data[:,shift+train_len+1:shift+train_len+predict_len]","category":"page"},{"location":"esn_tutorials/deep_esn/","page":"Deep Echo State Networks","title":"Deep Echo State Networks","text":"Again, it is important to notice that the data needs to be formatted in a matrix, with the features as rows and time steps as columns, as in this example. This is needed even if the time series consists of single values. ","category":"page"},{"location":"esn_tutorials/deep_esn/","page":"Deep Echo State Networks","title":"Deep Echo State Networks","text":"The construction of the ESN is also really similar. The only difference is that the reservoir can be fed as an array of reservoirs. ","category":"page"},{"location":"esn_tutorials/deep_esn/","page":"Deep Echo State Networks","title":"Deep Echo State Networks","text":"using ReservoirComputing\n\nreservoirs = [RandSparseReservoir(99, radius=1.1, sparsity=0.1),\n RandSparseReservoir(100, radius=1.2, sparsity=0.1),\n RandSparseReservoir(200, radius=1.4, sparsity=0.1)]\n\nesn = ESN(input_data; \n variation = Default(),\n reservoir = reservoirs,\n input_layer = DenseLayer(),\n reservoir_driver = RNN(),\n nla_type = NLADefault(),\n states_type = StandardStates())","category":"page"},{"location":"esn_tutorials/deep_esn/","page":"Deep Echo State Networks","title":"Deep Echo State Networks","text":"As it is possible to see, different sizes can be chosen for the different reservoirs. The input layer and bias can also be given as vectors, but of course, they have to be of the same size of the reservoirs vector. If they are not passed as a vector, the value passed will be used for all the layers in the deep ESN.","category":"page"},{"location":"esn_tutorials/deep_esn/","page":"Deep Echo State Networks","title":"Deep Echo State Networks","text":"In addition to using the provided functions for the construction of the layers, the user can also choose to build their own matrix, or array of matrices, and feed that into the ESN in the same way.","category":"page"},{"location":"esn_tutorials/deep_esn/","page":"Deep Echo State Networks","title":"Deep Echo State Networks","text":"The training and prediction follow the usual framework:","category":"page"},{"location":"esn_tutorials/deep_esn/","page":"Deep Echo State Networks","title":"Deep Echo State Networks","text":"training_method = StandardRidge(0.0) \noutput_layer = train(esn, target_data, training_method)\n\noutput = esn(Generative(predict_len), output_layer)","category":"page"},{"location":"esn_tutorials/deep_esn/","page":"Deep Echo State Networks","title":"Deep Echo State Networks","text":"Plotting the results:","category":"page"},{"location":"esn_tutorials/deep_esn/","page":"Deep Echo State Networks","title":"Deep Echo State Networks","text":"using Plots\n\nts = 0.0:0.02:200.0\nlorenz_maxlyap = 0.9056\npredict_ts = ts[shift+train_len+1:shift+train_len+predict_len]\nlyap_time = (predict_ts .- predict_ts[1])*(1/lorenz_maxlyap)\n\np1 = plot(lyap_time, [test_data[1,:] output[1,:]], label = [\"actual\" \"predicted\"], \n ylabel = \"x(t)\", linewidth=2.5, xticks=false, yticks = -15:15:15);\np2 = plot(lyap_time, [test_data[2,:] output[2,:]], label = [\"actual\" \"predicted\"], \n ylabel = \"y(t)\", linewidth=2.5, xticks=false, yticks = -20:20:20);\np3 = plot(lyap_time, [test_data[3,:] output[3,:]], label = [\"actual\" \"predicted\"], \n ylabel = \"z(t)\", linewidth=2.5, xlabel = \"max(λ)*t\", yticks = 10:15:40);\n\n\nplot(p1, p2, p3, plot_title = \"Lorenz System Coordinates\", \n layout=(3,1), xtickfontsize = 12, ytickfontsize = 12, xguidefontsize=15, yguidefontsize=15,\n legendfontsize=12, titlefontsize=20)","category":"page"},{"location":"esn_tutorials/deep_esn/","page":"Deep Echo State Networks","title":"Deep Echo State Networks","text":"Note that there is a known bug at the moment with using WeightedLayer as the input layer with the deep ESN. We are in the process of investigating and solving it. The leak coefficient for the reservoirs has to always be the same in the current implementation. This is also something we are actively looking into expanding.","category":"page"},{"location":"esn_tutorials/deep_esn/#Documentation","page":"Deep Echo State Networks","title":"Documentation","text":"","category":"section"},{"location":"esn_tutorials/deep_esn/","page":"Deep Echo State Networks","title":"Deep Echo State Networks","text":"[1]: Gallicchio, Claudio, and Alessio Micheli. \"Deep echo state network (deepesn): A brief survey.\" arXiv preprint arXiv:1712.04323 (2017).","category":"page"},{"location":"general/states_variation/#Altering-States","page":"Altering States","title":"Altering States","text":"","category":"section"},{"location":"general/states_variation/","page":"Altering States","title":"Altering States","text":"In every ReservoirComputing model is possible to perform some alteration on the states in the training stage. Depending on the chosen modification, this can improve the results of the prediction. Or more simply, they can be used to reproduce results in the literature. The alterations are divided into two possibilities: the first concerns padding or extending the states, and the second concerns non-linear algorithms performed over the states.","category":"page"},{"location":"general/states_variation/#Padding-and-Extending-States","page":"Altering States","title":"Padding and Extending States","text":"","category":"section"},{"location":"general/states_variation/","page":"Altering States","title":"Altering States","text":"Extending the states means appending to them the corresponding input values. If textbfx(t) is the reservoir state at time t corresponding to the input textbfu(t) the extended state will be represented as textbfx(t) textbfu(t) where is intended as vertical concatenation. This procedure is, for example, used in Jaeger's Scholarpedia description of Echo State Networks. The extension of the states can be obtained in every ReservoirComputing.jl model by using the keyword argument states_type and calling the method ExtendedStates(). No argument is needed.","category":"page"},{"location":"general/states_variation/","page":"Altering States","title":"Altering States","text":"Padding the states means appending a constant value, 1.0 for example, to each state. Using the notation introduced before, we can define the padded states as textbfx(t) 10. This approach is detailed in the seminal guide to Echo State Networks by Mantas Lukoševičius. By using the keyword argument states_type the user can call the method PaddedStates(padding) where padding represents the value that will be concatenated to the states. As default, the value is set to unity, so the majority of the time, calling PaddedStates() will suffice.","category":"page"},{"location":"general/states_variation/","page":"Altering States","title":"Altering States","text":"Though not easily found in the literature, it is also possible to pad the extended states by using the method PaddedExtendedStates(padding) that has unity as padding default as well.","category":"page"},{"location":"general/states_variation/","page":"Altering States","title":"Altering States","text":"Of course, it is also possible to not apport any of these changes to the states by calling StandardStates(). This is also the default choice for the states.","category":"page"},{"location":"general/states_variation/#Non-Linear-Algorithms","page":"Altering States","title":"Non-Linear Algorithms","text":"","category":"section"},{"location":"general/states_variation/","page":"Altering States","title":"Altering States","text":"First introduced in [1] and expanded in [2] these are nonlinear combinations of the columns of the matrix states. There are three such algorithms implemented. Using the keyword argument nla_type it is possible to choose in every model in ReservoirComputing.jl the specific non-linear algorithm to use. The default value is set to NLADefault(), where no non-linear algorithm takes place.","category":"page"},{"location":"general/states_variation/","page":"Altering States","title":"Altering States","text":"Following the nomenclature used in [2], the algorithms can be called as NLAT1(), NLAT2() and NLAT3(). To better explain what they do, let textbfx_i j be elements of the state matrix, with i=1T j=1N where T is the length of the training and N is the reservoir size. ","category":"page"},{"location":"general/states_variation/","page":"Altering States","title":"Altering States","text":"NLAT1","category":"page"},{"location":"general/states_variation/","page":"Altering States","title":"Altering States","text":"tildetextbfx_ij = textbfx_ij times textbfx_ij textif textitj is odd \ntildetextbfx_ij = textbfx_ij textif textitj is even","category":"page"},{"location":"general/states_variation/","page":"Altering States","title":"Altering States","text":"NLAT2","category":"page"},{"location":"general/states_variation/","page":"Altering States","title":"Altering States","text":"tildetextbfx_ij = textbfx_ij-1 times textbfx_ij-2 textif textitj 1 is odd \ntildetextbfx_ij = textbfx_ij textif textitj is 1 or even","category":"page"},{"location":"general/states_variation/","page":"Altering States","title":"Altering States","text":"NLAT3","category":"page"},{"location":"general/states_variation/","page":"Altering States","title":"Altering States","text":"tildetextbfx_ij = textbfx_ij-1 times textbfx_ij+1 textif textitj 1 is odd \ntildetextbfx_ij = textbfx_ij textif textitj is 1 or even","category":"page"},{"location":"general/states_variation/","page":"Altering States","title":"Altering States","text":"[1]: Pathak, Jaideep, et al. \"Using machine learning to replicate chaotic attractors and calculate Lyapunov exponents from data.\" Chaos: An Interdisciplinary Journal of Nonlinear Science 27.12 (2017): 121102.","category":"page"},{"location":"general/states_variation/","page":"Altering States","title":"Altering States","text":"[2]: Chattopadhyay, Ashesh, Pedram Hassanzadeh, and Devika Subramanian. \"Data-driven predictions of a multiscale Lorenz 96 chaotic system using machine-learning methods: reservoir computing, artificial neural network, and long short-term memory network.\" Nonlinear Processes in Geophysics 27.3 (2020): 373-389.","category":"page"},{"location":"esn_tutorials/different_training/#Using-Different-Training-Methods","page":"Using Different Training Methods","title":"Using Different Training Methods","text":"","category":"section"},{"location":"esn_tutorials/different_training/#Linear-Methods","page":"Using Different Training Methods","title":"Linear Methods","text":"","category":"section"},{"location":"esn_tutorials/different_training/#Echo-State-Gaussian-Processes","page":"Using Different Training Methods","title":"Echo State Gaussian Processes","text":"","category":"section"},{"location":"esn_tutorials/different_training/#Support-Vector-Echo-State-Machines","page":"Using Different Training Methods","title":"Support Vector Echo State Machines","text":"","category":"section"},{"location":"api/predict/#Prediction-Types","page":"Prediction Types","title":"Prediction Types","text":"","category":"section"},{"location":"api/predict/","page":"Prediction Types","title":"Prediction Types","text":" Generative\n Predictive","category":"page"},{"location":"api/predict/#ReservoirComputing.Generative","page":"Prediction Types","title":"ReservoirComputing.Generative","text":"Generative(prediction_len)\n\nThis prediction methodology allows the models to produce an autonomous prediction, feeding the prediction into itself to generate the next step. The only parameter needed is the number of steps for the prediction.\n\n\n\n\n\n","category":"type"},{"location":"api/predict/#ReservoirComputing.Predictive","page":"Prediction Types","title":"ReservoirComputing.Predictive","text":"Predictive(prediction_data)\n\nGiven a set of labels as prediction_data, this method of prediction will return the corresponding labels in a standard Machine Learning fashion.\n\n\n\n\n\n","category":"type"},{"location":"api/reca/#Reservoir-Computing-with-Cellular-Automata","page":"ReCA","title":"Reservoir Computing with Cellular Automata","text":"","category":"section"},{"location":"api/reca/","page":"ReCA","title":"ReCA","text":" RECA","category":"page"},{"location":"api/reca/#ReservoirComputing.RECA","page":"ReCA","title":"ReservoirComputing.RECA","text":"RECA(train_data,\n automata;\n generations = 8,\n input_encoding=RandomMapping(),\n nla_type = NLADefault(),\n states_type = StandardStates())\n\n[1] Yilmaz, Ozgur. “Reservoir computing using cellular automata.” arXiv preprint arXiv:1410.0162 (2014).\n\n[2] Nichele, Stefano, and Andreas Molund. “Deep reservoir computing using cellular automata.” arXiv preprint arXiv:1703.02806 (2017).\n\n\n\n\n\n","category":"type"},{"location":"api/reca/","page":"ReCA","title":"ReCA","text":"The input encodings are the equivalent of the input matrices of the ESNs. These are the available encodings:","category":"page"},{"location":"api/reca/","page":"ReCA","title":"ReCA","text":" RandomMapping","category":"page"},{"location":"api/reca/#ReservoirComputing.RandomMapping","page":"ReCA","title":"ReservoirComputing.RandomMapping","text":"RandomMapping(permutations, expansion_size)\nRandomMapping(permutations; expansion_size=40)\nRandomMapping(;permutations=8, expansion_size=40)\n\nRandom mapping of the input data directly in the reservoir. The expansion_size determines the dimension of the single reservoir, and permutations determines the number of total reservoirs that will be connected, each with a different mapping. The detail of this implementation can be found in [1].\n\n[1] Nichele, Stefano, and Andreas Molund. “Deep reservoir computing using cellular automata.” arXiv preprint arXiv:1703.02806 (2017).\n\n\n\n\n\n","category":"type"},{"location":"api/reca/","page":"ReCA","title":"ReCA","text":"The training and prediction follow the same workflow as the ESN. It is important to note that currently we were unable to find any papers using these models with a Generative approach for the prediction, so full support is given only to the Predictive method.","category":"page"},{"location":"esn_tutorials/change_layers/#Using-Different-Layers","page":"Using Different Layers","title":"Using Different Layers","text":"","category":"section"},{"location":"esn_tutorials/change_layers/","page":"Using Different Layers","title":"Using Different Layers","text":"A great deal of effort in the ESNs field is devoted to finding the ideal construction for the reservoir matrices. With a simple interface using ReservoirComputing.jl it is possible to leverage the currently implemented matrix construction methods for both the reservoir and the input layer. On this page, it is showcased how it is possible to change both of these layers.","category":"page"},{"location":"esn_tutorials/change_layers/","page":"Using Different Layers","title":"Using Different Layers","text":"The input_init keyword argument provided with the ESN constructor allows for changing the input layer. The layers provided in ReservoirComputing.jl are the following:","category":"page"},{"location":"esn_tutorials/change_layers/","page":"Using Different Layers","title":"Using Different Layers","text":"WeightedLayer(scaling)\nDenseLayer(scaling)\nSparseLayer(scaling, sparsity)\nMinimumLayer(weight, sampling)\nInformedLayer(model_in_size; scaling=0.1, gamma=0.5)","category":"page"},{"location":"esn_tutorials/change_layers/","page":"Using Different Layers","title":"Using Different Layers","text":"In addition, the user can define a custom layer following this workflow:","category":"page"},{"location":"esn_tutorials/change_layers/","page":"Using Different Layers","title":"Using Different Layers","text":"#creation of the new struct for the layer\nstruct MyNewLayer <: AbstractLayer\n #the layer params go here\nend\n\n#dispatch over the function to actually build the layer matrix\nfunction create_layer(input_layer::MyNewLayer, res_size, in_size)\n #the new algorithm to build the input layer goes here\nend","category":"page"},{"location":"esn_tutorials/change_layers/","page":"Using Different Layers","title":"Using Different Layers","text":"Similarly the reservoir_init keyword argument provides the possibility to change the construction for the reservoir matrix. The available reservoir are:","category":"page"},{"location":"esn_tutorials/change_layers/","page":"Using Different Layers","title":"Using Different Layers","text":"RandSparseReservoir(res_size, radius, sparsity)\nPseudoSVDReservoir(res_size, max_value, sparsity, sorted, reverse_sort)\nDelayLineReservoir(res_size, weight)\nDelayLineBackwardReservoir(res_size, weight, fb_weight)\nSimpleCycleReservoir(res_size, weight)\nCycleJumpsReservoir(res_size, cycle_weight, jump_weight, jump_size)","category":"page"},{"location":"esn_tutorials/change_layers/","page":"Using Different Layers","title":"Using Different Layers","text":"And, like before, it is possible to build a custom reservoir by following this workflow:","category":"page"},{"location":"esn_tutorials/change_layers/","page":"Using Different Layers","title":"Using Different Layers","text":"#creation of the new struct for the reservoir\nstruct MyNewReservoir <: AbstractReservoir\n #the reservoir params go here\nend\n\n#dispatch over the function to build the reservoir matrix\nfunction create_reservoir(reservoir::AbstractReservoir, res_size)\n #the new algorithm to build the reservoir matrix goes here\nend","category":"page"},{"location":"esn_tutorials/change_layers/#Example-of-a-minimally-complex-ESN","page":"Using Different Layers","title":"Example of a minimally complex ESN","text":"","category":"section"},{"location":"esn_tutorials/change_layers/","page":"Using Different Layers","title":"Using Different Layers","text":"Using [1] and [2] as references, this section will provide an example of how to change both the input layer and the reservoir for ESNs. The full script for this example can be found here. This example was run on Julia v1.7.2.","category":"page"},{"location":"esn_tutorials/change_layers/","page":"Using Different Layers","title":"Using Different Layers","text":"The task for this example will be the one step ahead prediction of the Henon map. To obtain the data, one can leverage the package DynamicalSystems.jl. The data is scaled to be between -1 and 1.","category":"page"},{"location":"esn_tutorials/change_layers/","page":"Using Different Layers","title":"Using Different Layers","text":"using PredefinedDynamicalSystems\ntrain_len = 3000\npredict_len = 2000\n\nds = PredefinedDynamicalSystems.henon()\ntraj, time = trajectory(ds, 7000)\ndata = Matrix(traj)'\ndata = (data .-0.5) .* 2\nshift = 200\n\ntraining_input = data[:, shift:shift+train_len-1]\ntraining_target = data[:, shift+1:shift+train_len]\ntesting_input = data[:,shift+train_len:shift+train_len+predict_len-1]\ntesting_target = data[:,shift+train_len+1:shift+train_len+predict_len]","category":"page"},{"location":"esn_tutorials/change_layers/","page":"Using Different Layers","title":"Using Different Layers","text":"Now it is possible to define the input layers and reservoirs we want to compare and run the comparison in a simple for loop. The accuracy will be tested using the mean squared deviation msd from StatsBase.","category":"page"},{"location":"esn_tutorials/change_layers/","page":"Using Different Layers","title":"Using Different Layers","text":"using ReservoirComputing, StatsBase\n\nres_size = 300\ninput_layer = [MinimumLayer(0.85, IrrationalSample()), MinimumLayer(0.95, IrrationalSample())]\nreservoirs = [SimpleCycleReservoir(res_size, 0.7), \n CycleJumpsReservoir(res_size, cycle_weight=0.7, jump_weight=0.2, jump_size=5)]\n\nfor i=1:length(reservoirs)\n esn = ESN(training_input;\n input_layer = input_layer[i],\n reservoir = reservoirs[i])\n wout = train(esn, training_target, StandardRidge(0.001))\n output = esn(Predictive(testing_input), wout)\n println(msd(testing_target, output))\nend","category":"page"},{"location":"esn_tutorials/change_layers/","page":"Using Different Layers","title":"Using Different Layers","text":"As it is possible to see, changing layers in ESN models is straightforward. Be sure to check the API documentation for a full list of reservoirs and layers.","category":"page"},{"location":"esn_tutorials/change_layers/#Bibliography","page":"Using Different Layers","title":"Bibliography","text":"","category":"section"},{"location":"esn_tutorials/change_layers/","page":"Using Different Layers","title":"Using Different Layers","text":"[1]: Rodan, Ali, and Peter Tiňo. “Simple deterministically constructed cycle reservoirs with regular jumps.” Neural computation 24.7 (2012): 1822-1852.","category":"page"},{"location":"esn_tutorials/change_layers/","page":"Using Different Layers","title":"Using Different Layers","text":"[2]: Rodan, Ali, and Peter Tiňo. “Minimum complexity echo state network.” IEEE transactions on neural networks 22.1 (2010): 131-144.","category":"page"},{"location":"general/different_training/#Changing-Training-Algorithms","page":"Changing Training Algorithms","title":"Changing Training Algorithms","text":"","category":"section"},{"location":"general/different_training/","page":"Changing Training Algorithms","title":"Changing Training Algorithms","text":"Notably Echo State Networks have been trained with Ridge Regression algorithms, but the range of useful algorithms to use is much greater. In this section of the documentation, it is possible to explore how to use other training methods to obtain the readout layer. All the methods implemented in ReservoirComputing.jl can be used for all models in the library, not only ESNs. The general workflow illustrated in this section will be based on a dummy RC model my_model = MyModel(...) that needs training to obtain the readout layer. The training is done as follows:","category":"page"},{"location":"general/different_training/","page":"Changing Training Algorithms","title":"Changing Training Algorithms","text":"training_algo = TrainingAlgo()\nreadout_layer = train(my_model, train_data, training_algo)","category":"page"},{"location":"general/different_training/","page":"Changing Training Algorithms","title":"Changing Training Algorithms","text":"In this section, it is possible to explore how to properly build the training_algo and all the possible choices available. In the example section of the documentation it will be provided copy-pasteable code to better explore the training algorithms and their impact on the model.","category":"page"},{"location":"general/different_training/#Linear-Models","page":"Changing Training Algorithms","title":"Linear Models","text":"","category":"section"},{"location":"general/different_training/","page":"Changing Training Algorithms","title":"Changing Training Algorithms","text":"The library includes a standard implementation of ridge regression, callable using StandardRidge(regularization_coeff) where the default value for the regularization coefficient is set to zero. This is also the default model called when no model is specified in train(). This makes the default call for training train(my_model, train_data) use Ordinary Least Squares (OLS) for regression.","category":"page"},{"location":"general/different_training/","page":"Changing Training Algorithms","title":"Changing Training Algorithms","text":"Leveraging MLJLinearModels it is possible to expand the choices of linear models used for the training. The wrappers provided are structured in the following way:","category":"page"},{"location":"general/different_training/","page":"Changing Training Algorithms","title":"Changing Training Algorithms","text":"struct LinearModel\n regression\n solver\n regression_kwargs\nend","category":"page"},{"location":"general/different_training/","page":"Changing Training Algorithms","title":"Changing Training Algorithms","text":"to call the ridge regression using the MLJLinearModels APIs, one can use LinearModel(;regression=LinearRegression). It is also possible to use a specific solver, by calling LinearModel(regression=LinearRegression, solver=Analytical()). For all the available solvers, please refer to the MLJLinearModels documentation. To change the regularization coefficient in the ridge example, using for example lambda = 0.1, it is needed to pass it in the regression_kwargs like so LinearModel(;regression=LinearRegression, solver=Analytical(), regression_kwargs=(lambda=lambda)). The nomenclature of the coefficients must follow the MLJLinearModels APIs, using lambda, gamma for LassoRegression and delta, lambda, gamma for HuberRegression. Again, please check the relevant documentation if in doubt. When using MLJLinearModels based regressors, do remember to specify using MLJLinearModels.","category":"page"},{"location":"general/different_training/#Gaussian-Processes","page":"Changing Training Algorithms","title":"Gaussian Processes","text":"","category":"section"},{"location":"general/different_training/","page":"Changing Training Algorithms","title":"Changing Training Algorithms","text":"Another way to obtain the readout layer is possible using Gaussian regression. This is provided through a wrapper of GaussianProcesses structured in the following way:","category":"page"},{"location":"general/different_training/","page":"Changing Training Algorithms","title":"Changing Training Algorithms","text":"struct GaussianProcess\n mean\n kernel\n lognoise\n optimize\n optimizer\nend","category":"page"},{"location":"general/different_training/","page":"Changing Training Algorithms","title":"Changing Training Algorithms","text":"While it is necessary to specify a mean and a kernel, the other defaults are lognoise=-2, optimize=false, optimizer=Optim.LBFGS(). For the choice of means and kernels, please refer to the proper documentation, here and here, respectively. ","category":"page"},{"location":"general/different_training/","page":"Changing Training Algorithms","title":"Changing Training Algorithms","text":"Building on the simple example given in the GaussianProcesses documentation, it is possible to build an intuition of how to use these algorithms for training ReservoirComputing.jl models.","category":"page"},{"location":"general/different_training/","page":"Changing Training Algorithms","title":"Changing Training Algorithms","text":"mZero = MeanZero() #Zero mean function\nkern = SE(0.0,0.0) #Squared exponential kernel (note that hyperparameters are on the log scale)\nlogObsNoise = -1.0\n\ngp = GaussianProcess(mZero, kern, lognoise=logObsNoise)","category":"page"},{"location":"general/different_training/","page":"Changing Training Algorithms","title":"Changing Training Algorithms","text":"Like in the previous case, if one uses GaussianProcesses based regressors, it is necessary to specify using GaussianProcesses. Additionally, if the optimizer chosen is from an external package, i.e. Optim, that package needs to be used in the script as well by adding using Optim.","category":"page"},{"location":"general/different_training/#Support-Vector-Regression","page":"Changing Training Algorithms","title":"Support Vector Regression","text":"","category":"section"},{"location":"general/different_training/","page":"Changing Training Algorithms","title":"Changing Training Algorithms","text":"Contrary to the LinearModels and GaussianProcesses, no wrappers are needed for support vector regression. By using LIBSVM.jl, LIBSVM wrappers in Julia, it is possible to call both epsilonSVR() or nuSVR() directly in train(). For the full range of kernels provided and the parameters to call, we refer the user to the official documentation. Like before, if one intends to use LIBSVM regressors, it is necessary to specify using LIBSVM.","category":"page"},{"location":"api/states/#States-Modifications","page":"States Modifications","title":"States Modifications","text":"","category":"section"},{"location":"api/states/#Padding-and-Estension","page":"States Modifications","title":"Padding and Estension","text":"","category":"section"},{"location":"api/states/","page":"States Modifications","title":"States Modifications","text":" StandardStates\n ExtendedStates\n PaddedStates\n PaddedExtendedStates","category":"page"},{"location":"api/states/#ReservoirComputing.StandardStates","page":"States Modifications","title":"ReservoirComputing.StandardStates","text":"StandardStates()\n\nNo modification of the states takes place, default option.\n\n\n\n\n\n","category":"type"},{"location":"api/states/#ReservoirComputing.ExtendedStates","page":"States Modifications","title":"ReservoirComputing.ExtendedStates","text":"ExtendedStates()\n\nThe states are extended with the input data, for the training section, and the prediction data, during the prediction section. This is obtained with a vertical concatenation of the data and the states.\n\n\n\n\n\n","category":"type"},{"location":"api/states/#ReservoirComputing.PaddedStates","page":"States Modifications","title":"ReservoirComputing.PaddedStates","text":"PaddedStates(padding)\nPaddedStates(;padding=1.0)\n\nThe states are padded with a chosen value. Usually, this value is set to one. The padding is obtained through a vertical concatenation of the padding value and the states.\n\n\n\n\n\n","category":"type"},{"location":"api/states/#ReservoirComputing.PaddedExtendedStates","page":"States Modifications","title":"ReservoirComputing.PaddedExtendedStates","text":"PaddedExtendedStates(padding)\nPaddedExtendedStates(;padding=1.0)\n\nThe states are extended with the training data or predicted data and subsequently padded with a chosen value. Usually, the padding value is set to one. The padding and the extension are obtained through a vertical concatenation of the padding value, the data, and the states.\n\n\n\n\n\n","category":"type"},{"location":"api/states/#Non-Linear-Transformations","page":"States Modifications","title":"Non Linear Transformations","text":"","category":"section"},{"location":"api/states/","page":"States Modifications","title":"States Modifications","text":" NLADefault\n NLAT1\n NLAT2\n NLAT3","category":"page"},{"location":"api/states/#ReservoirComputing.NLADefault","page":"States Modifications","title":"ReservoirComputing.NLADefault","text":"NLADefault()\n\nReturns the array untouched, default option.\n\n\n\n\n\n","category":"type"},{"location":"api/states/#ReservoirComputing.NLAT1","page":"States Modifications","title":"ReservoirComputing.NLAT1","text":"NLAT1()\n\nApplies the $ \\text{T}_1 $ transformation algorithm, as defined in [1] and [2].\n\n[1] Chattopadhyay, Ashesh, et al. \"Data-driven prediction of a multi-scale Lorenz 96 chaotic system using a hierarchy of deep learning methods: Reservoir computing, ANN, and RNN-LSTM.\" (2019).\n\n[2] Pathak, Jaideep, et al. \"Model-free prediction of large spatiotemporally chaotic systems from data: A reservoir computing approach.\" Physical review letters 120.2 (2018): 024102.\n\n\n\n\n\n","category":"type"},{"location":"api/states/#ReservoirComputing.NLAT2","page":"States Modifications","title":"ReservoirComputing.NLAT2","text":"NLAT2()\n\nApply the $ \\text{T}_2 $ transformation algorithm, as defined in [1].\n\n[1] Chattopadhyay, Ashesh, et al. \"Data-driven prediction of a multi-scale Lorenz 96 chaotic system using a hierarchy of deep learning methods: Reservoir computing, ANN, and RNN-LSTM.\" (2019).\n\n\n\n\n\n","category":"type"},{"location":"api/states/#ReservoirComputing.NLAT3","page":"States Modifications","title":"ReservoirComputing.NLAT3","text":"NLAT3()\n\nApply the $ \\text{T}_3 $ transformation algorithm, as defined in [1].\n\n[1] Chattopadhyay, Ashesh, et al. \"Data-driven prediction of a multi-scale Lorenz 96 chaotic system using a hierarchy of deep learning methods: Reservoir computing, ANN, and RNN-LSTM.\" (2019).\n\n\n\n\n\n","category":"type"},{"location":"api/esn_drivers/#ESN-Drivers","page":"ESN Drivers","title":"ESN Drivers","text":"","category":"section"},{"location":"api/esn_drivers/","page":"ESN Drivers","title":"ESN Drivers","text":" RNN\n MRNN\n GRU","category":"page"},{"location":"api/esn_drivers/#ReservoirComputing.RNN","page":"ESN Drivers","title":"ReservoirComputing.RNN","text":"RNN(activation_function, leaky_coefficient)\nRNN(;activation_function=tanh, leaky_coefficient=1.0)\n\nReturns a Recurrent Neural Network initializer for the ESN. This is the default choice.\n\n\n\n\n\n","category":"type"},{"location":"api/esn_drivers/#ReservoirComputing.MRNN","page":"ESN Drivers","title":"ReservoirComputing.MRNN","text":"MRNN(activation_function, leaky_coefficient, scaling_factor)\nMRNN(;activation_function=[tanh, sigmoid], leaky_coefficient=1.0,\n scaling_factor=fill(leaky_coefficient, length(activation_function)))\n\nReturns a Multiple RNN initializer, where multiple functions are combined in a linear combination with chosen parameters scaling_factor. The activation_function and scaling_factor arguments must be vectors of the same size. Multiple combinations are possible. The implementation is based upon the double activation function idea, found in [1].\n\n[1] Lun, Shu-Xian, et al. \"A novel model of leaky integrator echo state network for time-series prediction.\" Neurocomputing 159 (2015): 58-66.\n\n\n\n\n\n","category":"type"},{"location":"api/esn_drivers/#ReservoirComputing.GRU","page":"ESN Drivers","title":"ReservoirComputing.GRU","text":"GRU(;activation_function=[NNlib.sigmoid, NNlib.sigmoid, tanh],\n inner_layer = fill(DenseLayer(), 2),\n reservoir = fill(RandSparseReservoir(), 2),\n bias = fill(DenseLayer(), 2),\n variant = FullyGated())\n\nReturns a Gated Recurrent Unit [1] reservoir driver.\n\n[1] Cho, Kyunghyun, et al. “Learning phrase representations using RNN encoder-decoder for statistical machine translation.” arXiv preprint arXiv:1406.1078 (2014).\n\n\n\n\n\n","category":"type"},{"location":"api/esn_drivers/","page":"ESN Drivers","title":"ESN Drivers","text":"The GRU driver also provides the user with the choice of the possible variants:","category":"page"},{"location":"api/esn_drivers/","page":"ESN Drivers","title":"ESN Drivers","text":" FullyGated\n Minimal","category":"page"},{"location":"api/esn_drivers/#ReservoirComputing.FullyGated","page":"ESN Drivers","title":"ReservoirComputing.FullyGated","text":"FullyGated()\n\nReturns a standard Gated Recurrent Unit ESN initializer, as described in [1].\n\n[1] Cho, Kyunghyun, et al. “Learning phrase representations using RNN encoder-decoder for statistical machine translation.” arXiv preprint arXiv:1406.1078 (2014).\n\n\n\n\n\n","category":"type"},{"location":"api/esn_drivers/#ReservoirComputing.Minimal","page":"ESN Drivers","title":"ReservoirComputing.Minimal","text":"Minimal()\n\nReturns a minimal GRU ESN initializer as described in [1].\n\n[1] Zhou, Guo-Bing, et al. \"Minimal gated unit for recurrent neural networks.\" International Journal of Automation and Computing 13.3 (2016): 226-234.\n\n\n\n\n\n","category":"type"},{"location":"api/esn_drivers/","page":"ESN Drivers","title":"ESN Drivers","text":"Please refer to the original papers for more detail about these architectures.","category":"page"},{"location":"api/training/#Training-Algorithms","page":"Training Algorithms","title":"Training Algorithms","text":"","category":"section"},{"location":"api/training/#Linear-Models","page":"Training Algorithms","title":"Linear Models","text":"","category":"section"},{"location":"api/training/","page":"Training Algorithms","title":"Training Algorithms","text":" StandardRidge\n LinearModel","category":"page"},{"location":"api/training/#ReservoirComputing.StandardRidge","page":"Training Algorithms","title":"ReservoirComputing.StandardRidge","text":"StandardRidge(regularization_coeff)\nStandardRidge(;regularization_coeff=0.0)\n\nRidge regression training for all the models in the library. The regularization_coeff is the regularization, it can be passed as an arg or kwarg.\n\n\n\n\n\n","category":"type"},{"location":"api/training/#ReservoirComputing.LinearModel","page":"Training Algorithms","title":"ReservoirComputing.LinearModel","text":"LinearModel(;regression=LinearRegression, \n solver=Analytical(), \n regression_kwargs=(;))\n\nLinear regression training based on MLJLinearModels for all the models in the library. All the parameters have to be passed into regression_kwargs, apart from the solver choice. MLJLinearModels.jl needs to be called in order to use these models.\n\n\n\n\n\n","category":"type"},{"location":"api/training/#Gaussian-Regression","page":"Training Algorithms","title":"Gaussian Regression","text":"","category":"section"},{"location":"api/training/","page":"Training Algorithms","title":"Training Algorithms","text":"Currently, v0.9 is unavailable.","category":"page"},{"location":"api/training/#Support-Vector-Regression","page":"Training Algorithms","title":"Support Vector Regression","text":"","category":"section"},{"location":"api/training/","page":"Training Algorithms","title":"Training Algorithms","text":"Support Vector Regression is possible using a direct call to LIBSVM regression methods. Instead of a wrapper, please refer to the use of LIBSVM.AbstractSVR in the original library.","category":"page"},{"location":"api/esn_layers/#ESN-Layers","page":"ESN Layers","title":"ESN Layers","text":"","category":"section"},{"location":"api/esn_layers/#Input-Layers","page":"ESN Layers","title":"Input Layers","text":"","category":"section"},{"location":"api/esn_layers/","page":"ESN Layers","title":"ESN Layers","text":" WeightedLayer\n DenseLayer\n SparseLayer\n InformedLayer\n MinimumLayer\n NullLayer","category":"page"},{"location":"api/esn_layers/#ReservoirComputing.WeightedLayer","page":"ESN Layers","title":"ReservoirComputing.WeightedLayer","text":"WeightedInput(scaling)\nWeightedInput(;scaling=0.1)\n\nReturns a weighted layer initializer object, that will produce a weighted input matrix with random non-zero elements drawn from [-scaling, scaling], as described in [1]. The scaling factor can be given as arg or kwarg.\n\n[1] Lu, Zhixin, et al. \"Reservoir observers: Model-free inference of unmeasured variables in chaotic systems.\" Chaos: An Interdisciplinary Journal of Nonlinear Science 27.4 (2017): 041102.\n\n\n\n\n\n","category":"type"},{"location":"api/esn_layers/#ReservoirComputing.DenseLayer","page":"ESN Layers","title":"ReservoirComputing.DenseLayer","text":"DenseLayer(scaling)\nDenseLayer(;scaling=0.1)\n\nReturns a fully connected layer initializer object, that will produce a weighted input matrix with random non-zero elements drawn from [-scaling, scaling]. The scaling factor can be given as arg or kwarg. This is the default choice in the ESN construction.\n\n\n\n\n\n","category":"type"},{"location":"api/esn_layers/#ReservoirComputing.SparseLayer","page":"ESN Layers","title":"ReservoirComputing.SparseLayer","text":"SparseLayer(scaling, sparsity)\nSparseLayer(scaling; sparsity=0.1)\nSparseLayer(;scaling=0.1, sparsity=0.1)\n\nReturns a sparsely connected layer initializer object, that will produce a random sparse input matrix with random non-zero elements drawn from [-scaling, scaling] and given sparsity. The scaling and sparsity factors can be given as args or kwargs.\n\n\n\n\n\n","category":"type"},{"location":"api/esn_layers/#ReservoirComputing.InformedLayer","page":"ESN Layers","title":"ReservoirComputing.InformedLayer","text":"InformedLayer(model_in_size; scaling=0.1, gamma=0.5)\n\nReturns a weighted input layer matrix, with random non-zero elements drawn from [-scaling, scaling], where some γ of reservoir nodes are connected exclusively to the raw inputs, and the rest to the outputs of the prior knowledge model, as described in [1].\n\n[1] Jaideep Pathak et al. \"Hybrid Forecasting of Chaotic Processes: Using Machine Learning in Conjunction with a Knowledge-Based Model\" (2018)\n\n\n\n\n\n","category":"type"},{"location":"api/esn_layers/#ReservoirComputing.MinimumLayer","page":"ESN Layers","title":"ReservoirComputing.MinimumLayer","text":"MinimumLayer(weight, sampling)\nMinimumLayer(weight; sampling=BernoulliSample(0.5))\nMinimumLayer(;weight=0.1, sampling=BernoulliSample(0.5))\n\nReturns a fully connected layer initializer object. The matrix constructed with this initializer presents the same absolute weight value, decided by the weight factor. The sign of each entry is decided by the sampling struct. Construction detailed in [1] and [2].\n\n[1] Rodan, Ali, and Peter Tino. \"Minimum complexity echo state network.\" IEEE transactions on neural networks 22.1 (2010): 131-144. [2] Rodan, Ali, and Peter Tiňo. \"Simple deterministically constructed cycle reservoirs with regular jumps.\" Neural computation 24.7 (2012): 1822-1852.\n\n\n\n\n\n","category":"type"},{"location":"api/esn_layers/#ReservoirComputing.NullLayer","page":"ESN Layers","title":"ReservoirComputing.NullLayer","text":"NullLayer(model_in_size; scaling=0.1, gamma=0.5)\n\nReturns a vector of zeros.\n\n\n\n\n\n","category":"type"},{"location":"api/esn_layers/","page":"ESN Layers","title":"ESN Layers","text":"The signs in the MinimumLayer are chosen based on the following methods:","category":"page"},{"location":"api/esn_layers/","page":"ESN Layers","title":"ESN Layers","text":" BernoulliSample\n IrrationalSample","category":"page"},{"location":"api/esn_layers/#ReservoirComputing.BernoulliSample","page":"ESN Layers","title":"ReservoirComputing.BernoulliSample","text":"BernoulliSample(p)\nBernoulliSample(;p=0.5)\n\nReturns a Bernoulli sign constructor for the MinimumLayer call. The p factor determines the probability of the result, as in the Distributions call. The value can be passed as an arg or kwarg. This sign weight determination for input layers is introduced in [1].\n\n[1] Rodan, Ali, and Peter Tino. \"Minimum complexity echo state network.\" IEEE transactions on neural networks 22.1 (2010): 131-144.\n\n\n\n\n\n","category":"type"},{"location":"api/esn_layers/#ReservoirComputing.IrrationalSample","page":"ESN Layers","title":"ReservoirComputing.IrrationalSample","text":"IrrationalSample(irrational, start)\nIrrationalSample(;irrational=pi, start=1)\n\nReturns an irrational sign constructor for the MinimumLayer call. The values can be passed as args or kwargs. The sign of the weight is decided from the decimal expansion of the given irrational. The first start decimal digits are thresholded at 4.5, then the n-th input sign will be + and - respectively.\n\n[1] Rodan, Ali, and Peter Tiňo. \"Simple deterministically constructed cycle reservoirs with regular jumps.\" Neural computation 24.7 (2012): 1822-1852.\n\n\n\n\n\n","category":"type"},{"location":"api/esn_layers/","page":"ESN Layers","title":"ESN Layers","text":"To derive the matrix one can call the following function:","category":"page"},{"location":"api/esn_layers/","page":"ESN Layers","title":"ESN Layers","text":" create_layer","category":"page"},{"location":"api/esn_layers/#ReservoirComputing.create_layer","page":"ESN Layers","title":"ReservoirComputing.create_layer","text":"create_layer(input_layer::AbstractLayer, res_size, in_size)\n\nReturns a res_size times in_size matrix layer, built according to the input_layer constructor.\n\n\n\n\n\n","category":"function"},{"location":"api/esn_layers/","page":"ESN Layers","title":"ESN Layers","text":"To create new input layers, it suffices to define a new struct containing the needed parameters of the new input layer. This struct will need to be an AbstractLayer, so the create_layer function can be dispatched over it. The workflow should follow this snippet:","category":"page"},{"location":"api/esn_layers/","page":"ESN Layers","title":"ESN Layers","text":"#creation of the new struct for the layer\nstruct MyNewLayer <: AbstractLayer\n #the layer params go here\nend\n\n#dispatch over the function to actually build the layer matrix\nfunction create_layer(input_layer::MyNewLayer, res_size, in_size)\n #the new algorithm to build the input layer goes here\nend","category":"page"},{"location":"api/esn_layers/#Reservoirs","page":"ESN Layers","title":"Reservoirs","text":"","category":"section"},{"location":"api/esn_layers/","page":"ESN Layers","title":"ESN Layers","text":" RandSparseReservoir\n PseudoSVDReservoir\n DelayLineReservoir\n DelayLineBackwardReservoir\n SimpleCycleReservoir\n CycleJumpsReservoir\n NullReservoir","category":"page"},{"location":"api/esn_layers/#ReservoirComputing.RandSparseReservoir","page":"ESN Layers","title":"ReservoirComputing.RandSparseReservoir","text":"RandSparseReservoir(res_size, radius, sparsity)\nRandSparseReservoir(res_size; radius=1.0, sparsity=0.1)\n\nReturns a random sparse reservoir initializer, that will return a matrix with given sparsity and scaled spectral radius according to radius. This is the default choice in the ESN construction.\n\n\n\n\n\n","category":"type"},{"location":"api/esn_layers/#ReservoirComputing.PseudoSVDReservoir","page":"ESN Layers","title":"ReservoirComputing.PseudoSVDReservoir","text":"PseudoSVDReservoir(max_value, sparsity, sorted, reverse_sort)\nPseudoSVDReservoir(max_value, sparsity; sorted=true, reverse_sort=false)\n\nReturns an initializer to build a sparse reservoir matrix, with given sparsity created using SVD as described in [1].\n\n[1] Yang, Cuili, et al. \"Design of polynomial echo state networks for time series prediction.\" Neurocomputing 290 (2018): 148-160.\n\n\n\n\n\n","category":"type"},{"location":"api/esn_layers/#ReservoirComputing.DelayLineReservoir","page":"ESN Layers","title":"ReservoirComputing.DelayLineReservoir","text":"DelayLineReservoir(res_size, weight)\nDelayLineReservoir(res_size; weight=0.1)\n\nReturns a Delay Line Reservoir matrix constructor to obtain a deterministic reservoir as described in [1]. The weight can be passed as arg or kwarg, and it determines the absolute value of all the connections in the reservoir.\n\n[1] Rodan, Ali, and Peter Tino. \"Minimum complexity echo state network.\" IEEE transactions on neural networks 22.1 (2010): 131-144.\n\n\n\n\n\n","category":"type"},{"location":"api/esn_layers/#ReservoirComputing.DelayLineBackwardReservoir","page":"ESN Layers","title":"ReservoirComputing.DelayLineBackwardReservoir","text":"DelayLineBackwardReservoir(res_size, weight, fb_weight)\nDelayLineBackwardReservoir(res_size; weight=0.1, fb_weight=0.2)\n\nReturns a Delay Line Reservoir constructor to create a matrix with Backward connections as described in [1]. The weight and fb_weight can be passed as either args or kwargs, and they determine the only absolute values of the connections in the reservoir.\n\n[1] Rodan, Ali, and Peter Tino. \"Minimum complexity echo state network.\" IEEE transactions on neural networks 22.1 (2010): 131-144.\n\n\n\n\n\n","category":"type"},{"location":"api/esn_layers/#ReservoirComputing.SimpleCycleReservoir","page":"ESN Layers","title":"ReservoirComputing.SimpleCycleReservoir","text":"SimpleCycleReservoir(res_size, weight)\nSimpleCycleReservoir(res_size; weight=0.1)\n\nReturns a Simple Cycle Reservoir constructor to build a reservoir matrix as described in [1]. The weight can be passed as arg or kwarg, and it determines the absolute value of all the connections in the reservoir.\n\n[1] Rodan, Ali, and Peter Tino. \"Minimum complexity echo state network.\" IEEE transactions on neural networks 22.1 (2010): 131-144.\n\n\n\n\n\n","category":"type"},{"location":"api/esn_layers/#ReservoirComputing.CycleJumpsReservoir","page":"ESN Layers","title":"ReservoirComputing.CycleJumpsReservoir","text":"CycleJumpsReservoir(res_size; cycle_weight=0.1, jump_weight=0.1, jump_size=3)\nCycleJumpsReservoir(res_size, cycle_weight, jump_weight, jump_size)\n\nReturn a Cycle Reservoir with Jumps constructor to create a reservoir matrix as described in [1]. The weight and jump_weight can be passed as args or kwargs, and they determine the absolute values of all the connections in the reservoir. The jump_size can also be passed either as arg or kwarg, and it detemines the jumps between jump_weights.\n\n[1] Rodan, Ali, and Peter Tiňo. \"Simple deterministically constructed cycle reservoirs with regular jumps.\" Neural computation 24.7 (2012): 1822-1852.\n\n\n\n\n\n","category":"type"},{"location":"api/esn_layers/#ReservoirComputing.NullReservoir","page":"ESN Layers","title":"ReservoirComputing.NullReservoir","text":"NullReservoir()\n\nReturn a constructor for a matrix zeros(res_size, res_size).\n\n\n\n\n\n","category":"type"},{"location":"api/esn_layers/","page":"ESN Layers","title":"ESN Layers","text":"Like for the input layers, to actually build the matrix of the reservoir, one can call the following function:","category":"page"},{"location":"api/esn_layers/","page":"ESN Layers","title":"ESN Layers","text":" create_reservoir","category":"page"},{"location":"api/esn_layers/#ReservoirComputing.create_reservoir","page":"ESN Layers","title":"ReservoirComputing.create_reservoir","text":"create_reservoir(reservoir::AbstractReservoir, res_size)\ncreate_reservoir(reservoir, args...)\n\nGiven an ``AbstractReservoir constructor and the reservoir size, it returns the corresponding matrix. Alternatively, it accepts a given matrix.\n\n\n\n\n\n","category":"function"},{"location":"api/esn_layers/","page":"ESN Layers","title":"ESN Layers","text":"To create a new reservoir, the procedure is similar to the one for the input layers. First, the definition of the new struct of type AbstractReservoir with the reservoir parameters is needed. Then the dispatch over the create_reservoir function makes the model actually build the reservoir matrix. An example of the workflow is given in the following snippet:","category":"page"},{"location":"api/esn_layers/","page":"ESN Layers","title":"ESN Layers","text":"#creation of the new struct for the reservoir\nstruct MyNewReservoir <: AbstractReservoir\n #the reservoir params go here\nend\n\n#dispatch over the function to build the reservoir matrix\nfunction create_reservoir(reservoir::AbstractReservoir, res_size)\n #the new algorithm to build the reservoir matrix goes here\nend","category":"page"},{"location":"#ReservoirComputing.jl","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"","category":"section"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"ReservoirComputing.jl provides an efficient, modular, and easy to use implementation of Reservoir Computing models such as Echo State Networks (ESNs). Reservoir Computing (RC) is an umbrella term used to describe a family of models such as ESNs and Liquid State Machines (LSMs). The key concept is to expand the input data into a higher dimension and use regression to train the model; in some ways, Reservoir Computers can be considered similar to kernel methods. ","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"info: Introductory material\nThis library assumes some basic knowledge of Reservoir Computing. For a good introduction, we suggest the following papers: the first two are the seminal papers about ESN and LSM, the others are in-depth review papers that should cover all the needed information. For the majority of the algorithms implemented in this library, we cited in the documentation the original work introducing them. If you ever have doubts about a method or a function, just type ? function in the Julia REPL to read the relevant notes.Jaeger, Herbert: The “echo state” approach to analyzing and training recurrent neural networks-with an erratum note.\nMaass W, Natschläger T, Markram H: Real-time computing without stable states: a new framework for neural computation based on perturbations.\nLukoševičius, Mantas: A practical guide to applying echo state networks.\" Neural networks: Tricks of the trade.\nLukoševičius, Mantas, and Herbert Jaeger: Reservoir computing approaches to recurrent neural network training.","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"info: Performance tip\nFor faster computations on the CPU, it is suggested to add using MKL to the script. For clarity's sake, this library will not be indicated under every example in the documentation.","category":"page"},{"location":"#Installation","page":"ReservoirComputing.jl","title":"Installation","text":"","category":"section"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"To install ReservoirComputing.jl, use the Julia package manager:","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"using Pkg\nPkg.add(\"ReservoirComputing\")","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"The support for this library is for Julia v1.6 or greater.","category":"page"},{"location":"#Features-Overview","page":"ReservoirComputing.jl","title":"Features Overview","text":"","category":"section"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"This library provides multiple ways of training the chosen RC model. More specifically, the available algorithms are:","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"StandardRidge: a naive implementation of Ridge Regression. The default choice for training.\nLinearModel: a wrap around MLJLinearModels.\nLIBSVM.AbstractSVR: a direct call of LIBSVM regression methods.","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"Also provided are two different ways of making predictions using RC:","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"Generative: the algorithm uses the prediction of the model in the previous step to continue the prediction. It only needs the number of steps as input.\nPredictive: standard Machine Learning type of prediction. Given the features, the RC model will return the label/prediction.","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"It is possible to modify the RC obtained states in the training and prediction steps using the following:","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"StandardStates: default choice, no changes will be made to the states.\nExtendedStates: the states are extended using a vertical concatenation, with the input data.\nPaddedStates: the states are padded using a vertical concatenation with the chosen padding value.\nPaddedExtendedStates: a combination of the first two. First, the states are extended and then padded.","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"In addition, another modification is possible through the choice of non-linear algorithms:","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"NLADefault: default choice, no changes will be made to the states.\nNLAT1\nNLAT2\nNLAT3","category":"page"},{"location":"#Echo-State-Networks","page":"ReservoirComputing.jl","title":"Echo State Networks","text":"","category":"section"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"For ESNs the following input layers are implemented :","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"WeightedLayer: weighted layer matrix with weights sampled from a uniform distribution.\nDenseLayer: dense layer matrix with weights sampled from a uniform distribution.\nSparseLayer: sparse layer matrix with weights sampled from a uniform distribution.\nMinimumLayer: matrix with constant weights and weight sign decided following one of the two:\nBernoulliSample\nIrrationalSample\nInformedLayer: special kin of weighted layer matrix for Hybrid ESNs.","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"The package also contains multiple implementations of Reservoirs:","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"RandSparseReservoir: random sparse matrix with scaling of spectral radius\nPseudoSVDReservoir: Pseudo SVD construction of a random sparse matrix\nDelayLineReservoir: minimal matrix with chosen weights\nDelayLineBackwardReservoir: minimal matrix with chosen weights\nSimpleCycleReservoir: minimal matrix with chosen weights\nCycleJumpsReservoir: minimal matrix with chosen weights","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"In addition, multiple ways of driving the reservoir states are also provided:","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"RNN: standard Recurrent Neural Network driver.\nMRNN: Multiple RNN driver, it consists of a linear combination of RNNs\nGRU: gated Recurrent Unit driver, with all the possible GRU variants available:\nFullyGated\nVariant1\nVariant2\nVariant3\nMinimal","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"A hybrid version of the model is also available through Hybrid","category":"page"},{"location":"#Reservoir-Computing-with-Cellular-Automata","page":"ReservoirComputing.jl","title":"Reservoir Computing with Cellular Automata","text":"","category":"section"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"The package provides also an implementation of Reservoir Computing models based on one dimensional Cellular Automata through the RECA call. For the moment, the only input encoding available (an input encoding plays a similar role to the input matrix for ESNs) is a random mapping, called through RandomMapping. ","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"All the training methods described above can be used, as can all the modifications to the states. Both prediction methods are also possible in theory, although in the literature only Predictive tasks have been explored.","category":"page"},{"location":"#Contributing","page":"ReservoirComputing.jl","title":"Contributing","text":"","category":"section"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"Please refer to the SciML ColPrac: Contributor's Guide on Collaborative Practices for Community Packages for guidance on PRs, issues, and other matters relating to contributing to SciML.\nSee the SciML Style Guide for common coding practices and other style decisions.\nThere are a few community forums:\nThe #diffeq-bridged and #sciml-bridged channels in the Julia Slack\nThe #diffeq-bridged and #sciml-bridged channels in the Julia Zulip\nOn the Julia Discourse forums\nSee also SciML Community page","category":"page"},{"location":"#Citing","page":"ReservoirComputing.jl","title":"Citing","text":"","category":"section"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"If you use this library in your work, please cite:","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"@article{JMLR:v23:22-0611,\n author = {Francesco Martinuzzi and Chris Rackauckas and Anas Abdelrehim and Miguel D. Mahecha and Karin Mora},\n title = {ReservoirComputing.jl: An Efficient and Modular Library for Reservoir Computing Models},\n journal = {Journal of Machine Learning Research},\n year = {2022},\n volume = {23},\n number = {288},\n pages = {1--8},\n url = {http://jmlr.org/papers/v23/22-0611.html}\n}","category":"page"},{"location":"#Reproducibility","page":"ReservoirComputing.jl","title":"Reproducibility","text":"","category":"section"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"The documentation of this SciML package was built using these direct dependencies,","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"using Pkg # hide\nPkg.status() # hide","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"and using this machine and Julia version.","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"using InteractiveUtils # hide\nversioninfo() # hide","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"A more complete overview of all dependencies and their versions is also provided.","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"using Pkg # hide\nPkg.status(;mode = PKGMODE_MANIFEST) # hide","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"","category":"page"},{"location":"","page":"ReservoirComputing.jl","title":"ReservoirComputing.jl","text":"using TOML\nusing Markdown\nversion = TOML.parse(read(\"../../Project.toml\", String))[\"version\"]\nname = TOML.parse(read(\"../../Project.toml\", String))[\"name\"]\nlink_manifest = \"https://github.com/SciML/\" * name * \".jl/tree/gh-pages/v\" * version *\n \"/assets/Manifest.toml\"\nlink_project = \"https://github.com/SciML/\" * name * \".jl/tree/gh-pages/v\" * version *\n \"/assets/Project.toml\"\nMarkdown.parse(\"\"\"You can also download the\n[manifest]($link_manifest)\nfile and the\n[project]($link_project)\nfile.\n\"\"\")","category":"page"},{"location":"api/esn/#Echo-State-Networks","page":"Echo State Networks","title":"Echo State Networks","text":"","category":"section"},{"location":"api/esn/","page":"Echo State Networks","title":"Echo State Networks","text":" ESN","category":"page"},{"location":"api/esn/#ReservoirComputing.ESN","page":"Echo State Networks","title":"ReservoirComputing.ESN","text":"ESN(train_data;\n variation = Default(),\n input_layer = DenseLayer(),\n reservoir = RandSparseReservoir(),\n bias = NullLayer(),\n reservoir_driver = RNN(),\n nla_type = NLADefault(),\n states_type = StandardStates())\n(esn::ESN)(prediction::AbstractPrediction,\n output_layer::AbstractOutputLayer;\n initial_conditions=output_layer.last_value,\n last_state=esn.states[:, end])\n\nConstructor for the Echo State Network model. It requires the reservoir size as the input and the data for the training. It returns a struct ready to be trained with the states already harvested.\n\nAfter the training, this struct can be used for the prediction following the second function call. This will take as input a prediction type and the output layer from the training. The initial_conditions and last_state parameters can be left as they are, unless there is a specific reason to change them. All the components are detailed in the API documentation. More examples are given in the general documentation.\n\n\n\n\n\n","category":"type"},{"location":"api/esn/","page":"Echo State Networks","title":"Echo State Networks","text":"In addition to all the components that can be explored in the documentation, a couple components need a separate introduction. The variation arguments can be","category":"page"},{"location":"api/esn/","page":"Echo State Networks","title":"Echo State Networks","text":" Default\n Hybrid","category":"page"},{"location":"api/esn/#ReservoirComputing.Default","page":"Echo State Networks","title":"ReservoirComputing.Default","text":"Default()\n\nSets the type of the ESN as the standard model. No parameters are needed.\n\n\n\n\n\n","category":"type"},{"location":"api/esn/#ReservoirComputing.Hybrid","page":"Echo State Networks","title":"ReservoirComputing.Hybrid","text":"Hybrid(prior_model, u0, tspan, datasize)\n\nGiven the model parameters, returns an Hybrid variation of the ESN. This entails a different training and prediction. Construction based on [1].\n\n[1] Jaideep Pathak et al. \"Hybrid Forecasting of Chaotic Processes: Using Machine Learning in Conjunction with a Knowledge-Based Model\" (2018)\n\n\n\n\n\n","category":"type"},{"location":"api/esn/","page":"Echo State Networks","title":"Echo State Networks","text":"These arguments detail a deeper variation of the underlying model, and they need a separate call. For the moment, the most complex is the Hybrid call, but this can and will change in the future. All ESN models can be trained using the following call:","category":"page"},{"location":"api/esn/","page":"Echo State Networks","title":"Echo State Networks","text":" train","category":"page"},{"location":"api/esn/#ReservoirComputing.train","page":"Echo State Networks","title":"ReservoirComputing.train","text":"train(esn::AbstractEchoStateNetwork, target_data, training_method=StandardRidge(0.0))\n\nTraining of the built ESN over the target_data. The default training method is RidgeRegression. The output is an OutputLayer object to be fed to the esn call for the prediction.\n\n\n\n\n\n","category":"function"}]
}
