diff --git a/docs/pages.jl b/docs/pages.jl index 0c1465c7..32e6c373 100644 --- a/docs/pages.jl +++ b/docs/pages.jl @@ -7,7 +7,7 @@ pages = [ "Echo State Network Tutorials" => Any[ "Lorenz System Forecasting" => "esn_tutorials/lorenz_basic.md", #"Mackey-Glass Forecasting on GPU" => "esn_tutorials/mackeyglass_basic.md", - "Using Different Layers" => "esn_tutorials/change_layers.md", + #"Using Different Layers" => "esn_tutorials/change_layers.md", "Using Different Reservoir Drivers" => "esn_tutorials/different_drivers.md", #"Using Different Training Methods" => "esn_tutorials/different_training.md", "Deep Echo State Networks" => "esn_tutorials/deep_esn.md", @@ -17,7 +17,7 @@ pages = [ "States Modifications" => "api/states.md", "Prediction Types" => "api/predict.md", "Echo State Networks" => "api/esn.md", - "ESN Layers" => "api/esn_layers.md", + #"ESN Layers" => "api/esn_layers.md", "ESN Drivers" => "api/esn_drivers.md", "ReCA" => "api/reca.md"] ] diff --git a/docs/src/api/esn.md b/docs/src/api/esn.md index 1caacdd1..20b8b837 100644 --- a/docs/src/api/esn.md +++ b/docs/src/api/esn.md @@ -6,17 +6,6 @@ The core component of an ESN is the `ESN` type. It represents the entire Echo St ESN ``` -## Variations - -In addition to the standard `ESN` model, there are variations that allow for deeper customization of the underlying model. Currently, there are two available variations: `Default` and `Hybrid`. These variations provide different ways to configure the ESN. Here's the documentation for the variations: - -```@docs - Default - Hybrid -``` - -The `Hybrid` variation is the most complex option and offers additional customization. Note that more variations may be added in the future to provide even greater flexibility. - ## Training To train an ESN model, you can use the `train` function. It takes the ESN model, training data, and other optional parameters as input and returns a trained model. Here's the documentation for the train function: diff --git a/docs/src/api/esn_layers.md b/docs/src/api/esn_layers.md deleted file mode 100644 index bb3b53de..00000000 --- a/docs/src/api/esn_layers.md +++ /dev/null @@ -1,71 +0,0 @@ -# ESN Layers - -## Input Layers - -```@docs - WeightedLayer - DenseLayer - SparseLayer - InformedLayer - MinimumLayer - NullLayer -``` - -The signs in the `MinimumLayer` are chosen based on the following methods: - -```@docs - BernoulliSample - IrrationalSample -``` - -To derive the matrix one can call the following function: - -```@docs - create_layer -``` - -To create new input layers, it suffices to define a new struct containing the needed parameters of the new input layer. This struct will need to be an `AbstractLayer`, so the `create_layer` function can be dispatched over it. The workflow should follow this snippet: - -```julia -#creation of the new struct for the layer -struct MyNewLayer <: AbstractLayer - #the layer params go here -end - -#dispatch over the function to actually build the layer matrix -function create_layer(input_layer::MyNewLayer, res_size, in_size) - #the new algorithm to build the input layer goes here -end -``` - -## Reservoirs - -```@docs - RandSparseReservoir - PseudoSVDReservoir - DelayLineReservoir - DelayLineBackwardReservoir - SimpleCycleReservoir - CycleJumpsReservoir - NullReservoir -``` - -Like for the input layers, to actually build the matrix of the reservoir, one can call the following function: - -```@docs - create_reservoir -``` - -To create a new reservoir, the procedure is similar to the one for the input layers. First, the definition of the new struct of type `AbstractReservoir` with the reservoir parameters is needed. Then the dispatch over the `create_reservoir` function makes the model actually build the reservoir matrix. An example of the workflow is given in the following snippet: - -```julia -#creation of the new struct for the reservoir -struct MyNewReservoir <: AbstractReservoir - #the reservoir params go here -end - -#dispatch over the function to build the reservoir matrix -function create_reservoir(reservoir::AbstractReservoir, res_size) - #the new algorithm to build the reservoir matrix goes here -end -``` diff --git a/docs/src/api/training.md b/docs/src/api/training.md index a11b2034..f0106121 100644 --- a/docs/src/api/training.md +++ b/docs/src/api/training.md @@ -2,14 +2,9 @@ ## Linear Models -```@docs - StandardRidge - LinearModel -``` - ## Gaussian Regression -Currently, v0.9 is unavailable. +Currently, v0.10, is unavailable. ## Support Vector Regression diff --git a/docs/src/esn_tutorials/change_layers.md b/docs/src/esn_tutorials/change_layers.md deleted file mode 100644 index 86c70477..00000000 --- a/docs/src/esn_tutorials/change_layers.md +++ /dev/null @@ -1,99 +0,0 @@ -# Using Different Layers - -A great deal of effort in the ESNs field is devoted to finding the ideal construction for the reservoir matrices. With a simple interface using ReservoirComputing.jl it is possible to leverage the currently implemented matrix construction methods for both the reservoir and the input layer. On this page, it is showcased how it is possible to change both of these layers. - -The `input_init` keyword argument provided with the `ESN` constructor allows for changing the input layer. The layers provided in ReservoirComputing.jl are the following: - - - `WeightedLayer(scaling)` - - `DenseLayer(scaling)` - - `SparseLayer(scaling, sparsity)` - - `MinimumLayer(weight, sampling)` - - `InformedLayer(model_in_size; scaling=0.1, gamma=0.5)` - In addition, the user can define a custom layer following this workflow: - -```julia -#creation of the new struct for the layer -struct MyNewLayer <: AbstractLayer - #the layer params go here -end - -#dispatch over the function to actually build the layer matrix -function create_layer(input_layer::MyNewLayer, res_size, in_size) - #the new algorithm to build the input layer goes here -end -``` - -Similarly the `reservoir_init` keyword argument provides the possibility to change the construction for the reservoir matrix. The available reservoir are: - - - `RandSparseReservoir(res_size, radius, sparsity)` - - `PseudoSVDReservoir(res_size, max_value, sparsity, sorted, reverse_sort)` - - `DelayLineReservoir(res_size, weight)` - - `DelayLineBackwardReservoir(res_size, weight, fb_weight)` - - `SimpleCycleReservoir(res_size, weight)` - - `CycleJumpsReservoir(res_size, cycle_weight, jump_weight, jump_size)` - And, like before, it is possible to build a custom reservoir by following this workflow: - -```julia -#creation of the new struct for the reservoir -struct MyNewReservoir <: AbstractReservoir - #the reservoir params go here -end - -#dispatch over the function to build the reservoir matrix -function create_reservoir(reservoir::AbstractReservoir, res_size) - #the new algorithm to build the reservoir matrix goes here -end -``` - -## Example of a minimally complex ESN - -Using [^1] and [^2] as references, this section will provide an example of how to change both the input layer and the reservoir for ESNs. The full script for this example can be found [here](https://github.com/MartinuzziFrancesco/reservoir-computing-examples/blob/main/change_layers/layers.jl). This example was run on Julia v1.7.2. - -The task for this example will be the one step ahead prediction of the Henon map. To obtain the data, one can leverage the package [DynamicalSystems.jl](https://juliadynamics.github.io/DynamicalSystems.jl/dev/). The data is scaled to be between -1 and 1. - -```@example mesn -using PredefinedDynamicalSystems -train_len = 3000 -predict_len = 2000 - -ds = PredefinedDynamicalSystems.henon() -traj, time = trajectory(ds, 7000) -data = Matrix(traj)' -data = (data .- 0.5) .* 2 -shift = 200 - -training_input = data[:, shift:(shift + train_len - 1)] -training_target = data[:, (shift + 1):(shift + train_len)] -testing_input = data[:, (shift + train_len):(shift + train_len + predict_len - 1)] -testing_target = data[:, (shift + train_len + 1):(shift + train_len + predict_len)] -``` - -Now it is possible to define the input layers and reservoirs we want to compare and run the comparison in a simple for loop. The accuracy will be tested using the mean squared deviation `msd` from [StatsBase](https://juliastats.org/StatsBase.jl/stable/). - -```@example mesn -using ReservoirComputing, StatsBase - -res_size = 300 -input_layer = [ - MinimumLayer(0.85, IrrationalSample()), - MinimumLayer(0.95, IrrationalSample()) -] -reservoirs = [SimpleCycleReservoir(res_size, 0.7), - CycleJumpsReservoir(res_size, cycle_weight = 0.7, jump_weight = 0.2, jump_size = 5)] - -for i in 1:length(reservoirs) - esn = ESN(training_input; - input_layer = input_layer[i], - reservoir = reservoirs[i]) - wout = train(esn, training_target, StandardRidge(0.001)) - output = esn(Predictive(testing_input), wout) - println(msd(testing_target, output)) -end -``` - -As it is possible to see, changing layers in ESN models is straightforward. Be sure to check the API documentation for a full list of reservoirs and layers. - -## Bibliography - -[^1]: Rodan, Ali, and Peter Tiňo. “Simple deterministically constructed cycle reservoirs with regular jumps.” Neural computation 24.7 (2012): 1822-1852. -[^2]: Rodan, Ali, and Peter Tiňo. “Minimum complexity echo state network.” IEEE transactions on neural networks 22.1 (2010): 131-144. diff --git a/docs/src/esn_tutorials/deep_esn.md b/docs/src/esn_tutorials/deep_esn.md index bd4f7b46..08f4e9f1 100644 --- a/docs/src/esn_tutorials/deep_esn.md +++ b/docs/src/esn_tutorials/deep_esn.md @@ -21,6 +21,7 @@ end #solve and take data prob = ODEProblem(lorenz!, [1.0, 0.0, 0.0], (0.0, 200.0)) data = solve(prob, ABM54(), dt = 0.02) +data = reduce(hcat, data.u) #determine shift length, training length and prediction length shift = 300 @@ -40,20 +41,18 @@ The construction of the ESN is also really similar. The only difference is that ```@example deep_lorenz using ReservoirComputing -reservoirs = [RandSparseReservoir(99, radius = 1.1, sparsity = 0.1), - RandSparseReservoir(100, radius = 1.2, sparsity = 0.1), - RandSparseReservoir(200, radius = 1.4, sparsity = 0.1)] +reservoirs = [rand_sparse(; radius = 1.1, sparsity = 0.1), + rand_sparse(; radius = 1.2, sparsity = 0.1), + rand_sparse(; radius = 1.4, sparsity = 0.1)] -esn = ESN(input_data; - variation = Default(), +esn = DeepESN(input_data, 3, 200; reservoir = reservoirs, - input_layer = DenseLayer(), reservoir_driver = RNN(), nla_type = NLADefault(), states_type = StandardStates()) ``` -As it is possible to see, different sizes can be chosen for the different reservoirs. The input layer and bias can also be given as vectors, but of course, they have to be of the same size of the reservoirs vector. If they are not passed as a vector, the value passed will be used for all the layers in the deep ESN. +The input layer and bias can also be given as vectors, but of course, they have to be of the same size of the reservoirs vector. If they are not passed as a vector, the value passed will be used for all the layers in the deep ESN. In addition to using the provided functions for the construction of the layers, the user can also choose to build their own matrix, or array of matrices, and feed that into the `ESN` in the same way. @@ -89,8 +88,6 @@ plot(p1, p2, p3, plot_title = "Lorenz System Coordinates", legendfontsize = 12, titlefontsize = 20) ``` -Note that there is a known bug at the moment with using `WeightedLayer` as the input layer with the deep ESN. We are in the process of investigating and solving it. The leak coefficient for the reservoirs has to always be the same in the current implementation. This is also something we are actively looking into expanding. - ## Documentation [^1]: Gallicchio, Claudio, and Alessio Micheli. "_Deep echo state network (deepesn): A brief survey._" arXiv preprint arXiv:1712.04323 (2017). diff --git a/docs/src/esn_tutorials/different_drivers.md b/docs/src/esn_tutorials/different_drivers.md index 0a669451..5a6fc983 100644 --- a/docs/src/esn_tutorials/different_drivers.md +++ b/docs/src/esn_tutorials/different_drivers.md @@ -72,9 +72,9 @@ case5 = MRNN([tanh, f4], 0.9, [0.43, 0.13]) #tests test_cases = [base_case, case3, case4, case5] for case in test_cases - esn = ESN(training_input, - input_layer = WeightedLayer(scaling = 0.3), - reservoir = RandSparseReservoir(100, radius = 0.4), + esn = ESN(training_input, 1, 100, + input_layer = weighted_init(; scaling = 0.3), + reservoir = rand_sparse(; radius = 0.4), reservoir_driver = case, states_type = ExtendedStates()) wout = train(esn, training_target, StandardRidge(10e-6)) @@ -186,19 +186,19 @@ res_size = 300 res_radius = 1.4 Random.seed!(42) -esn = ESN(training_input; - reservoir = RandSparseReservoir(res_size, radius = res_radius), +esn = ESN(training_input, 1, res_size; + reservoir = rand_sparse(; radius = res_radius), reservoir_driver = GRU()) ``` The default inner reservoir and input layer for the GRU are the same defaults for the `reservoir` and `input_layer` of the ESN. One can use the explicit call if they choose to. ```@example gru -gru = GRU(reservoir = [RandSparseReservoir(res_size), - RandSparseReservoir(res_size)], - inner_layer = [DenseLayer(), DenseLayer()]) -esn = ESN(training_input; - reservoir = RandSparseReservoir(res_size, radius = res_radius), +gru = GRU(reservoir = [rand_sparse, + rand_sparse], + inner_layer = [scaled_rand, scaled_rand]) +esn = ESN(training_input, 1, res_size; + reservoir = rand_sparse(; radius = res_radius), reservoir_driver = gru) ``` @@ -230,8 +230,8 @@ It is interesting to see a comparison of the GRU driven ESN and the standard RNN ```@example gru using StatsBase -esn_rnn = ESN(training_input; - reservoir = RandSparseReservoir(res_size, radius = res_radius), +esn_rnn = ESN(training_input, 1, res_size; + reservoir = rand_sparse(; radius = res_radius), reservoir_driver = RNN()) output_layer = train(esn_rnn, training_target, training_method) diff --git a/docs/src/esn_tutorials/hybrid.md b/docs/src/esn_tutorials/hybrid.md index 25797089..88e79707 100644 --- a/docs/src/esn_tutorials/hybrid.md +++ b/docs/src/esn_tutorials/hybrid.md @@ -70,7 +70,7 @@ The training and prediction of the Hybrid ESN can proceed as usual: ```@example hybrid output_layer = train(hesn, target_data, StandardRidge(0.3)) -output = esn(Generative(predict_len), output_layer) +output = hesn(Generative(predict_len), output_layer) ``` It is now possible to plot the results, leveraging Plots.jl: diff --git a/docs/src/esn_tutorials/lorenz_basic.md b/docs/src/esn_tutorials/lorenz_basic.md index 1a66f834..9b46a338 100644 --- a/docs/src/esn_tutorials/lorenz_basic.md +++ b/docs/src/esn_tutorials/lorenz_basic.md @@ -19,6 +19,7 @@ end #solve and take data prob = ODEProblem(lorenz!, [1.0, 0.0, 0.0], (0.0, 200.0)) data = solve(prob, ABM54(), dt = 0.02) +data = reduce(hcat, data.u) ``` After obtaining the data, it is necessary to determine the kind of prediction for the model. Since this example will use the `Generative` prediction type, this means that the target data will be the next step of the input data. In addition, it is important to notice that the Lorenz system just obtained presents a transient period that is not representative of the general behavior of the system. This can easily be discarded by setting a `shift` parameter. diff --git a/docs/src/reca_tutorials/reca.md b/docs/src/reca_tutorials/reca.md index 65c88044..e1e52c96 100644 --- a/docs/src/reca_tutorials/reca.md +++ b/docs/src/reca_tutorials/reca.md @@ -11,8 +11,8 @@ The data can be read as follows: ```@example reca using DelimitedFiles -input = readdlm("./5bitinput.txt", ',', Float32) -output = readdlm("./5bitoutput.txt", ',', Float32) +input = readdlm("./5bitinput.txt", ',', Float64) +output = readdlm("./5bitoutput.txt", ',', Float64) ``` To use a ReCA model, it is necessary to define the rule one intends to use. To do so, ReservoirComputing.jl leverages [CellularAutomata.jl](https://github.com/MartinuzziFrancesco/CellularAutomata.jl) that needs to be called as well to define the `RECA` struct: diff --git a/src/ReservoirComputing.jl b/src/ReservoirComputing.jl index 18a6da79..23a9fbdc 100644 --- a/src/ReservoirComputing.jl +++ b/src/ReservoirComputing.jl @@ -18,10 +18,11 @@ export StandardRidge export scaled_rand, weighted_init, informed_init, minimal_init export rand_sparse, delay_line, delay_line_backward, cycle_jumps, simple_cycle, pseudo_svd export RNN, MRNN, GRU, GRUParams, FullyGated, Minimal -export ESN, train +export train +export ESN export HybridESN, KnowledgeModel export DeepESN -export RECA, train +export RECA export RandomMapping, RandomMaps export Generative, Predictive, OutputLayer diff --git a/src/esn/deepesn.jl b/src/esn/deepesn.jl index 569a1ac8..478a31b3 100644 --- a/src/esn/deepesn.jl +++ b/src/esn/deepesn.jl @@ -11,6 +11,73 @@ struct DeepESN{I, S, N, T, O, M, B, ST, W, IS} <: AbstractEchoStateNetwork states::IS end +""" + DeepESN(train_data, in_size, res_size; kwargs...) + +Constructs a Deep Echo State Network (ESN) model for +processing sequential data through a layered architecture of reservoirs. +This constructor allows for the creation of a deep learning model that +benefits from the dynamic memory and temporal processing capabilities of ESNs, +enhanced by the depth provided by multiple reservoir layers. It's particularly +suited for complex sequential tasks where depth can help capture hierarchical +temporal features. + +# Parameters + + - `train_data`: The training dataset used for the ESN. + This should be structured as sequential data where applicable. + - `in_size`: The size of the input layer, i.e., the number of + input units to the ESN. + - `res_size`: The size of each reservoir, i.e., the number of neurons + in each hidden layer of the ESN. + +# Optional Keyword Arguments + + - `depth`: The number of reservoir layers in the Deep ESN. Default is 2. + - `input_layer`: A function or an array of functions to initialize the input + matrices for each layer. Default is `scaled_rand` for each layer. + - `bias`: A function or an array of functions to initialize the bias vectors + for each layer. Default is `zeros64` for each layer. + - `reservoir`: A function or an array of functions to initialize the reservoir + matrices for each layer. Default is `rand_sparse` for each layer. + - `reservoir_driver`: The driving system for the reservoir. + Default is an RNN model. + - `nla_type`: The type of non-linear activation used in the reservoir. + Default is `NLADefault()`. + - `states_type`: Defines the type of states used in the ESN (e.g., standard states). + Default is `StandardStates()`. + - `washout`: The number of initial timesteps to be discarded in the ESN's training phase. + Default is 0. + - `rng`: Random number generator used for initializing weights. Default is the package's + default random number generator. + - `T`: The data type for the matrices (e.g., `Float64`). Influences computational + efficiency and precision. + - `matrix_type`: The type of matrix used for storing the training data. + Default is inferred from `train_data`. + +# Returns + + - A `DeepESN` instance configured according to the provided parameters + and suitable for further training and prediction tasks. + +# Example + +```julia +# Prepare your training data +train_data = [your_training_data_here] + +# Create a DeepESN with specific parameters +deepESN = DeepESN(train_data, 10, 100, depth = 3, washout = 100) + +# Proceed with training and prediction (pseudocode) +train(deepESN, target_data) +prediction = predict(deepESN, new_data) +``` + +The DeepESN model is ideal for tasks requiring the processing of sequences with +complex temporal dependencies, benefiting from the multiple reservoirs to capture +different levels of abstraction and temporal dynamics. +""" function DeepESN(train_data, in_size::Int, res_size::Int; diff --git a/src/esn/hybridesn.jl b/src/esn/hybridesn.jl index 567bf3f5..ad134a9e 100644 --- a/src/esn/hybridesn.jl +++ b/src/esn/hybridesn.jl @@ -22,7 +22,7 @@ struct KnowledgeModel{T, K, O, I, S, D} end """ - Hybrid(prior_model, u0, tspan, datasize) +KnowledgeModel(prior_model, u0, tspan, datasize) Constructs a `Hybrid` variation of Echo State Networks (ESNs) integrating a knowledge-based model (`prior_model`) with ESNs for advanced training and prediction in chaotic systems. @@ -55,6 +55,66 @@ function KnowledgeModel(prior_model, u0, tspan, datasize) return KnowledgeModel(prior_model, u0, tspan, dt, datasize, model_data) end +""" + HybridESN(model, train_data, in_size, res_size; kwargs...) + +Construct a Hybrid Echo State Network (ESN) model that integrates +traditional Echo State Networks with a predefined knowledge model for +enhanced performance on chaotic systems or complex datasets. This +constructor allows for the creation of a customized ESN architecture by +specifying the reservoir size, input size, and various other parameters that +influence the network's behavior and learning capacity. + +# Parameters + + - `model`: A `KnowledgeModel` instance representing the knowledge-based model + to be integrated with the ESN. + - `train_data`: The training dataset used for the ESN. This data can be + preprocessed or raw data depending on the nature of the problem and the + preprocessing steps considered. + - `in_size`: The size of the input layer, i.e., the number of input units + to the ESN. + - `res_size`: The size of the reservoir, i.e., the number of neurons in + the hidden layer of the ESN. + +# Optional Keyword Arguments + + - `input_layer`: A function to initialize the input matrix. Default is `scaled_rand`. + - `reservoir`: A function to initialize the reservoir matrix. Default is `rand_sparse`. + - `bias`: A function to initialize the bias vector. Default is `zeros64`. + - `reservoir_driver`: The driving system for the reservoir. Default is an RNN model. + - `nla_type`: The type of non-linear activation used in the reservoir. + Default is `NLADefault()`. + - `states_type`: Defines the type of states used in the ESN (e.g., standard states). + Default is `StandardStates()`. + - `washout`: The number of initial timesteps to be discarded in the ESN's training phase. + Default is 0. + - `rng`: Random number generator used for initializing weights. Default is the package's + default random number generator. + - `T`: The data type for the matrices (e.g., `Float32`). Influences computational + efficiency and precision. + - `matrix_type`: The type of matrix used for storing the training data. Default is + inferred from `train_data`. + +# Returns + + - A `HybridESN` instance configured according to the provided parameters and + suitable for further training and prediction tasks. + +# Example + +```julia +# Define a KnowledgeModel +km = KnowledgeModel(prior_model_function, u0, (0, 100), 1000) + +# Create a HybridESN +hesn = HybridESN(km, train_data, 10, 100; washout = 100) + +# Train and predict +train(hesn, target_data) +prediction = hesn(prediction_object, output_layer) +``` +""" function HybridESN(model, train_data, in_size::Int, diff --git a/src/reca/reca.jl b/src/reca/reca.jl index 39685e90..8af5b2d3 100644 --- a/src/reca/reca.jl +++ b/src/reca/reca.jl @@ -41,7 +41,7 @@ end #training dispatch function train(reca::AbstractReca, target_data, training_method = StandardRidge; kwargs...) states_new = reca.states_type(reca.nla_type, reca.states, reca.train_data) - return train(training_method, states_new, target_data; kwargs...) + return train(training_method, Float32.(states_new), Float32.(target_data); kwargs...) end #predict dispatch diff --git a/src/train/linear_regression.jl b/src/train/linear_regression.jl index a291d34b..1a271cc3 100644 --- a/src/train/linear_regression.jl +++ b/src/train/linear_regression.jl @@ -11,8 +11,8 @@ function StandardRidge() end function train(sr::StandardRidge, - states::AbstractArray{T}, - target_data::AbstractArray{T}) where {T <: Number} + states, + target_data) #A = states * states' + sr.reg * I #b = states * target_data #output_layer = (A \ b)'