Reacher-ppo.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Start the Environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from mlagents.envs import UnityEnvironment\n",
    "import numpy as np\n",
    "\n",
    "# Some more magic so that the notebook will reload external python modules;\n",
    "# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**_Before running the code cell below_**, change the `file_name` parameter to match the location of the Reacher Unity environment.\n",
    "\n",
    "For instance, if you are using a Mac, then you downloaded `Reacher.app`.  If this file is in the same folder as the notebook, then the line below should appear as follows:\n",
    "```\n",
    "env = UnityEnvironment(file_name=\"Reacher.app\")\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:mlagents.envs:\n",
      "'Academy' started successfully!\n",
      "Unity Academy name: Academy\n",
      "        Number of Brains: 1\n",
      "        Number of Training Brains : 1\n",
      "        Reset Parameters :\n",
      "\t\tgoal_speed -> 1.0\n",
      "\t\tgoal_size -> 5.0\n",
      "Unity brain name: ReacherLearning\n",
      "        Number of Visual Observations (per agent): 0\n",
      "        Vector Observation space size (per agent): 33\n",
      "        Number of stacked Vector Observation: 1\n",
      "        Vector Action space type: continuous\n",
      "        Vector Action space size (per agent): [4]\n",
      "        Vector Action descriptions: , , , \n"
     ]
    }
   ],
   "source": [
    "# load Unity environment for Linux\n",
    "env = UnityEnvironment(file_name='unity_envs/Reacher_Linux/Reacher_Linux.x86_64')\n",
    "\n",
    "# # load Unity env for macos\n",
    "# env = UnityEnvironment(file_name='unity_envs/Reacher_macos/Reacher.app')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Environments contain **_brains_** which are responsible for deciding the actions of their associated agents. Here we check for the first brain available, and set it as the default brain we will be controlling from Python."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# get the default brain\n",
    "brain_name = env.brain_names[0]\n",
    "brain = env.brains[brain_name]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Examine the State and Action Spaces\n",
    "\n",
    "* Set-up: Double-jointed arm which can move to target locations.\n",
    "* Goal: The agents must move its hand to the goal location, and keep it there.\n",
    "* Agents: The environment contains 10 agent linked to a single Brain.\n",
    "* Agent Reward Function (independent):\n",
    "  * +0.1 Each step agent's hand is in goal location.\n",
    "* Brains: One Brain with the following observation/action space.\n",
    "  * Vector Observation space: 26 variables corresponding to position, rotation,\n",
    "    velocity, and angular velocities of the two arm Rigidbodies.\n",
    "  * Vector Action space: (Continuous) Size of 4, corresponding to torque\n",
    "    applicable to two joints.\n",
    "  * Visual Observations: None.\n",
    "* Reset Parameters: Two, corresponding to goal size, and goal movement speed.\n",
    "* Benchmark Mean Reward: 30\n",
    "\n",
    "Lets print some information about the environment."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of agents: 20\n",
      "Size of each action: 4\n",
      "There are 20 agents. Each observes a state with length: 33\n",
      "The state for the first agent looks like: [ 0.00000000e+00 -4.00000000e+00  0.00000000e+00  1.00000000e+00\n",
      " -0.00000000e+00 -0.00000000e+00 -4.37113883e-08  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00 -1.00000000e+01  0.00000000e+00\n",
      "  1.00000000e+00 -0.00000000e+00 -0.00000000e+00 -4.37113883e-08\n",
      "  0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00\n",
      "  0.00000000e+00  0.00000000e+00  5.75471878e+00 -1.00000000e+00\n",
      "  5.55726671e+00  0.00000000e+00  1.00000000e+00  0.00000000e+00\n",
      " -1.68164849e-01]\n"
     ]
    }
   ],
   "source": [
    "# reset the environment\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "\n",
    "# number of agents\n",
    "num_agents = len(env_info.agents)\n",
    "print('Number of agents:', num_agents)\n",
    "\n",
    "# size of each action\n",
    "action_size = brain.vector_action_space_size[0]\n",
    "print('Size of each action:', action_size)\n",
    "\n",
    "# examine the state space \n",
    "states = env_info.vector_observations\n",
    "state_size = states.shape[1]\n",
    "print('There are {} agents. Each observes a state with length: {}'.format(states.shape[0], state_size))\n",
    "print('The state for the first agent looks like:', states[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Take Random Actions in the Environment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total score (averaged over agents) this episode: 0.17549999607726932\n"
     ]
    }
   ],
   "source": [
    "env_info = env.reset(train_mode=True)[brain_name]     # reset the environment    \n",
    "states = env_info.vector_observations                  # get the current state (for each agent)\n",
    "scores = np.zeros(num_agents) # initialize the score (for each agent)\n",
    "step=0\n",
    "while True:\n",
    "    actions = np.random.randn(num_agents, action_size) # select an action (for each agent)\n",
    "    actions = np.clip(actions, -1, 1)                  # all actions between -1 and 1\n",
    "    env_info = env.step(actions)[brain_name]           # send all actions to tne environment\n",
    "    next_states = env_info.vector_observations         # get next state (for each agent)\n",
    "    rewards = env_info.rewards                         # get reward (for each agent)\n",
    "    dones = env_info.local_done                        # see if episode finished\n",
    "    scores += env_info.rewards                         # update the score (for each agent)\n",
    "    states = next_states                               # roll over states to next time step\n",
    "    step+=1\n",
    "    if np.any(dones):                                  # exit loop if episode finished\n",
    "        break\n",
    "print('Total score (averaged over agents) this episode: {}'.format(np.mean(scores)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Training the agent!\n",
    "\n",
    "Now it's turn to train an agent to solve the environment!  When training the environment, we have to set `train_mode=True`, so that the line for resetting the environment looks like the following:\n",
    "```python\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The autoreload extension is already loaded. To reload it, use:\n",
      "  %reload_ext autoreload\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "import datetime\n",
    "import torch\n",
    "import numpy as np\n",
    "from collections import deque\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "%config InlineBackend.figure_format = 'retina'\n",
    "\n",
    "#pytorch\n",
    "import torch\n",
    "import torch.nn as nn\n",
    "import torch.nn.functional as F\n",
    "import torch.optim as optim\n",
    "from torch.distributions import Normal\n",
    "\n",
    "# Some more magic so that the notebook will reload external python modules;\n",
    "# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython\n",
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "using cuda:0\n"
     ]
    }
   ],
   "source": [
    "# defining the device\n",
    "device = torch.device(\"cuda:0\" if torch.cuda.is_available() else \"cpu\")\n",
    "print (\"using\",device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Define policy network (Actor Critic style)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "state_size = brain.vector_observation_space_size\n",
    "action_size = brain.vector_action_space_size\n",
    "action_low = -1\n",
    "action_high = 1\n",
    "\n",
    "# define actor critic network\n",
    "class ActorCritic(nn.Module):\n",
    "    \n",
    "    def __init__(self,state_size,action_size,action_high,action_low,hidden_sizes=[512,256]):\n",
    "        super(ActorCritic, self).__init__()\n",
    "        \n",
    "        # action range\n",
    "        self.action_high = torch.tensor(action_high).to(device)\n",
    "        self.action_low = torch.tensor(action_low).to(device)\n",
    "        \n",
    "        self.std = nn.Parameter(torch.zeros(action_size))\n",
    "        \n",
    "        # common network\n",
    "        self.fc1 = nn.Linear(state_size,hidden_sizes[0])\n",
    "        \n",
    "        # actor network\n",
    "        self.fc2_actor = nn.Linear(hidden_sizes[0],hidden_sizes[1])\n",
    "        self.fc3_action = nn.Linear(hidden_sizes[1],action_size)\n",
    "        \n",
    "        # critic network\n",
    "        self.fc2_critic = nn.Linear(hidden_sizes[0],hidden_sizes[1])\n",
    "        self.fc3_critic = nn.Linear(hidden_sizes[1],1)\n",
    "    \n",
    "    def forward(self,state):\n",
    "        # common network\n",
    "        x = F.relu(self.fc1(state))\n",
    "        \n",
    "        # actor network\n",
    "        x_actor = F.relu(self.fc2_actor(x))\n",
    "        action_mean = F.sigmoid(self.fc3_action(x_actor))\n",
    "        ## rescale action mean\n",
    "        action_mean_ = (self.action_high-self.action_low)*action_mean + self.action_low\n",
    "        #action_std = F.sigmoid(self.fc3_std(x_actor))\n",
    "        \n",
    "        # critic network\n",
    "        x_critic = F.relu(self.fc2_critic(x))\n",
    "        v = self.fc3_critic(x_critic)\n",
    "        return action_mean_,v\n",
    "    \n",
    "    def act(self,state,action=None):\n",
    "        # converting state from numpy array to pytorch tensor on the \"device\"\n",
    "        state = torch.from_numpy(state).float().to(device)\n",
    "        action_mean,v = self.forward(state)\n",
    "        prob_dist = Normal(action_mean,F.softplus(self.std))\n",
    "        if action is None:\n",
    "            action = prob_dist.sample()\n",
    "        log_prob = prob_dist.log_prob(action).sum(-1).unsqueeze(-1)\n",
    "        entropy = prob_dist.entropy().sum(-1).unsqueeze(-1)\n",
    "        return {'a': action,\n",
    "                'log_pi_a': log_prob,\n",
    "                'ent': entropy,\n",
    "                'mean': action_mean,\n",
    "                'v': v}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Storage class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Storage:\n",
    "    def __init__(self, size, keys=None):\n",
    "        if keys is None:\n",
    "            keys = []\n",
    "        keys = keys + ['s', 'a', 'r', 'm',\n",
    "                       'v', 'q', 'pi', 'log_pi', 'ent',\n",
    "                       'adv', 'ret', 'q_a', 'log_pi_a',\n",
    "                       'mean']\n",
    "        self.keys = keys\n",
    "        self.size = size\n",
    "        self.reset()\n",
    "\n",
    "    def add(self, data):\n",
    "        for k, v in data.items():\n",
    "            assert k in self.keys\n",
    "            getattr(self, k).append(v)\n",
    "\n",
    "    def placeholder(self):\n",
    "        for k in self.keys:\n",
    "            v = getattr(self, k)\n",
    "            if len(v) == 0:\n",
    "                setattr(self, k, [None] * self.size)\n",
    "\n",
    "    def reset(self):\n",
    "        for key in self.keys:\n",
    "            setattr(self, key, [])\n",
    "\n",
    "    def cat(self, keys):\n",
    "        data = [getattr(self, k)[:self.size] for k in keys]\n",
    "        return map(lambda x: torch.cat(x, dim=0), data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. PPO agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from collections import deque\n",
    "from itertools import accumulate\n",
    "import torch.tensor as tensor\n",
    "\n",
    "# dir for saving trained models\n",
    "model_save_path = 'trained_models/ppo/'\n",
    "\n",
    "def random_sample(indices, batch_size):\n",
    "    indices = np.asarray(np.random.permutation(indices))\n",
    "    batches = indices[:len(indices) // batch_size * batch_size].reshape(-1, batch_size)\n",
    "    for batch in batches:\n",
    "        yield batch\n",
    "    r = len(indices) % batch_size\n",
    "    if r:\n",
    "        yield indices[-r:]\n",
    "        \n",
    "class Agent:\n",
    "    \n",
    "    def __init__(self,env,learning_rate=1e-3):\n",
    "        self.env = env\n",
    "        nS = brain.vector_observation_space_size\n",
    "        nA = brain.vector_action_space_size[0]\n",
    "        self.policy = ActorCritic(state_size=nS,hidden_sizes=[512,256],action_size=nA,\n",
    "                             action_low=action_low,action_high=action_high).to(device)\n",
    "        self.optimizer = optim.Adam(self.policy.parameters(), lr=learning_rate)\n",
    "        \n",
    "        # reset the environment\n",
    "        env_info = self.env.reset(train_mode=True)[brain_name]\n",
    "        self.states = env_info.vector_observations\n",
    "        \n",
    "        self.episode_rewards_window = deque(maxlen=100)\n",
    "        self.episode_rewards = []\n",
    "        num_trajectories = 20\n",
    "        self.online_rewards = np.zeros(num_trajectories)\n",
    "    \n",
    "        \n",
    "    def train(self,n_epochs=1000,num_trajectories=20,rollout_length=2048,mini_batch_size=64,gamma=.99,\n",
    "              target_score=-250,use_gae=False,gae_tau=0.95,PRINT_EVERY=100):\n",
    "        \n",
    "        for epoch in range(n_epochs):\n",
    "        \n",
    "            storage = Storage(rollout_length)\n",
    "            \n",
    "            for _ in range(rollout_length):\n",
    "                prediction = self.policy.act(self.states)\n",
    "                \n",
    "                # send all actions to tne environment\n",
    "                env_info = self.env.step((prediction['a']).cpu().numpy())[brain_name]\n",
    "                \n",
    "                next_states = np.array(env_info.vector_observations)         # get next state (for each agent)\n",
    "                rewards = np.array(env_info.rewards)                      # get reward (for each agent)\n",
    "                terminals = np.array(env_info.local_done)                  # see if episode finished\n",
    "                \n",
    "                self.online_rewards += rewards\n",
    "                for i, terminal in enumerate(terminals):\n",
    "                    if terminals[i]:\n",
    "                        self.episode_rewards.append(self.online_rewards[i])\n",
    "                        self.episode_rewards_window.append(self.online_rewards[i])\n",
    "                        self.online_rewards[i] = 0\n",
    "                \n",
    "                storage.add(prediction)\n",
    "                storage.add({'r': tensor(rewards).unsqueeze(-1).float().to(device),\n",
    "                             'm': tensor(1 - terminals).unsqueeze(-1).float().to(device),\n",
    "                             's': tensor(self.states).to(device)})\n",
    "                self.states = next_states\n",
    "\n",
    "            prediction = self.policy.act(self.states)\n",
    "            storage.add(prediction)\n",
    "            storage.placeholder()\n",
    "\n",
    "            advantages = tensor(np.zeros((num_trajectories, 1))).float().to(device)\n",
    "            returns = prediction['v'].detach()\n",
    "            for i in reversed(range(rollout_length)):\n",
    "                returns = storage.r[i] + gamma * storage.m[i] * returns\n",
    "                if not use_gae:\n",
    "                    advantages = returns - storage.v[i].detach()\n",
    "                else:\n",
    "                    td_error = storage.r[i] + gamma * storage.m[i] * storage.v[i + 1] - storage.v[i]\n",
    "                    advantages = advantages * gae_tau * gamma * storage.m[i] + td_error\n",
    "                storage.adv[i] = advantages.detach()\n",
    "                storage.ret[i] = returns.detach()\n",
    "\n",
    "            states, actions, log_probs_old, returns, advantages = storage.cat(['s', 'a', 'log_pi_a', 'ret', 'adv'])\n",
    "            actions = actions.detach()\n",
    "            log_probs_old = log_probs_old.detach()\n",
    "            advantages = (advantages - advantages.mean()) / advantages.std()\n",
    "            \n",
    "            ppo_ratio_clip = 0.2\n",
    "            gradient_clip = 0.5\n",
    "            entropy_weight = 0.0\n",
    "            \n",
    "            for _ in range(10):\n",
    "                sampler = random_sample(np.arange(states.size(0)), mini_batch_size)\n",
    "                for batch_indices in sampler:\n",
    "                    batch_indices = tensor(batch_indices).long()\n",
    "                    sampled_states = states[batch_indices]\n",
    "                    sampled_actions = actions[batch_indices]\n",
    "                    sampled_log_probs_old = log_probs_old[batch_indices]\n",
    "                    sampled_returns = returns[batch_indices]\n",
    "                    sampled_advantages = advantages[batch_indices]\n",
    "\n",
    "                    prediction = self.policy.act(sampled_states.cpu().numpy(), sampled_actions)\n",
    "                    ratio = (prediction['log_pi_a'] - sampled_log_probs_old).exp()\n",
    "                    obj = ratio * sampled_advantages\n",
    "                    obj_clipped = ratio.clamp(1.0 - ppo_ratio_clip,\n",
    "                                              1.0 + ppo_ratio_clip) * sampled_advantages\n",
    "                    policy_loss = -torch.min(obj, obj_clipped).mean() - entropy_weight * prediction['ent'].mean()\n",
    "\n",
    "                    value_loss = 0.5 * (sampled_returns - prediction['v']).pow(2).mean()\n",
    "\n",
    "                    self.optimizer.zero_grad()\n",
    "                    (policy_loss + value_loss).backward()\n",
    "                    nn.utils.clip_grad_norm_(self.policy.parameters(), gradient_clip)\n",
    "                    self.optimizer.step()\n",
    "            \n",
    "            #printing progress\n",
    "            if epoch % PRINT_EVERY == 0:\n",
    "                print (\"Epoch: {}\\t Avg reward: {:.2f}\\t action_std: {}\".format(epoch,np.mean(self.episode_rewards_window),\n",
    "                                                                             self.policy.std.detach().cpu().numpy()))\n",
    "                # save the policy\n",
    "                torch.save(self.policy, model_save_path+'ppo-reacher.policy')\n",
    "            \n",
    "            if np.mean(self.episode_rewards_window)>= target_score:\n",
    "                print (\"Environment solved in {} epochs! ... Avg reward : {:.2f}\".format(epoch-100,\n",
    "                                                                                          np.mean(self.episode_rewards_window)))\n",
    "                # save the policy\n",
    "                torch.save(self.policy, model_save_path+'ppo-reacher.policy')\n",
    "                break\n",
    "                \n",
    "        return self.episode_rewards"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 5. Train the agent"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# lets define and train our agent\n",
    "agent = Agent(env=env,learning_rate=3e-5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/jsingh/anaconda3/envs/reacher/lib/python3.6/site-packages/torch/nn/functional.py:1332: UserWarning: nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.\n",
      "  warnings.warn(\"nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.\")\n",
      "/home/jsingh/anaconda3/envs/reacher/lib/python3.6/site-packages/numpy/core/fromnumeric.py:3118: RuntimeWarning: Mean of empty slice.\n",
      "  out=out, **kwargs)\n",
      "/home/jsingh/anaconda3/envs/reacher/lib/python3.6/site-packages/numpy/core/_methods.py:85: RuntimeWarning: invalid value encountered in double_scalars\n",
      "  ret = ret.dtype.type(ret / rcount)\n",
      "/home/jsingh/anaconda3/envs/reacher/lib/python3.6/site-packages/torch/serialization.py:251: UserWarning: Couldn't retrieve source code for container of type ActorCritic. It won't be checked for correctness upon loading.\n",
      "  \"type \" + obj.__name__ + \". It won't be checked \"\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch: 0\t Avg reward: nan\t action_std: [-0.00508857 -0.00393812 -0.00068194  0.0005314 ]\n",
      "Epoch: 100\t Avg reward: 3.55\t action_std: [-0.16325815 -0.14115214 -0.15301496 -0.15037829]\n",
      "Epoch: 150\t Avg reward: 4.86\t action_std: [-0.21602997 -0.2180779  -0.20952952 -0.21437667]\n",
      "Epoch: 200\t Avg reward: 6.31\t action_std: [-0.27631453 -0.2708306  -0.26795822 -0.2564206 ]\n",
      "Epoch: 250\t Avg reward: 8.74\t action_std: [-0.32977566 -0.3458866  -0.3198295  -0.30882463]\n",
      "Epoch: 300\t Avg reward: 10.58\t action_std: [-0.374599   -0.4272292  -0.36870542 -0.38470435]\n",
      "Epoch: 350\t Avg reward: 12.94\t action_std: [-0.40225667 -0.47929788 -0.43741363 -0.42528996]\n",
      "Epoch: 400\t Avg reward: 15.48\t action_std: [-0.4524423  -0.51775837 -0.46896106 -0.4716383 ]\n",
      "Epoch: 450\t Avg reward: 16.71\t action_std: [-0.49641842 -0.5770384  -0.5013828  -0.51696676]\n",
      "Epoch: 500\t Avg reward: 18.41\t action_std: [-0.51913977 -0.6247325  -0.5606662  -0.5707567 ]\n",
      "Epoch: 550\t Avg reward: 20.41\t action_std: [-0.5897517  -0.67796326 -0.58788157 -0.6171231 ]\n",
      "Epoch: 600\t Avg reward: 21.69\t action_std: [-0.63157487 -0.7359719  -0.6559167  -0.649368  ]\n",
      "Epoch: 650\t Avg reward: 23.35\t action_std: [-0.669249   -0.79674906 -0.7090673  -0.6838516 ]\n",
      "Epoch: 700\t Avg reward: 24.95\t action_std: [-0.73934287 -0.8691567  -0.7716407  -0.7089834 ]\n",
      "Epoch: 750\t Avg reward: 23.74\t action_std: [-0.77253604 -0.8987815  -0.80865383 -0.7605746 ]\n",
      "Epoch: 800\t Avg reward: 24.99\t action_std: [-0.8142485  -0.92885935 -0.8631858  -0.8207625 ]\n",
      "Epoch: 850\t Avg reward: 26.08\t action_std: [-0.857562   -0.9808845  -0.8914893  -0.85932547]\n",
      "Epoch: 900\t Avg reward: 25.36\t action_std: [-0.897349   -1.0371611  -0.9247373  -0.90051377]\n",
      "Epoch: 950\t Avg reward: 27.26\t action_std: [-0.9200347  -1.0597184  -0.9632217  -0.94853544]\n",
      "Epoch: 1000\t Avg reward: 28.12\t action_std: [-0.9586835  -1.0851088  -1.0091133  -0.96756583]\n",
      "Epoch: 1050\t Avg reward: 29.75\t action_std: [-0.9841376 -1.0988591 -1.0493741 -1.0154595]\n",
      "Environment solved in 951 epochs! ... Avg reward : 30.30\n"
     ]
    }
   ],
   "source": [
    "scores = agent.train(n_epochs=2000,gamma=0.99,rollout_length=200,target_score=30,PRINT_EVERY=50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAwAAAAIqCAYAAACTwS1qAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAAWJQAAFiUBSVIk8AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvnQurowAAIABJREFUeJzs3XeUG+XVBvDn2sYFG9tgwHQMAQKh914MgZieUBJCCPDFJCGBQIAQIEDowRACmBKaAYMpphMM2IB777337u32etfbd+/3x4zWWu1ImpGmSs/vnD3Sjqa8Gs2u7p23iaqCiIiIiIjyQ5ugC0BERERERP5hAkBERERElEeYABARERER5REmAEREREREeYQJABERERFRHmECQERERESUR5gAEBERERHlESYARERERER5hAkAEREREVEeYQJARERERJRHmAAQEREREeURJgBERERERHmECQARERERUR5hAkBElMdE5CYRUREZHnRZiIjIH0wAiChSRGSgGbAm/lSIyAIR+a+IHBZ0OfONiJwoIv1FZI6IlIhInfk4SUT6ichPgi5jVIjIeIvru0lENovIFBF5UER2ttjup0n+NmpEZI2IfCwi59o4/n4i8riITBORYhGpFZENIvKDiPxFRHb05p0TkV9EVYMuAxGRbSIyEMANAOoBlMUWA9gV229q1AG4TlU/8b2AESMiNwF4A8AIVf1pBtt3BfAmgKviFjcCKAfQDUBbc5kCeF9Vf5tdiXOfiIwHcDqAagBbzcVtYVzjMesBnKuqy+K2+ymAH8xfiwE0mc+7AegYt+3TqnpPkmPfDeDRuPUbAFQA6A7j7wwANgD4raqOcvzmiCgUWANARFE1UVX3MH96wghYLgSwGkB7AG+LyG5BFjDXiUg3AONhBP8NAF4GcAKA9qraA8bncDSAJ2AEkb8OqKhR9UHcNb4bjED+LgC1APYB8GGKbY+LbQugM4DjAEwyX/u7iFySuIGIPA3gaRh/S98BOBNAB1XdBUAXAL8EsBzA3gCGiciFbrxJIvIfEwAiygmqWq+qwwD8xlzUGcCVARYpHwwAcCSMO9UXquqtqjpDVZsAQFWbVHWuqj4A4EcwgkrKkKpuVdVnATxlLjpeRE60sV2Tqs4CcCm215r9KX4dEbkcwN3mry+oah9VHR/3WVaZNWrHA5gGI7kbJCJ7Zv3GiMh3TACIKNdMAlBpPk/a7lxE2ojI9SIyPK7N+gYRGWwVVInII2Z76lZ3XUXk5Lj21i9avH6x+dryhOW7icitIvKViCw2+zFUmn0ZnhGRPZKU/SBzfw3m76eJyOcisklEGkXkmYT19xGRASKy0WwPvsLcf7dk5ycdETkJ25v93KeqKTsRq2oJgBZ3nePaul+X4jjrzXXOSFj+uLl8gIi0FZHbzDbr5ebyI0Rkpfn85jTvZYS53r8tXmtv7nu8iJSZ7eFXi8ibIvLjVPv1UPw1eLzdjVS1FNubCDVvJyICoJ/56wwAd6bYx1YA18BI+noA+Lvd4xNReDABIKJcFGur3NbyRSPwHQ7gHQDnAdgFRkCzF4BfAZhsETSOMR/Pstjl2XHPU70+JmH5/QBehHFn9kcw+i50hJG43AVgtogcYfUe4t7LtQDGAvgFgE4w2t/Hv34EgNkA+gLYE0bfib3M/U+B0bY7E7HzUwLgFTsbqDedztoA+B+A/gCOwfZ278D2QPnaZBubd7DPMX/9IOG1vQFMN/d9OoCuMJrf7A/gdwBmichlFvtsF5cQPpDBe0pnQ9zzrhluG7/dWQAONZ8/paqNSEFVVwIYbP7aV0TaOSwDEQWMCQAR5ZrTYDT/AYCVSdZ5D0BvGMHdBQB2VNVuMO5oPgQjiHxJRE6J22YSzOBZRA5K2F8swK8AcKSI7JLk9cQEYA2Ae2E0o+lotpvvAOBEGHdqe5plTaYNgNcBfAagl6p2B7AjgJcAQEQ6APjUfF/LAZypqjvBOD8/N5dnGqD2Nh+/U9W6DPfhhqthJHE3A+imqjsD2APGuY0F9GeIyL5Jtv8ljPO42GwmA8C48w/gKxifzQ8ATgXQSVW7wmgD/wKMhOsDEenl8ntKZ7+451sy3DZ+u3PMxwYAX9vcz5fm404w+hcQUYQwASCinCAiO4jIz7A9YK4H8JHFen1gNEVZBGMUlR9UtQYAVLVMVR8F8AiM2oN7Y9upajWMts9A3B1/EWkL4AwYAdV7MGofzox7vQu2B0gtEgBVfU5Vn1LV+bG7rqraqKrTAVwGYAmAo0Xk9GRvG0YSc42qrjG3b1DV1ebr1wL4MYy71heq6nhznSZV/R+M4NdxMyAR6Qigl/nrHKfbu6wLgFtU9TVVrQIAVS1U1QpVXQBgLozzdE2S7WMdkz9IWP47GJ/baAAXqepkVa03979RVW+HMfpRZwB/dfMN2fD7uOdT7G5k1mj0sdgu1lRumXmd2zE37jmH3SWKGCYARBRVp4lIgflTCKAGwDAYgWkTgD+q6nqL7W4wH19T1Yok+37ffDxPROL/T441H+Ob/BwDoznFOACjLF4/HUA7AGtiQbodZlISa1efLAEAgP+kaFoTa6P/iaouT3zRHMZxot0yxekR97ws6Vr+KILRlCuZWGDfagQiETkQwMkJ68XErpPnVbUhyb5j18n58QvNJEzMn8dTlM02s5/DweZIPbeai8epatoETER2NhPfH2AkTIBZS2SK1ViVOihSSdzzHknXIqJQYrs9IoqqHWA0kUlUBuBn5l10K6eZjw+JyH1pjtEFRhv5WJA7BkatQHyAH9+8xypBSNb8BwAgxgRZt8Boh90Lxh1lSVhtrxRlnJTiNcuahwRjsP2cRNG0NG3WPwTwJIBjReRQVV0c91osKZiqqitiC83mP7FOsgNE5LUk+471MUnWvChbfUWkb5LXFiFF3wYA64y+va00AbhXVUdkWzgiii4mAEQUVWNU9Rygua37oTDas18F4E0ROUdVN1tsFxtZp9VMqknsiO0JwAQYnWz3E5FeZlObWIA/WlULRWQxgGNEpJuqlse9PhYJROQ3AAZi+//iJhhNiWJt6rvASAg6J25rUqS+axubOGpjinU2pHgtmfhjJvZ38FtxqhdVda2ITIDRTOvXMPp4xCRr/rMrjAQz9jwdr2bGjZ8IrMl8vgxGO/130zTXiZ8IrAZGTckUAAMsag1i17eTO/nx5yXoWiAicohNgIgo8lS11gxqfgljrPmjACS7axv7v3dpXDONVD/NzYjMJkOxjqJnm82DzoQx6+1sc/kY8xhniMiOMDr0xpY3E5GeMDrwtoMRgB4PoyPwLnETOMWGFLW8lQugyaORdVIymyetNn892u/jJ0g5Yo2pVTMgETkSwOHm9ol9ReK/G4+0cY14dTMtfiKwvVT1UFW91OzvkK6t/nFx2/ZS1ZNU9S9JmgwtMh8PFpFONst2VNzzhTa3IaKQYAJARDnDDIZvgxHUXS0iZ1usVmQ+7mfxmh2xQP5sGCPE7AxgfFwzlPjXT4VxJ3mjRRv8i2HcOZ4H4DpVnRnrZBrHqomTE7F22qmaEKV6LZVYf4efmU1mMhFrW98xxTpOh7m08ol5rINF5ARzWSwZGKWqBQnrl2D73fNMr5MoiX2W7ZAwV0MKPzcfKwDMdL1EROQpJgBElFNUdSm239F9wmKVWJv5CzM8RHw7f6v2/elej9nHfJxjdRffrF3onbjcoVhgZjU3QYxVkmTH6+bjrtg+J0BK0rpRemwoyn0S1zXXPxTGMJNZMSch+978NdZuPjYqUGLzn1gNR6ymJ9PrJErGwhhxCgDuMUe2SsrsPB07f2+l6CRNRCHFBICIclFsJtzTReSchNcGmo8Xi8j5SEFErPoJjINxd/hAALEZbEfHXlTVDQBWwOiAG7ubapUAlJuPySb6uhnbh9rM1Cfm49Vm0NaCiJyF1CMMJaWqkwF8bv7aT0TOS7W+iOwKYEjC4nnm4+VJNrs3yfJMxAL9X5nDqh4Ao23850nWH2g+9rUxGZvd/iShZCagsQ7xxwP4T7J1RaQrjEnAOsFo+/+05wUkItcxASCinGNO6BQbQvOBhNe+hjFzrAD4n4jcZQanAAAR6SEivxCRr2ER3Jgdi+ebv54I6yYQY2A0pzg27vdEsfIdIyLPm7MTQ0S6ici9MCaacjIso5UPYNzZ7QhgqIicZh6jjYhcCmOSsPIU26fTF0b7707m/l8UkWNjd/rN4xwlIo/BSIr6JGwfS1COEZFn485BTxF5GcZdZrvj0qfzJYAqGE2eXjCXfWt21LbyOox5HzoBGC0ifc3gF2YZ9xSR60RkHIxRnBD3mtczAbtOVb8A8Jz56+0iMlREzogNgysinUTkKhjzTpwIY56N61U1VQdzIgopJgBElKtiwft5CTP6Asad+yEwgrtnABSJSJmIbIXR/vtzGG30k4kP6OPb/1u9Xpgw9CQAwJykKtbJ93YAW0SkDMZd1SdhdGZ+I0UZ0jJn6L0KRiJxCIAJIlIBoBLGLLdlADIep15Vt8AYQvQLGH0dboWRDNWJSCmM0YzmwEjCuiBhvH5VnYvt5+AOAJtFZDOATQD+CGPCK1dGmFHVbTDeM7B9eNRWzX/i1q8DcCmAyTBGxxlglq9URCphjKw0CMboQr53xPaCqt4J4B8wJo7rA6O2q9a8LrfBSNgOhvH5XKSq3wRVViLKDhMAIspJqvoDtrfjfjDhtUpVvQzGbLtfwAhoOsO4a78MRh+CG5F8htcxSZ5bLRuXooy3wWjqMxtG0NUWRgD9FxjBp50RblJS1fkwJit7C8b73AFAAYxmHidjezv8TPdfrqpXADgFxuRS82DUinSFUbswCUZCc5iqWo1pfzuMxGEujHPQBGNCt3NUdVA2ZbMQH/BvBZAygFXVQhgB/m8BDIWRHMZqARbBSGiuBvBvl8sZGFV9Esbs0U8CmAHjPHUBUAhgBIy/iYNVdXjSnRBR6EkAI8gREREREVFAWANARERERJRHmAAQEREREeURJgBERERERHmECQARERERUR5hAkBERERElEeYABARERER5REmAEREREREeYQJABERERFRHmECQERERESUR5gAEBERERHlkXZBFyDqRGQVgK4AVgdcFCIiIiLKbb0AbFXVA7LZCROA7HXt1KnTLocddtguQReEiIiIiHLXokWLUF1dnfV+mABkb/Vhhx22y4wZM4IuBxERERHlsOOPPx4zZ85cne1+2AeAiIiIiCiPMAEgIiIiIsojTACIiIiIiPIIEwAiIiIiojzCBICIiIiIKI8wASAiIiIiyiNMAIiIiIiI8ggTACIiIiKiPBL5BEBErhMRNX9uSrLOJSIyWkTKRaRSRKaIyA1+l5WIiIiIKGiRTgBEZF8ALwGoTLHOrQCGADgCwHsA3gCwF4CBIvKMH+UkIiIiIgqLyCYAIiIA3gZQCuDVJOv0AvAMgDIAJ6jqLap6B4CjAKwAcJeInOpLgYmIiIiIQiCyCQCA2wCcC+D/AGxLss7vAHQA8JKqro4tVNXNAP5l/nqzh2UkIiIiIgqVSCYAInIYgH4A+qvq2BSrnms+DrN4bWjCOkREREREOa9d0AVwSkTaARgEYC2Af6RZ/cfm49LEF1R1k4hsA7CPiOyoqlVpjjsjyUuHpikDEREREVFoRC4BAPBPAMcCOENVq9Os2818LE/yejmAzuZ6KRMAIiIiIqJcEKkEQEROhnHX/z+qOsnPY6vq8UnKNAPAcX6WhYiIiOxRVRjjhhBRTGT6AJhNf96F0ZznQZubxe78d0vyeroaAiIickhVgy4CJSivrsfTwxbjme+WRPbzmbNuC/oOnIZ3Jq62tb6q4o6PZuOUJ0dgxKJC18qxYGM57vl0LkYtLnJtn0R+i0wCAKALgEMAHAagJm7yLwXwkLnOG+ay583fl5iPhyTuTET2hNH8Z3269v9ERGTPZzPW48QnhuPRIQuDLkqzfkMX49IXx2PyytKgixKYR4YswH9Hr8BLo5bj1g9nBV0cWxITlStemYgRi4vw0FcLsKI46fQ/zcYtK8EXszagcGst+r4z3bVyXfHfifho+jr838Bp2FpT79p+yR019Y34bMZ6TF1V5up+N2ypjmzybCVKTYBqAbyZ5LXjYPQLGA8j6I81DxoJ4HQAfeKWxVwYtw4REbngrk/mAADemrAKvz11fxywa+fAyqKqmLl2C14dswIAcM3rk7G638WBlSdIn8/c0Pz8m7mbcOy+KzFxRSnuPP8QHLF3skry4Hw8fR2eGroYVxy3N+6/+CcAgMam7cHXvPXl+NFuXVLuY6WNJCETtQ1Nzc/XlFShU/s2mLC8FG3bCPbo2hHnHbY7mxwFaMC4lXjme2Psl5F3nY0D01wndjz81QIMnLgaPz2sJwbccELW+wuDyCQAZoffm6xeE5GHYSQA76jqgLiX3gbwdwC3isjbsbkARGRnbB9ByHISMSIiyk5xRW0gCUBlbQNueGsqSitrccbBu/pyzKi1M3/8m0UAgEkrSrHosT4Bl6a1v386FwDwxrhVuP7UXth3lx0DLpG1uRu24P4v5rdY9s7vTsLZh+wWUIkoFvwDwH9+WIqXr82+m+ZAs9nZ8EWFuOmd6bjg8J745Qn7Zr3fIEWpCZBjqroKwN0AdgEwXUReFpHnAMwF8CME0JmYiIi89dwPSzFjzWasLq3Ce5PXen68V8eswHGP/YCXRy33/Fhuq65vDLoIAIAVxZW459O5+N/sDa1eK9tWF0CJ7EkM/gHg9sHpm1itKK7E08MWY+76LV4Uy5aJy0vwn++XYOOW1AMqVtTU46Z3puHaNyZjU3m6wRdDJqHFztaaegydtwnl1Zk33Rq+qBB//3QuFhdszbJwwcrpBAAAVPVFAJcBWADgegB/AFAA4EZV/VuQZSMiymVW7WXnbyjHoMlrsvoCTmfaanfb/qbTb+hibK6qx7+/W4KmpvRthCvMIGRLVXgDW7+d958x+Gj6Otw+eDbWlqbvlhehyhZLv359Mv47egUue2kC6uKaFFlZXbIN9342F59MX+fa8Usra3HtgCl4ceRy3JbQJ2RdWRV+WFiI+kajXP/5fimGLyrCxBWluOezea6VIZmmJsX3Cwrw9dyNaGhMfW6c6jtwGv70/kz0HTgt632NWBTtTuCRaQKUiqo+DODhFK8PATDEr/IQEVFrW2vq8fOXJ6ChSTFrzWY8+6tjgi6S6+x0Efzz+zMxblkJjti7K4bcekakmg55obSytsXvk1e501k7qO6advqJFlXUxj2vwT47J2/i9IdB07G0sBKDp63DMft2x8E9d8q6jOOWlTQ/n75mc/PzytoG9Hl+LLbVNeLms3+Eey88FEPnb2p+fezS4qyPnc6YZcX4wyBj7tX+1xyDy4/Z27V9T1ttvNfpazajqUmxsmQbPpu5Hn0O3wNH79vd0b6i3iE452sAiIgoHL6ZuwkN5h3yz2e1buqRL2LB1/wNW7GliqPIrN+cullJtMOs7C0t3N6ZefQSbwPwwVPXYlud0Sws1nneb7d9sL1G4vbBs7Pal6a4ekSMgQFeGb0Cl7+cviYm1zABICIiT+T7ne0gPP71QvR5fiwmLi9Jv3KOC+rqc3rZh+nvpL4x+HTLrxKoAiWVLWtinG4fZUwAiIjIE1GvIo+aGWvKMGD8KiwuqMC1A6YEXRzXhCc8toeXfXBWlWxL+lri/6NsP6aof8xMAIiIiFziNOlxM4hYXFDh4t6IDBKhFOxP781o8buTP8fXx650uTThxgSAiIiIwsOlrCjqd2jJOSdJcGKy/u6kNZZDzm6rbUiyvbOyhQ0TACIiIpeEqT13VEQ8jspamK6YXL587QTsmy2G5k02v0eqDsZRkBPDgBIREVHmFm7citfHrkDvQ3d3ddhF2q6uoQkPfTU/FBObOQn0oxzopgr6rV5SNeYh+Nsnc7Bw01b0u/Io/Hd0MKMheY0JABERkUuMZgXRu436q9cmoaK2AV/O3oiTD+iBPbp1DLpIWQvqU0jWD+TtCavw4VT3JvPKRtSbr3hpyNyNzcMU//LVSUnXi/o5ZBMgIiLyRMS/H/NKRVw75/kbygMsiT1RbGr1v9kbgy5CRqLUCdiJZAH8zLiJ0epSzEQc9f9vTACIiIgotKybaoQ3/IpCchKBIroivvmS3SsmvFeWu5gAEBERZSjMgWi+C+qTSXZNROFKyZO8AIALfRsi/rfPBICIiDyRGEzkU3ARZWELa6yumyjcZbcrh96KK6KSVEejlMkxASAiIk9E/QsyE/n4nsMqVVzd2OTdJ5UsOWGcb8+2ukZH6xdurcGMNZstE4f4Ra1mAs7zP1aOAkRERL7I8+9bS27e7XSrs2bYAlU3r5uGxiZc9+YULCmoQP9rjsVZh+zm4t4NUbmDnQvKttXhrKdHobahCb167OjCHu1/dlH/mFkDQERElAOiPF57tlQVm22Mr//R9HWYvLIMm6vqcf1bU30oWfSFuYnSCyOWobbBGKlndWmVK/ucs97eKFhR/3tjAkBERETNgg5rEgMryz4A8eur4to3puD4x3/AgHEr4/bT2rLCSlfK6CYnNTdhDsaDUFXXkPJ1TfI8mYqaBsxZt8XWsVkDQEREZEMuxi6JQUDUg4IgZNtkZvqazZi0shRNCjz+zSKXSkW5zuqym7ii1P72LpYlCEwAiIiIckCu9gFIx07TnzCIQsAYtRqGdLkjE/LkmAAQEZEn+OVLbsj0MgoqlvXysufflHus2vBPW11mf/uIfxZMAIiIiKhZxOOa0EqWkETtrnuYODl3dgL20UuK7e8v4n8pTACIiIgoNBKbMmUaHwcVniUrrxvliVqysKpkG7bW1Hu2//RBffIVsr6DH+34n/MAEBERuSXqdwXDwM45zDQQ9iOAjvIV4FY/EgD4fOZ63PnxHOzUoR3G3dMb3Xds79q+lxVW4LnhS/HtvALX9plvWANARESUoSgHezEPfDmvxe9hu8ns5jmOerttJ+oamjB6SZGjTtJWyVe662FrTT1uHjQDN70zDaWVtc3L7/x4DgCgorYBzw9fZrsMdlz/1lTHwX/ie8vzCgAmAEREREEJQxDx3uS1LX4PQ5mCUFPfiMUFW7MeljRpW3+Hy7P1z//Nx41vT8MlL45HY1Pm7yndlk8PW4xhCwowfFERHvpqgeU6m6syH6mpvrEJj3+9EHd9PAclZoKxqbzG1rZeJnxRn/GZCQARERGFlvVEYOnDZsvtkmzW0NiE858bgz7Pj0P/EdndrU4WFvodLg6etg4AsGFLNSavtDe+fSZNgD6fuaH5+ddzN1muk02s/N7kNRgwfhU+m7k+aYKRiagH8NliAkBERESRYqefgJPwbuj8AqwrqwYA15urhEFDFjUA6VICr+PowVPXNT//JkmCEYSo5w9MAIiIiFwS9aAA8L8PQOIpazW7sg9lqKpr8OEoucnLTtvZipWstqERIxcVWb6W7b6jiqMAERERUbOoBzaRE7Ze1x7IJjF2Y2Stez+bhy9mbWixrMBmP4JcxRoAIiLyBIfEJK+4OVwlbRe1eQbSqawxanYSg38AuOC5sVntO+q1fUwAiIg8oqqYv6EclbVsXpCr8r0joR8iF5OG8JKwew4zuZztbJNNYpFNslfb0Jj5gdOI+g0OJgBERB55ZcwKXPLiePR+ZjRq6r37Igor3qWlTOTaXegwGLGoEEUV3jR5sRMGB9UEKNohureYABAReeTpYUsAAMUVtfhkxvqAS+O/qN8ho3CwuopCnSSEsGzvTFqDi/qPR11DU8r1rM6rhPpkByfqlX9MAIiIfFDlQjOgkspaDJq0GqtKtmVfoAAwjiA77ARWoQ6+nJbNp/dSUlmL4YsK3d+xx02AyBscBYiIKCLu+Gg2xi0rwe47dcDEe89Fu7bRuofjV9AWpVgj1IFsSETp8wy7bGYETsbWnAwBXef8+0ouWt8eRER5bNyyEgBAUUUtlhZWZrSPT2esx/VvTcXE5SVuFi1U8vU7P5/ussa/13z9vDMRO1duNuvxOsgOaxAf9QEAIpcAiMhTIjJCRNaJSLWIlInILBF5SER6JKzbS0Q0xc/goN4HEeWXMARnW6rq8LdP5mDs0mJcO2BK0MXJCekmsUrHzesi4vFIUjn6tpoF8f6iHrza5WU/pKifwSg2AboDwEwAPwAoAtAZwCkAHgbwBxE5RVXXJWwzB8CXFvua72E5iYg8k8kX28Yt+T3xTRjlSRzmqRDk1pHhxbny+hLOJkn28u8r6n+7UUwAuqpqq28xEXkCwD8A3Afgzwkvz1bVh30oGxERJRGGWpBsLC+qxKcz1qPPEXvgmH27B12cnJFJIBXxSykwXsSsXtcmZDWEqJcJQMTrACLXBMgq+Dd9bD4e7FdZiIiCEom7TwGV0avg8JrXJ+PVMSvw85cnpB1OMQhuJVhBNw9JPHq6txWFP4WwcbUPgGt7Ij9FsQYgmUvNx7kWr+0lIn8E0ANAKYBJqmq1HhERRZxXAUlJZW3z86KKGuyz844eHSkzkUgKM5Cjb6tZEJ9b0EmeX7x8l1E/hZFNAETkbwC6AOgG4AQAZ8AI/vtZrH6++RO//WgAN6jqWpvHm5HkpUNtFpmIKL8k3GSM+hemHVFvFgAEP/GT06M7Wd+P2amjfwWQHVH/nCObAAD4G4Cecb8PA3CjqhbHLasC8BiMDsArzWVHwegw3BvACBE5RlWjOasOEVGYRewbsqa+EV/N3ojdunZA7x/vbmubMCU1Ue9j4YdcSNDcEKVhQMkbkU0AVHUPABCRngBOg3Hnf5aIXKKqM811igD8M2HTsSJyAYDxAE4GcBOA/jaOd7zVcrNm4LhM3wcR5Qc/7jyGnV8BaqaHeWfiajw5dDEA4H+3nI6j87Sjr9/NQ9JdF1H7y4laeeMFXftjZVlRZnOeAN5ey1FPfCLXCTiRqhaq6hcALoDRxv9dG9s0ABhg/nqWh8UjIiKfZfq9HAv+AeCJbxelPoZLX/68I936XHp5RoJsApQsGM2Ha8DNd/jf0ctd3Fs2ov25RT4BiFHVNQAWAjhcRHa1sUmsqVBn70pFRET5JOp3BYHw3QW2OqUhKyL56OlhS4IuAoDo/63nTAJg2st8bLSx7ilGAOLnAAAgAElEQVTm48qUaxERESVwKwBl07D8EbbEyo7Hv17oyn6i985zX6QSABE5RES6WSxvY04EtjuAiaq62Vx+nIi0eo8ich6MGYUB4D0vy0xE5IWo330iQxibf+TLEJF+i+J5HTB+FUq31aZf0ZQsxwnqnXMm4OSi1gn4IgBPish4AKtgjOnfE8DZAA4EUADg93HrPwvgYBGZCGC9uewoAOeazx9U1Yl+FJyIiDLT2KRo28b+PUQ/7zaGMYDPNbl+99jPQDKWhCQ7p1bLa+rtT3qX659VvKj/7UeqBgDAcABvAtgNwBUA7gZwJYAyAI8AOFxV4+urBgGYBeBEGInBn2HMFPwxgLNU9XH/ik5E+czt2v8ofPm4UcLRS4pw4hPDcd2AKWhssrdHP85M1O/+hUvqk8lT7T6/z2lQiYGX/yc/nr4e1XV2WpyHU6RqAFR1PoBbHaz/JoyEgYiIIujGt6cBAMYvL8FnM9fjlyfsG3CJUksXbkShGUjQbdWdnqLwn1FD0OfVjzIk239UPiOnXhq1DHf/LJrzwUatBoCIiFxy/xfzPN2/26HGiuLMxwN3m2txVAgjoygkKVFsbJJ8GFD/y+DV2Qvbp7K0sBITlpd4tv+xS73bt9eYABAR5an3p6zFxCRfjos2bUWf58fipnemo77RfhvgeEGFkW4EIWELZOyIYpmtJCZX6d5XsteDymOikUC15FaZkyXGQV6bvxkwxbN9R6EpZjJMAIiI8tjSwgrL5Te8NRWLCyowfFEhBk5Y7W+hQiCKX+tRLLMdbr6vIFvhhKEJUDIDJ672dP+5em1GGRMAIqII8vomY1HF9qH/pqwq8/ZgLvOzE3CrWWwdfjBhDIz8D1SzO16yc2j1NoK8OR/mmoFHhrgz3n++CfFHmhYTACKiPBbh7y9XLCusQG2DfyN5RCFg8D9QbXk8O4cP8c10x8KUGGR7XjmxXXQwASAiorz03A9Lcf5zY3Fh/3GthhhN2+5cgK019fhs5vo0a/onn0Kv+JjZyfvOpcQhKKOWFCV/Mc/Ob4hyN8eYABARUTTY/LK1G4P0H7EMALCyeBtGLW4Z1Ng51H2fzcP9X8y3eTRrbgYQEY5FUso0poxycBZm//f2NJRW2p8dOJdF+RJjAkBEFEFR/uLxWibnZltdQ9LXGpsUTw1b3Gr5N/M2ZXAkcsrq8+Sd/GDNWb/Fcjk/luhgAkBERJ5IvAObdfvggKKLT6avwyujV7RYluzucpCJGYOvaApTTYXdayjZ33K+JWaNTZkNkRwGTACIiCLIrY6DYQo+wiQ+jvlo+jrX9hvF0/3K6BU49ckRGDR5jSf7z+QaXFJQgfOfHYPfvjkFtQ3WQVi+BaNusPtR9Bu6GFe9MhFz1rWsCci3TsBLC8MzOaFT7YIuABER5abEACzrSXNc7gOQDbeCSzcnEvIiuaiua2xu/vTgl/OxsrgS/7zkJ74OFWp1pGd/WAoAWFZUibJtdb6VhQxLzPlDrn51UsAloUyxBoCIKI/ZieMyjfWiVLsQ5kmaglRT33KI1LcnrMaIRSlGgfFAustowcatvpTDrghd9q04/SuoS5glnH9G0cEEgIjIB2ENMO0E6aEJ5G2ewrAU129eXGFW53LammhMDBea6zaPhPO/HFlhAkBEFEFRjG1yqX1w0k7ADj+YSAapLpc5pLlxTvPqnKe60VFUUYNRS4pQ3xjdjrO5hH0AiIjyWKTiz4AK62asFKZZXyl63KpJ9PsyrK1vRJ/nx6FsWx36nnEAHrzkJ/4WgFphDQARUQ4prazF+1PWYG1plWv7jNod2kyK63SbqJ0TJ/xOUbwKRqPyGTl5/14nkFYJhhvn8fuFhc2dtd8cvyr7HVLWWANARJRD7vh4DsYuLcbe3Tth7N97o22b1N/eEYmRDJEqrD2sD2jNzZGRyJpXyVFlbQOamhRt0vzfoeCxBoCIKIeMXVoMANiwpRqrStKPUW0n1Ipaq5VAixuxc2UlV0K3qF23uWLI3I1BF4FsYAJAROSDx75eiKKtNa7tLwrBTS7fyY3C+XdDrvVZ8CO5ybFT5vicvTVhddp1RiwqzKgs5B4mAEREPrn/y/mBHt8qILcT4EWlLXVMRn0AfHqPUYgNo1DGTPnx3qoT5k5Ix0miHNbhhJ3q+850lFfXAzCSgXs+nYsFG8sDLlV+YQJAROSTHxa6edfLTtAQbLCQS8N+JnJtJuBcu12cAa/OQJCxciy4dVsuXS8riitRtLUGfd+Zjo+mr8OVr0wMukh5hQkAEVGesArIvbyj6HoTIJu7y50QKRjprogon1+/coKGHBrr3vH/CAdJyt2fzm1+XlOfO+csCpgAEBHlsVy6o+gVJwFQLvd7iJqoXNpOyul1EyC/a03GmIMWkP+YABARUTTYDE6i1NEzjEFqCItEAWpscnhF5Eg/hVzHeQCIiEKuvLoeX81pObReJoEj707b51UME8aAPxU/istwkch/TACIiELusa8X4tMZ6x1vF8UbcXUNRjvg9u0sKqg9jEYT+0dELVD3E5uNRVeYProwlSUfsQkQEVHIWQX/mXx3hn1UnuVFlTit3wic/tRIrC2tyng/vty1TnIqcyGo8fsqycVRgJz1G8lXrd/5hi3VAZQjPzEBICKilPyKo279YCZKKutQXFGLOz+eHVxB8ly6gNTrjqi53L/CqXVlmSfCVsJeK3hR/3FBFyFvsAkQERGllGkc5TQAW1xQ0fx83obMJwVyI8ZJFyjlQnCZqSg0ASqvqscLI5cFXYysvD52Bf717eIWyyprGrLaZ5g+uueHt/58vJo/gVpjDQARUZ4IvBNwtpF5iIKXTHn5GYT97q6f+g1bhC1V4QgmZ63djIv6j2uR4NqRGPwDwHPDl6Joa03WZbJqDlhT34hv523Ket92jVtW4tuxqDUmAEREOSroeLBVQOpTAO/2Ybw8j27ekXVrX0FfN274cOo6W+u9M3G1twUB8MvXJmHhpq1JX1dVFFfU4pnvlmCojQD8+RHZ12xYJaKPDFmAySvLst43RQMTACKiCMok2Mu0E3CmAaHrzQ0Cikyt3kbSTsCelsQffr+HIJulPPTVguaRp7xS35j+Dd73+Ty8NGo5/vT+TCwrTF1TUFZZ51bRWrCbNFFuYAJARJTHwtQm2C2Z5AleNJ+prG3AnR/Nxi0fzMTmbd4EbfH8agKU6poZMG4lrn1jMqatjs6d5IYmbxMAO4YvKmx+/vmsDSnX3bF926yPF/YRwch77ARMRJSj3BqtJTQ5QkAFsTqLdhKn/sOXNgdzHdq1wbO/PKb1fsJzdrO2srgSj3+zCAAwccUkrO53ccAlas2vJMnTw4Q8dq+oqceCjZl34id/sAaAiChHJY7WEniwGfLAxYnq+sa06wyOa1Lx+UwjEfCyxsWv2pxkQXRiO/f6xiYMm1+ARSnav/upcGtNKGu8Mi3Sh1PX4px/j8LbE1a5Wp5srSzehotfGB90MSiNyCUAIvKUiIwQkXUiUi0iZSIyS0QeEpEeSbY5TUS+NdetFpG5IvJXEcm+Ho2IKAB+DsUYmrjdZkH8ODMXPDc2FOXwmtVlluzS21bbcojKN8evws3vzcAlL47HpvLkEzy5fVe+Jkly9vdP51ofPzxXuD3m+b/v83lYXVqFR4YsRF1DEz6athbPfr8EY5cW4y8fzgq2jBR6UWwCdAeAmQB+AFAEoDOAUwA8DOAPInKKqjbfdhGRywF8BqAGwEcAygBcCuA5AKcDuNrPwhMR+SWxCVDkAp1EHkbUbp0Zp4mZm3lckMOATllZins+m9diWb+hxjCWjU2KZ79fin9ffbQvZXll9ArL5WOWFuOQnl18KYPfRi8panX+iVKJYgLQVVVbDYIrIk8A+AeA+wD82VzWFcAbABoBnKOq083lDwIYCeAqEblGVQf7VXgiIjdE8u6yg0JnE8xGPM2JpF+9Pjnl641NyT/8UpdHtRm/PPjx5f/17SKcd1hP9Dlij7TrOk0CrVYfMD5czYAo/CLXBMgq+Dd9bD4eHLfsKgC7ARgcC/7j9vGA+eufXC8kEVEIRDkQDlNbbU6w5a1Hhixo8Xu2H32qj8vpddXYpBk1t/tkxnrc/N4MrC2tcrxtRmwWMfB+QBQakUsAUrjUfIxv5Heu+TjMYv2xAKoAnCYiHbwsGBFREBK/6gP/8s82kI5YIO5nPw2vWF0zmbyrz2dtwNIk49tvKs9+Ztt4biVsy4sqcNbTo/DTZ8eguKI2o33ED+/pFqvrKvC/bYqcKDYBAgCIyN8AdAHQDcAJAM6AEfz3i1vtx+bj0sTtVbVBRFYBOBzAgQAWpTnejCQvHeqs5ERE9qgq7k7ScdG1Y4Q0cLAM4rzsA+BS0Oi0iOE8+9741WuTMOufF3h+nFR9XZw0lbn5vZnYsMXovPzo1wvx4q+PzbpsXrGba8bOTWKHbco/kU0AAPwNQM+434cBuFFVi+OWdTMfkw1IG1ve3eWyERFlbcSiInw6Y73la3a+8BPDIL87AWcT3ObAzfPcYPE5ZHoVba6qz6ootjksoEJRtq112ZYXVTY/n7lmc7alSlkCv/YQS/ifG97qvijlmcgmAKq6BwCISE8Ap8G48z9LRC5R1ZkeHO94q+VmzcBxbh+PiChxXHWKFjtB2XcLCjB77RbceHovnHKg5UjWvkuXfHmdm/ndUuyW92di1JLi9CuGhNX5d9rczO1mVxQ9ke8DoKqFqvoFgAsA9ADwbtzLsTv83Vpt2HL5Fo+KR0TkCTtNd0LXedXL3p0u86u2pN/QxRi2oADXpBlFJyhRrIhxet3bCf7bhChaspybwf9iUMSF6JLOjqquAbAQwOEisqu5eIn5eEji+iLSDsABABoArPSlkEREDrjdDCas7f2tZNMHwI136dW5ikLTpti5X7CxPJBaqOzzRPeTtzahy6ZbmrWW9zHJmcg2AUpiL/MxNg3gSAC/AdAHwIcJ654FYEcAY1U1s+79REQJnvluCb6YtQF/+1mr+w6uiuSkXg6KHIVA2UpUy51o0opS/PqNYGolyhLmBXDavMWLWN3LBCBXrhmKlkjVAIjIISLSqjmPiLQxJwLbHcBEVY311vkUQAmAa0TkhLj1OwJ43Pz1FY+LTUR5YuOWarw0ajk2bKnGHR/NyXp/qe5CZ3KH2ippyMXgI5NQ7cH/LUj6WlSSLbfKqQr8/bPsr99MhaGD6tz1Le+o+3kFpDtWDv7JUgCiVgNwEYAnRWQ8gFUASmGMBHQ2jKE8CwD8Prayqm4Vkd/DSARGi8hgAGUALoMxROinAD7y9R0QUc7KdKzwsAui9YPlMT0sR+Jn50Yw7Xdy5Wazpeq6puTH8fl9icML0I3r9bKXJri+T7tKKlP/H8mF+SUoeFFLAIYDOAjGmP/Hwhi+cxuMcf4HAXhBVcviN1DVL0XkbAD3A7gSQEcAywHcaa7PvyQicoWv/0x8PFho/kv6+Z49ONiK4kr8d9QKR9vUNTRhh7biOAjOVqf24Wkg8MXM9WjXxv77j1ofgMQr7ePp67F/j86ptwnNHyVFVaQSAFWdD+DWDLabAKP2gIgobyQGQlHqBBxVqc7x9W9ObZ5Yyo6JK0rwp/dmYu/unfD5n09Dxx3aulHEtPzINVQVM9duQdeO6cOQdyatwQdT19retxflb/I54P73d0vSr0SUhUglAEREYRaNluJkhxd3kZ0E/wBw7RtTAADl1fV4dcwK/PWnqTuWu9kHwGvfzivALR/Yn7KnvjHY5DVMN9wV2ZWntqEx/UqU88JTx0dEFHFuxwhuBx2WnYDtbJdhXJnvzRSc1rjMW1+OxibrbZYVVlouT3W8sUuLMWz+pqT7dENDY1NG+3cS/Dvld3OpbF3w3FjfjqUKDJq0xrfjUXixBoCIKILyNbReV1aF9u3aoGfXjkEXxXWXvjQeFx+1J16+tvXk8naSiW/mFbT4/fq3pgIA+l9zjDsFTLCqZBt+/fpktHXQPh/wPjH0Mvx3Wvbqem/utmdzBh//ZpFr5aDoYg0AERFFwuRVZTjr36NwWr+RWFyQ3QRVju5a+3hD+Zu5mzLeduxS6xltbx8829F+7N5A/+vgWSjYWuO4adOUVWXpVwohVcW4ZSWOtvGkLX8W0X/EKkfIQ6wBICIKqVTf83ZuRCZ+2Ue9E/CcdcbY7I2quP3D2fjujrMy3tenM9bZX9nJaYv2KQaQ/tqKXUdz1pc73vf/vT0Vo5ZYJypu8SLI3VReg4tfGB/IzMiJFJpxLUqet8qjOKwBICIiX7gZlxVV1GS1/dqyKpdKkpu8mvzM6+Af8KbCprq+MRTBf7aYAFAMEwAiojyRLzMB2+EowHWw6tqyquYmMW6eWz8/p+dDMBNvNsqq6oMugucyvRzu+iS4GZ4pXNgEiIiIHKmua0Sn9s7HpHczhs12X161hb7q1UloI8CQv5zh6n79TAAWF1T4dzAPxJqK5ap8TdrJXawBICKKILfa8zsNhP/17SIc8fB3+Ne3wY4kEuYgqEmBWz+YFXQxPBPmc58Phs4vwMujlgddDIo4JgBERGHlcqRllTQ4OYSq4vWxK9HYZDx6yU4Tneq6xtDONbCqZBtqG5qCLgblqOeHLwu6CBRxbAJEROSSsAajYeGksiFdDUd5dT2OfvR7HLtvd3z4+1PQxuFY9OnWnrSy1NH+vBamEZw+mb4Oe3XPvXkYiPIJawCIiHJUYvMe65mAvQssvQ5Z6xqaMGVVGb5fWJB+5UQ22j6tM0cKCvvQ6XUNTbjlfe9m1k20ra4R//p2sW/HIyL3MQEgIgqpbANotyokkjXH8bLGw8koPQXlzocEtbP30m11AMIxtH+qUz1g/Ep8My/zCcSIKP8wASAiiiCvg3svjulsPq3gw+6mgJp0rSrZ1mrZ9wsLsbK40nL90T6MrU9EuYUJABFRjrIzE7CdQDsMwXgQhs7bhOq6Rt+bAN08aIbl8mten4zGpvz8LIjIXewETEREvghTW3o7w5++MW4VFhdUYMqqMu8LFGdJofU4/EUVtVhXVoVeu3ZusTxM55WIooE1AEREecLR7Lc2tvOyhUymZbVr0KQ1ttYbt6zE03I4dd2bU/DL1yZl1O+BiCiGCQARkUvcjodTBdh2jiUuT3frZ+MTr5sdxTr4Rs36zdWYuqoMD3w5P+iiEFGEMQEgIsoT+dqWPxcNX1TY/NzlPI+I8gATACKiCHJrCM5sduNlOuF1EyAionzGBICIiEIn09qKyStLMWd9uculCafnflgKgMkSETnHUYCIiHJEulqBbANFx7UOPrc4WldWhWten+zvQQPUf8QyHNyzCyatLA26KEQUMUwAiIhCyuldcC9H5XltzAqMWFTk3QESZJKsvD1htfsFCblbP5gVdBGIKIKYABARRZBVrP/1vE0t10nICDJtVrO8uBLDFhS0LoMqOAq9Ny5/eULQRSCiHMY+AEREOeKT6es82e/yokpP9uuWmoYmALk1Gs6cdVuCLgIR5TAmAEREIZWySQ9H9GzWb+hiLE0yey4REbXGBICIyCVetsF34/hujxbj9O26PTFZvD+9N8OzfRMR5RomAEREUWQRS4d9oi+35i6wsqJ4m630pqkp3OeIiKLj9vMODroIGWMCQETkEl/boGcQx4Y9QUjkRbD+xawNru+TiChqmAAQEbkk6CZATU0tfw+6PNkq2FrjaP0vZ6cP7j+Z4U1HaSKiKGECQEQUUqn7AAcf3XudYDz29UJH65dU1nlUEiKi1oL/L5w5JgBERHnC7U7A6SQmKU47AQ+d33ruASIiyh4TACKiHNGUwS15LzvmJip02KTHKx9MWRt0EYiIAsUEgIgopNyOzd1uNuRkf3PXb8G/v1vi6vEzoQr844t5QReDiChQTACIiCLIKjkIc3vUmwc5G6ffqxGVot4xmojIDZFKAESkh4jcJCJfiMhyEakWkXIRGS8ifUWkTcL6vUREU/wMDuq9EFEuCnomsNQv+90HIN7GcmfNf7wK1KeuLvNmx0REEdIu6AI4dDWAVwBsAjAKwFoAPQFcAWAAgAtF5Gpt3ah1DoAvLfY338OyElEOqG9swvjlJTh6n+7YpXN71/evqvjP90uxbnMV7ulzKPbq3snmdhbLgk5AEvBuOxFROEUtAVgK4DIA36hq84jXIvIPAFMBXAkjGfgsYbvZqvqwX4Ukotzx0FcL8MGUtdija0eMv6c32rVNVXHq/A77kLmb8NKo5QCMTrKD/3BqhiXNTDZBerptnxq2GOcd1jOjfVfXN2a0HRERpRepJkCqOlJVh8QH/+byAgCvmr+e43vBiChnxUaMKdhag7HLitOs7Tya/mr2xubnk1e2bJ4SZKddNywtrEz5+qy1m30dhYiIyI727SIVHmckajUAqdSbjw0Wr+0lIn8E0ANAKYBJqjrXyc5FJFkPtkOd7IeIoquh0Ytg1b19poqlXxuzItBReERal+8X/52It248AecemlktARGR207stTPeu+lktGvTBi+NXI7nhi8NukieyIkUR0TaAbje/HWYxSrnw6gheMJ8nCMio0RkP5+KSETkXIqA3uqlZPMAbNhSjSeHLkZDU3B325M1jvrdwOm+loOIKJ0O7dqibRvBBYfn7s2JXKkB6AfgCADfqup3ccurADwGowPwSnPZUQAeBtAbwAgROUZVt6U7gKoeb7XcrBk4LvOiE1E+y7QFjJ1Jv2KrrCuryuwgLkqVe6iq41mCiYi8lsv/liJfAyAitwG4C8BiAL+Nf01Vi1T1n6o6U1W3mD9jAVwAYAqAgwDc5HuhiYiy9MdBM7Clqq7FsmQxdtib2f/8vxPx5LeLgi4GEVELYf/fmY1IJwAiciuA/gAWAuitqrYGeFbVBhjDhgLAWR4Vj4jIU09+u7jF78m+rFJ1/g3D99ucdVvw2tiV6VckIiJXRDYBEJG/AngRxlj+vc2RgJyIDefR2dWCERH5ZFHB1kCPrwqUV9ejoqY+6TrvTV7jY4mIiNyTrgbg7EN286cgHohkHwARuQdGu//ZAM5X1ZIMdnOK+cjbTkRkqbrO+7Hos7kDn9g8Ndu7+dNWl+Hjaetw5fH72Fp/wcZyXPfmFLQRwVe3no6Ddt+p1ToPfDkfh+7RejkRUZQ9cPFhOH7/nYMuRsYilwCIyIMAHgUwA8AFqZr9iMhxMCYBa0pYfh6AO8xf3/OqrEQUTaqKvu9Mx/hlmdxbcLEc6VZI7KGWcLvK6bj/V786CQDwyYz1ttb/w6AZqKk3/r3e9uFsfHv7mZbrfTl7g6NyEBGFQapOwDedeaB/BfFApBIAEbkBRvDfCGAcgNssRo5YraoDzefPAjhYRCYCiH2jHQXgXPP5g6o60dNCE1HkTFu9GSMXFznezo0OY1NXleGkA3bJaNukh/eooX/Ztu2dkFeXbuOkXkSUU3L5X1qkEgAAB5iPbQH8Nck6YwAMNJ8PAvALACcCuBDADgAKAXwM4CVVHedZSYkosooran07VmLQ/LuB0zD/kZ9ZvpbIjRHq7HzBWU3i5UVZiIjIH5FKAFT1YRhj+Ntd/00Ab3pVHiLKL36MVV9Zu30y8+lrNjvaNqx3q8JaLiKifBXZUYCIiHKBVWy8orgSADBr7ZaU27bqApCkrU+q+NutnCZVcrSkoMKdgxAR2fD6b4/HPjt3CroYocYEgIgoAN8vKMCdH8/G/A3lrV477z9jMHZpscVWqWVyp93ONraaCaV4zWlNBhFRNtq1FRzSk6OPpRKpJkBERLlga009/jBoRsp1rn9rquP9Jgbqfje9YVMfIvLakXt3wzyLGyfx3Ppf5HQktShhDQARkc/Wlla5sp90rXfWlBnH8SUwZy9gIvLBrl3a21qP/5JSYwJARJQg07s+hVv9Gz3ISmKpb/twlm/H5pctEYUFayPTYwJAROSCxibFLR/M9PWYMxM6CW+LG0EIAMqr623vK9sx/P0YIYmIiNzBBICIKIFkcD971lr7HV3djJXXmc18xi0rxtoy66ZFqWo0crmNKxG5Y89uHQEAN57WK9BynNhrZ9vrtmvLmxKpMAEgInJBk4M4OpMEI5lN5TUAgN++6bzTsJtEPJtwmIgCNuz2s/DRH07Bg5f8JNByvPDrY22ve/9FwZY17DgKEBFRAq/vivvdWsatoT5T4b02oty1U8d2OPnAHkEXA7t16WBrPQWwX48dvS1MxGWVAIjIWZluq6pjszk2EVFUuZ0ApGrrX1Fjvx8AEVGYtRFhfyOXZFsDMBqZ1/q2zfLYRESeyKSJjpPvJDebAAHAmtJtSV874fHh+NWJ+7p6PCsigpGLizw/DhH5z4uYu10bQYOTtpMA2rSRrAcscCKXRxPKNgF4FK0TgJMB9AGwAsB4AAUA9gBwBoAfARgKINjGqkREAVFV179MG1N8idY2NOHdSWtSlMd8zLIM9Y1N+P2707PcCxEF6aVrj8WtH/gzfPCPduuCJYUVvhzLiTMP3i3oIvgiqwRAVR+O/11ETgFwH4DbAbysqk1xr7UB8BcA/WAkDkREoeRlH4DvFhTiR7t1dnWfTu+ieaEyYQhSIoqeS47ayzIBCFOzGztlyaaW4I9nH5jxtlHi9ihAjwEYrqovxgf/AKCqTaraH8BIMAEgojx1ywczXa8BaGjMPgHItlo9l6vKiSg8nPyvuu28gx3t+4yDdkWHdvnRQt3tBOAkALPTrDMbwCkuH5eIyHNuxO1Gcx13M4CXRi1zdX+ZCNENQiIiAMB1J++X1fZ7d+/Uatmhe+yE9286Oav9hoHbw4AKjHb+qRzk8jGJiCLF7WB5wvJSd3eYAQHnASAi7zlqjuTwf+0OCZOH7dy5PX60W2esKN4+0MKwv2Y8AGaouF0DMBHAlSJyidWLInIZgCsATHD5uEREkeFm/J9tMqEJj9nuh4gomfbtjLDzsqP3yngfdpoAZfr/6OHLDm+17J3fndT8/MnxJ+UAACAASURBVOxDcqeDsNs1APcDGAvgfyIyxnxeCKAngLMBnAWg2lyPiCgvudmh7l/fLnJlP+OXl2S1PfsAEFEqHdq1wed/Pg0z1mzGZUfvhV+9NtmV/Z558K4Ytyzz/197dO2IJ684Ent174T9e7QeoGGfnXfE2zeeiFlrN+O6U/fPpqih4moCoKozROR8AG8BOMf8iW/wugRAX1X1Z4wpIiKfOAnp3RzHetbaLVlt/8KIZSitrMX7U9a6VCKi4OzSuT3KttUFXYyctle3jthYXpPRtofv1Q2H79XN1fIM6nsyet37jeVr6eZc2b/HjvjmtjPRpUPqcLj3obuj96G7Z1zGMHK7CRBUdaKqHgpj3P/bAPzTfDxDVQ9T1YluH5OIKEpCMGpnCwz+KVecl2NBWhi92/ek9CtFxNH7dE8b/OcqV9+1iJwFYKuqzjYDfQb7REQJ3p20OugiEOWktm04HJXXDtp9J9x27kF4YeTyoIuSlN1K1nwevcztGoBRAP7g8j6JiHJKqpl5iShz+RzQUWu8HpJzOwEogdHJl4gor/CLhih4bUL0hzj5vvOw204dgi6GN1w4z09eeeT251ccmWJNwykH7mJ736ce2COjMuUTtxs+jQZwmsv7JCIiIkorTAnAHt06Bl2EUDt23+4Y1PckbKmqR58j9sB9n89Luu71p+6Pm89ON82UYd9dOqHbjjvYWjefRy9zOwF4AMAUEXkMwKOqWu/y/omIiIgshSj+B+D2nN+5RURw5sH2xtV/9PIjbO/3pF7b7/6nO/+3//Rg2/vNNW4nAPcBmA/gHwD6isgcAAVoPSeDqmpfl49NREREeSxMNQD5rm0bQWPYhjxLsHf3TkEXITBuJwA3xj3fw/yxogCYABBRpDC2IAo3/o2Gx2d/Og0/f3lC0MWgJNxOAA5weX9ElKdUFXWNTejQrm3QRWmWz+1FiaKANQDhccy+3fHEL47A/V/MB2Cvo6+VN284wdH6Ti6BfL5c3J4JmGPbEVHWquoa8POXJ6Bway1e++3xOCUCIzqsKa0KughEeS9s8dxlR++FAeNXpV1v1y4dUFJZ60OJ3GH3PP/yhH0hELRtY5yLTJzrcHK3+Bs1ks8RfhquzwRMRJStF0Ysx9LCSpRX1+Oa1yfb2qaxSVHf2ORxyZK78+M5gR2biAxhi/fuOP8QXHRkstbQ2w38vxOxa5fcGzJ0h7ZtcO3J++FXJ+6Hdm0zCzm9DOLzucbIswRARNqKSE8R2c/qx6vjElH0LSuscLT+pvJqnPX0KJzWb6TjbYkod/h9x/ekXqnHpu/coR3++5vj0+7niL27YdJ952Llvy7CGQftCgA4/yc98cpvjnOlnPnEySWwQ4ZJSS5wuw8ARORIAP0A9AaQLJ1VL45NRPnp3s/mYcMWYw7CP70/E8PvPNvR9m+MXYkvZm3AbecdjD5HJL9bF/tiGbesGK+NWYlLj94TvzqR9zOIwsLv+7k7tLM+4i297Y1Z32JfZjA6qO9JKKmsw247dcCqkm1ZlS/fpboerj5+H9/KEUaupj4ichiAiQDOAvADjHM/13xeav4+GsAgN49LRPltzvotzc+XF1U62ra4ohZPfLsICzdtxc3vzbC1zSNDFmL88hLc89k8rCnlFzRRWPhdA5BsYIB9dt4x432KSORmEP74j6cGXYRm+duoxxm36z4eALADgNNU9XJz2Req2gfGCEFvA/gJgH+6fFwiyiEVtQ2+Hcuq412yL/V1ZUZH3/gkY0kBmxwRhYXfTbqbPB4aLCrB7EkH7IIbT+sVdDHIAbcTgHMAfK2q8fM5CwCo6jYAfwSwGcBjLh+XiHLI1FVlvh2rXRv7X7EPD1nYahlHmSAKj047+DtssNfzXLlRE7Bj+7a45Kg9XSjNdn7829tn504YcL2zIUCdyPd/3W4nALsCWBb3ewOA5nowVW0AMArABZnsXER6iMhNIvKFiCwXkWoRKReR8SLSV0Qs34+InCYi34pImbnNXBH5q4iEZ4BxIspYNjfh2lgkAI7Gkc780ETksht8vgutSf75uPV/oXOHzLpL3v2zH+OE/XfGh78/BXMeugAvXRv+zsQnH9CyQ/X4e87FT3/SM6t9cuqW5NzuiFsGoEvc7yUAEnvI1QHoluH+rwbwCoBNMBKJtQB6ArgCwAAAF4rI1Rr3FykilwP4DEANgI/MMl4K4DkAp5v7JKI85aQGwEq+30UiCpOdMgyYM5WsBsDJ/4XeP94t5evt27ZBncMhjm/pfRBu6X2Qo22ClmmykynJ89s3bp/tFQB6xf0+A8D5IrK7qhaJSGcAlwNIPyuGtaUALgPwjao2/zWIyD8ATAVwJYxk4DNzeVcAbwBoBHCOqk43lz8IYCSAq0TkGlUdnGF5iCjiEseBVlVHNQpMAIjC4bKj94pcH4Crjt8Hd//sxy6Vxj/5HjznArebAH0PoLcZ6APAqwB2ATBLRD4BMA/A/jDu1jumqiNVdUh88G8uLzCPBRj9EGKuArAbgMGx4N9cvwZGh2UA+FMmZSGi3JAYMDht08svQqJw6H/NMb4fM9s+AM9cfTR6du3oTmGy9OQVRwZ6/GTNqcgbbtcAvAFgCYBOALap6jcicgeAh2Dcna8C8BSAF1w+LgDUm4/xw4ecaz4Os1h/rFme00Skg6qmnINbRJKND3ioo1ISUagkfuc4nk1Y0DwHAREFJ4gO+X4ErepTS/ZTD+zhy3G8Fn8ZpPp88r321tUaAFXdpKofqWpJ3LL+MO7C7wlgJ1X9R+Id/GyJSDsA15u/xgf7sXq1pRZlbYDRFKkdgAPdLA/lJ6PpCO9gRF1DkzruBHzH4NmelYeIwitZE6Ao1gxmGxBn+/3HEdX85UuPC1VtBFDo4SH6ATgCwLeq+l3c8lhn4/Ik28WWd093AFW1nMvbrBkIf/d68tSWqjr8ZsAUVNY24M0bTsBBu+8UdJHIpsTvrCaHfQDu/2I+awCI8lSTq7czrRnJhPc3l6KYtFiJfx9MKpJzeybggSJynYjs7eZ+0xzzNgB3AVgM4Ld+HZco3uPfLMKCjVuxprQKN783M+jihM6ywgrc+dFsfD5zfdJ1VBWDp67FSyOXJV3HC4nV600OG/Uy+CfKX15PBJZP3KpBj/+fziZAybldA3A9zCBcRJbBGGlnBIBRqur6zD4iciuA/gAWAjjP4hixO/zJhh2NLd/idtkov8RPXBU/SywZfjNgCooqavH5rA04Yf9dsF+PHVutM3JxEe79fJ7F1ull88WRuGljmgSgz/NjMz4WEeWWpP96Agwuf31S4ujr9uR7QJxv3B4F6CcAbgPwFYDdAdwM4GMARSIyU0SeEZEL40YJypiI/BXAiwDmA+htjgSUaIn5eIjF9u0AHACj0/DKbMtDRMkVVWzvYz9ttfW9gBdGLverOC0kfn9f+crElJXtiwsqvCwOEUVImGoA2rdtgxF3nY1//eKIQI6f7Zlwq7lOrjRl8prbnYAXq+rLqvoLAD0AnATgHzBqAg4BcAeArwGUZnMcEbkHxkRes2EE/0VJVh1pPvaxeO0sGLMUT0w3AhARuceLG2bZfHEk1h6sLq3Cd/Ot7icQEbXkRwJgdxSg9u3a4Ee7dfGl3Xtu1BbkxJvImNs1AM3UMF1VnwLwNwBPACiGccZ3yHS/5iRe/WBMMnZe/IhDFj6FMRvxNSJyQtw+OgJ43Pz1lUzLQkTZa2hswsriyqy+UBKDeFXF3Z/MwQXPjUla49C8rsWyooqazAtDRHmj+47tgy6Ca4IO6u025dypY8YhZAtBv9+geTIKkIgcAOA886c3jGFABcBqAG/C6BeQyX5vAPAojJl9xwG4zSLTXa2qAwFAVbeKyO9hJAKjRWQwgDIYswn/2Fz+USZlIaLMxP+TV1Vc/vIELNi41dVjfL+wEJ/MMDocX/3qJKzud7HleiuLK9HQ2PpLZ9rqza6Wh4hy01NXHoULnhvTakKwIGLLfIln77/4MAybX4C6xiY8c/XRQRcnslxNAETkDRhB//4wrsVCGMH+SAAjVHV1loc4wHxsC+CvSdYZA2Bg7BdV/VJEzgZwP4zJyDoCWA7gTgAvKAduJ/JV/F/clFVlrgf/ADB/Q7KRf7f7z/dL8OLI5WjfzrOKUKJQ+9nhPfHdAi9H6M59B+3eBWP/3huVtQ3o8/w4T45hdxhQt9rgv3rdcYGMZme36VLPrh0x7p7eKK6oxRF7tx7jpcVEYG4VLge5XQPQF8b5/gHAQ6o6xc2dq+rDAB7OYLsJAC5ysyxElL2quob0K2XATlr/otnpuK7Bh4G8iULosqP3jnQC0Ll9W2yrawy6GNhn59ajmkVRLG7uc8SeGPf33jjz6VGBlieVnl07omfXjlntI19qTJJx+9bXOAD1AC4AMFZExonIwyJypoi402iLKITyvS2hE35Na09EuS3M/0mCmIDKzSO2aeN8b9m2p3CrQQa/j+1xexSgswHsDGPUnf4wmts8AKNZzmYRGSYid4uI5ay6RFHFhmT2xZ8rr2bRfGlUMEOKElH+ig88j9qnZdOUq47fx/PjZ98EyMG6WR7LS/w+tsf1TsCqWg3ge/MHItIdRkfgcwFcDeB8GNepJx2QiSjcYv+bq+sacdO70zPax8KNW1HX2ISj9+nGqd6JMnDfhYcGXYSshe0v/4s/n45nf1iKMw/aFYf03KnFa4///Aicd+jueGTIQhRsdTbKWL7UmnrxvzxVMpDvXx2e9n4TkZ1hBP7nw2gWtDuMv9k8P+2Ua/L9H4kTsX/Ib4zLbP69MUuLcdEL4/DzlyfgqzkbXSwZUX7491VH4cbTewVdjKyFLSw+Zt/uePd3J+H3Zx3Y6rWOO7TFhUfuiZ7dsmu3nkq2X0O5MoEWv4/tcXsUoE4AzsT2IUCPwfaAfyuMScBGIMNhQIko+mJ3s9aVVdnepqlJm9uk/vu7xc3Ln/hmES4/Zu/QBQJEYXb1CfsGXQRywO4oQFkfx0kTIIt1s62p8HtQxsP27Orr8cLG7WY4W8x9CoAaAKOxfRjQaarK4Tb+v737jpOqOv8H/nlmK1tZll3YRoelL713lahYAFGsEVsUO9F8rTH4MySmGI0txkQllqhRLDFiSyyo2LGgIIhIkypIr7t7fn/cO8vu7JQ7M7fO/bxfr2GYW8/MnL1znnsapSS2OYzfW8u3GN523qJ1DYWW1VsPBw6bdx2AUgp1oYNwx7Bj76G4tidKRV5vWuLXG71jupVgQRzXz0Sk6mf7xM+GYdaTn6G6bT5OHdzO6eQ4yuwAYBGA/0Ir8C9USh0w+fhE5HHBYGnzLuOXh6c+WRfxruXxd7+DvXEOBXjfgm/j2p6I3MeL4YsZBeuHzx2Co/70Fr7ZvNuEox3WOG2tcu2f4diKPgAtMtOavB7WqRgLr53AvmMwOQBQSg0383hEXsFrSXibdu7H0/qMvEFJV/OG7P7l98YmEnv8wzV48qO1uHBMJ+xzwdjhRE5LlTbfXuKVoCU7Iw0PnzsE//58Pc4YauxOuXtqwg/n67ysdFxxRFc88v5qXD6hi7aWP9gALB6JR+8EnKeUWmvleYjIna584jO8t3Jr0sf58LttOHfuR7h1ap+E9j/x7nfw+TptduCZjy3CjBEdkk4TEcXvxkk98OsXl5pyLL8U42xrqhXygY7pVoIx3UrsOTes6wMw66huuPLIriz4hzB9FCARyROR20RkI4AfAHzXaN1QEZkvIgPMPi8RuU+4wr8C8PVGY3ftG3v968248bkvE/opDBb+ichZU/pXmHYs19xwjoOVRdCk5wFI4ZCKhf/mTA0ARKQQwHsAZgFYD2Apmub3xdBGCTrNzPMSkXcopdUMJOLVJZtMuUs0d+GqpI9BRPHze0HMi0FLOFZ8j37PG3YzuwbgBgC9AMxQSg0A8FTjlUqpvdBmBT7C5PMSkUcopbBhR3wT4TTZ38S0EJG9zCzi+aW4GO7OfLiyctLzACR5AF6bvcXsAGAqgFeUUg9H2WY1APPqAInIdZZv2oXrn10cdp0CO00TuYETw4Ca9bc/rFMrTxY4zbr0hasITb4JkLPM6gPA3xdjzO4EXAlgXoxtdgMoNPm8ROQip93/PrbuORh2nVJAIIkrtHtGmiAipxTnZjmdBCJPMzsA2AWgNMY2HaF1DiYij1NKhW23GanwDwD/7z9LkjunJ+/7EbmPE50+TTunOH/H2knWNAFy9hN1+vx+Y3YToI8AHCci+eFWikgZgGMBvGPyeYnIZp+s/hFj//AmznnoQ9TWaZN87z9Uh9eWbLL0vKwBIDKHI8G0aW1gnG9znp9l6UjqDYx+T05/HuQtZgcAfwZQDGC+iPRovEJ//RSAbAB3mnxeIrLJpp37sXrrHpzy1/ewZttevLFsCx77YA0A4KqnPscFD3/scAqJ3Ouonm3w7MUjnE6GY7x+kzcrPYBx1SXo1DoXj10w1OnkmMrpr8aqeQAoPLNnAn5FRG4G8CsAXwI4BAAi8gOAImj56xql1EIzz0vkNKcvnHZZsXkXfnLH26irb3qh/kIfZ//FLzZYngb+RJCX/e2ng5xOQgOvj/vuROoz0gKYe84QW8+pfU+xr3yOjwLkkouzt3O1fUyfCEwpdTO0YT7/DeBHAHXQcu58AEcqpf5g9jmJnOaS657lrnzys2aFf4B3boi8yJFRgEw8Fq86TbUpzHY6CUlhHwB7WdKATSn1BoA3rDg2ETln444DYZcrANv3Ru74a6aDtfW2nIeIzGdaIc+jZUWryrjpAcFdp/VP6hjx1AhZ8T54I8leptcAGCEiJU6cl4iSU1cfvvCtlMKcF5fanBoicsqJ/crRvjgHx/Ru61ganIgBkj2nVWXchddNQI+yAmsOTinJ1gBARApF5DcAvrXzvERW8+jNqLiFa/4DaDUAz3+23t7EEHlYcW6m00lIyp9P7Y+3fjEewzsXx7Wf15sAOXPO2GctzTeh+U/SX4477uCzJZExpgUAItJeRKaKyPEi0iZkXbaIXAdgJYBrzTwvkRu447JnvUgBQITFRBTBP84dgq6leZjQvRST+pTF3D4jzZ2lmnjvaJtWOPPoNce8FlDm5wenC87sA2AvUwriInIntLv6TwF4DsAqEblYXzcOwDIAvwaQA22o0E5mnJeI7FUboaS/72CdzSkh8rbeFYV47edj8eCMwUg3ULh/95oJNqTKemYWXL1YXDSrCVAqTojIPgD2SjoAEJGzAVyqv1yqPzIA3KmvexlAOYC/AuislJqllNqc7HmJ3MSLP0Tx+NuClZjx0Ic4EKED7n+XbsLBOnbOJfNdMr6zbeea2LNN7I2c4tKLjGM3bcWzlQBxs2u4VpdmMbKIGTUAMwAcBDBaKdVbKdUbwARow38+AGAjgAFKqYuVUmwkTOQyX36/A1f963O8+tXGJsv3HaxDfb3C1xt3Ys78pXhz2RaHUkipbO45g6OuL8jOsCUdpwyqxK+n9LblXKGq2+bH3MatY/bHumkbGlT5vZWHu5sA+fzL8RkzAoC+AJ5VSr0XXKCUWgCtKZAAOFcptdiE8xCRBabdtxDzFq3Dzx75BDv2HgIAvLV8Cwb9+jVMvGMB3vt2q8MppFQ2skvrmNvcmeTwhkb8flpN0h0pj+2T2Ig4543qGHMbL5bNpvavQN/KQsuOb+Qj6VfV0rLzkzu5NVh2GzMCgEIAK8Is/0Z/fi/MOiJyif2HDjfdWbFlNwDg7Ac/xJ6DdVixeTfuX7DSqaSRD8T6qRYBxnb1xsjRfSoSK2xmpafhvjMHmJwae0QNTKT5XeX0gLcLZ25ppz62+vDfRK9yc4b/jOebCVfIdslHQwaZMRFYAMChMMsPAYBSap8J5yAiG4Qb5WfDjv0OpITIe7xwl75NQZapx4tW6AtXSEzzeADghHAdfq84oiuWbtiJHfsO4c5Tra8ho9Rj1nCcjPuIUkAtO/KSzQy1O2aZEQBw1VHdkj7GQzOGmJCSxJnVztzoUcwOypJN/+hGTd66luYlfJzcrHQ8ct5Q/PvSUejQOjepNAUl+1n9bMzhAR7PGdkhuYOR5cyoAQCA2SIyO9wKEQk3PqBSSpl1biIyySEO6E82Y9neuEsndMGwzsXo1DoXA3/934SO0bNRcxEzmmxEKzRaWSPi1JUq2SZAF4/vgsXf78APuw/ijlP7GdpHq0mx/h0n23a+U0keHjt/KFZs3o2TBlaalCqyilmF8HhzDa/5RC5UV88aAHIfLzStsYOIYHCHVk4no4noTYAoVHZGGh46x9laGCuN7NLaUMd+K/F6YUzSAYBSirP6EqWI2jrWAJC9UunH2mudIK3+7N3w3bogCZ7hhu+L7MPCOxE1CNcJmMhpLJfEdv6ojnhoRvQ5FazgVKGRecJZDBa8z3MBgIhME5G7RORtEdkpIkpEHo2wbQd9faTHE3ann8jNahkAkM04+ZAm2dqDG4/rifHdS81JTByipTvA79YU4UYBIkqWFzvi3gigBsBuAOsAdDewz+fQJiYL9aWJ6SIiIgswSIiudV6m00kIi1+bt6TK95Uib8NyXgwAZkEr+K8AMBbAGwb2+UwpNdvKRBG5kVIK327Zgw7FOUhP81yFH5GnJHOnNlbhK/pqe4s83dvmHz5zjFOHm/jr1ql9cO0zi81OVlgMHq3BT9X7PFciUEq9oZT6RrllOj4il9lzoBbvrvgBB2vr8Zv5S3Hkn97CtPvec80MlkTx8kNhI9qfZ4+yArTKjXyXP9EybiKXhA7FObjnjMOzFkcdBUgEpw9th5zMNADAhWO1ceJPHdIu/hM76I7ph4frvMOBSbeSHZ7TivO0Lcy2MCXxO39Ux8P/H90pypYU5MUagESUi8iFAIoBbAXwnlLqC4fTRCnELXeZlFI45a/v4av1OzGpTxleXLwBAPDZ2u34av1O9K4ojLq/S94GETXy2PlDo15j7Pqz/c9lo9CrvMDw9U4A5Gdn4OUrxmDJhp2Y4EAfBTMc17cMgYAgM01whEffgxHxXP+P71uOuQtX4av1O/G7k/pYlyiDZh3VDR1LctG1NB9VrXKcTo4n+CUAOEp/NBCRNwGcrZRaY+QAIvJJhFVG+iAQ2WLttn34av1OAGgo/AftP9R0Tr5H31+ND7/b1mQZKwnIjfwemEa7+5+MRD7X0MK/kWO0K85Bu2JnCmVmZJ30tABOqCk34UipIxAQPDNzBPYerENulvNFydysdJwxtL3TyfAU5781a+0FcAu0DsAr9WV9AcwGMB7A/0Skn1JqjzPJIzJXXZQSfOMf6q837sSNz7EPPJGZnAqg7WwCFM8x/B64eU3cM7qKuKLwT4lJ6W9OKbUZwE0hixeIyEQA7wAYCuB8AH82cKyB4ZbrNQMDwq0jspvRC/jCFVvD788fbHIhu9pAOykQprOsUba1EY/zNG4YBtQFSUgahwElK3iuE7AZlFK1AP6uvxzjZFqInMCfEyJ3GV9ditZ5Wbae04zCcbRjWF327lvZsuH/3drkWXy21OeWvmxkD18GALot+nOuo6kgIqKo/FAuyUwP4MXLRyW0r1ubAFlJRPCHk/uiTUEWWudl4t4zUrci3r5RgMhPUroJUAzD9OeVUbciShm8vBNNHVCB/u2K8EsX9oFpU5DY0Ipu/cu2+o5yZVEO3rlmApTSAqiwaXDtp0PkrJSuARCRASLS7D2KyBHQJhQDgEftTRWRMxr/FnNOAPKr6jb5OGtYe+TqY9NbrazReOmnDKq05BxONt1w6tTBa1hGWiBi4R+Ae6MjF/JDTRsd5rkaABGZDGCy/rKt/jxcRObq//9BKXW1/v8/AegqIguhzR4MaKMATdD//0ul1EKLk0zkGTv2HcKGHfucTgZRSrjvzIEY0601Fizfgm+37MGZPhum0BUFSt7rMIx9APzFcwEAgH4Azg5Z1kl/AMBqAMEA4BEAUwAMBnAMgAwAmwD8C8DdSqm3LU8tkY2SvX7f8OyXyIp2N43IAWaWS8pbtjDvYDEc3but/lxm2zmjObFffGPZt87LxA+7D0bdZnL/Csx5cSkO1NZjSv8KPPvp9w3r3ND8prQgC+2Lc7B6614M7lCEj1b96HSS4mbWKEDXHtMdt770tSnHIu/z3C+9Umq2UkqiPDo02vYBpdRxSqkOSqk8pVSWUqqdUmo6C/9E4R2orXc6CUSWGNqxFSb1cUdh3ExGAqSrJ3bD7ON7xXXcj244MuY2BdkZeO6Skfjt1D64YVKPuNOVKKN3qwMiePS8obhlcm/ck8IdhY04Z2QHp5NALuLFGgAiSoDz9+KImrr8iK6GtjPrTvKTFw4/fMwUau5g5K1cOsHYZ930uMY+ox5lBehRVtBstnG3fMJVrXJw1jDvNr/S8n/ytQBZ6Wk4vqYcL3y+PvlEked5rgaAiJLHPsDkBj/p1caxc1tR/mfn+qZSKMYiSjkMAIhSiBva3BIZZTS/WlGQdMMstWaxb5z4+M6TlW7PSEtkXOrkekoWAwAiE4ReVM/4+/tYu22vI2mJJJWaPJD7XTi2U+yNbNSnorDJ67RA6vw92PWnHW9n1AtGO58HeNkjCo8BAJEJQn8W312xFVc++ZkjaTHCrFEliCLJyzSvi5kZZbhTBlc1eW1F+b9bm3zzD+phhTkZTicBZYX2jfrkBbzyUxADACKLfLLaXcPN8UYY2cltBY3Q/G9FE6CjerbB5H7lqCxqgUfPG2rKMf/200Ext7Hrb9srTQzvP2sgMtIEZYXZuHRCF6eTkzQrb9jkZ2uBeuu8LMvOQe7EUYCITOCWn0WjZRr2VSQvsaL5mhUBgIjgjlP7m3a8G47tgaN6xu4obVfzPq/UHE7s1RYfXn8k8rLTkZHG+5zRPDNzBOYv3ojjalJveFyKjgEAERE5wsn22V7oApCRZrCTdILHH9O1JME93a8oN9PpJJjGrGFAw+naJh9XsOmaLzE0JjKBF+6LsTMcYnnJswAAIABJREFUeZFY1PAk4IEIwOo7+4U5GZg3c3jsDXVuagJkVUqGdWrV5PVFYztbdCYiZzEAIPKIzTv34/bXlmPB8i1JH8sLAQu5x4wRHRw5b4uMNEwfUhV7wwR4YRhQw/MKNHorR3QvjescA9u3Mvz9eqUJUDJun94PM0Z0QM+yAlx+RFdclgJ9CIjCYRMgIhPYUZS4+ukvGgr/7103gaNbkG1mHdkN8xatw679tbacb3x1CY6vKUfvikIUZGdYMsGWByoADGv8Vm47pQavLtmEbXsO4taXvk762Ef2KMV/l25GZVELdG9bkPTxnHbq4Co88dFaVBa1wLof9zVbX1bYArNP6OVAyojsxRoAIo9ofOf/lS83Nvx/38E6LFrzI+rrFfYdqjN0LHYCpngU5mRg4bUTbDtfTmY6pg6otHRYTS80AUpEy5xMnDKoClVFOaYc77aT++EP0/riyQuHWzZ3wgNnxx7pKFSil7BbJvfGI+cNwYuXjU7wCPbzQ80L2Y81AEQeVl+vcNxdb+PbLXtibhtsv7vux7343cvJ3xkkf8nPjm9M92SCzNACjxVt4dM80ATIqHCfT3B4x2QV5mTg5EHWNMMKyky3715kRloAo1O483MsVtSmkTexBoDIg4I/+C98sd5Q4V/bR3u++qnPrUoWUVycHQUohQKAMMtGdWmNar0G5dLxqdeOPXW+vdjc1PmaUgdrAIg8avmmXbjiifhnG35/5TYLUkPUVDLlazsKPJcd0QWX/vNTAMAFoztafj4rhfusAwHBC5eNwppte9CllMM8ksauOSPI/RgAEHmQCDDx9gVOJ4MoIre3NDi2dxlmH38A2/Ye8nwAEElmesAThX/e4SayHwMAIpd6+csNePi91ThjaHtM6tt0lsb6epeXrohcJvTGZyAgmDEyNQr+LECTUewDQEEMAIhc6qJHFwEAFn67Fcf0PrbJuoN19U4kicgwIy0NWBYxB1t1EFG82AmYyAPqQkpKLDiR2zGP+o9Vw4SG46egh8OAkhUYABC5UKxq2kR+Dt77divWbtubWIKILOCnQpwfZKQFcNmELsjNTMPVE7sZ3i80H8wY0QGnDo4+9CgDTKLksAkQkRlMLsiE/rjFem3EnPlLMWf+0sQTReRh46tLnU6CZdxUGL5qYjWuPLJbUrUBl07ogtZ5WXjio7Umpsy7tD4eLvqSKSWwBoDIDCZfm+tDftG/2bwr5HT8MaCmZo7r7HQSmulRVuB0EgAAU/tXoLxlC6eT4Rt2NgUiosQwACByodBBfs564MMmr+e+u8q+xJDrzZs5Av2qWhrefmSXYgtTc9j9Zw2Mut6uO9cTeqTu3X+ANwSIKH4MAIhcKPQHfdueg01eb951wM7kkMsNbF8UVyu0v541KOK680eZNzRmVasc/OvC4fHvaHGTukTdd+YAcw7kAb3K3VF7Ewn7jySGoSIFMQAgMoNLCyzkH2bN8Hn9sT1wQk25KccCgCEdWzX8v09FYZN1wSSfGzoefxL5PzvDmp+1l64YjaN7l8Xe0AFmXy/6t2uJe063L9hhWT461vCQFRgAEJnB5OszAwBySiAgGNShyNRjrrp1EhbPnogXLhsVdv2so7qadq7fTu1jehv0gux01/RnsMOzF49Eh9a5TieDLMBgi4IYABC5UOi4/0SxuP2HPT87w/i6JN5Mh+JcvP1/4xM/QBgBdmq1loGP9+KQTu55Wf4ZxJAzPZMVGAAQmSHB63NthBl9b32Jw3VSfNb9aHyOB7cUJ6yKc80e8Sfg8gbnfrld8NupfQBozbyumljtcGq8yS95hWLzTwhN5DLvr9yKSx5bhKpWOXjywmHISk9rWPfo+2scTBl50YHa8MGk31hRwGEFgPnGV5fgjWVbkJeVjgHtmjY5i/RxnzakHfpUFKJtYTZa5WZan0iiFMYAgMghp97/PgBg656D+Pvb3+GS8V0cThF5mV/v7GWmBXAwQk2aWVxfA+DBJoN/PLkGL3y+HiO6tEZ2RlrM7YNfQe+QjuRElBg2ASJygT+8sgz3vLEC+w/VOZ0U8gER4O7T+6O6TX7Y9XaVJ80oV79zrbnt/cNxewDgRcV5WZgxsiO6RciDRGQtBgBELvGHV5bhkfdWO50M8qh4Cu0BERzXtxyvzBqDUwdXWZcoG5TmZzd5He5zSDaecXsTIO/d/6d4cBhQsgIDACIXmTOfnX/JekaaXMTroXMGm35Mt4g0ClDbguywy4mI3I4BABFRCjB6l/CWyb0tOf/46tK49zGrqVGLRgFNl5I8cw7aSOMmQA/OGITS/CwcX1OO8d3jf8+W8MENYiuapf1+Wl/zD2oBDgNKVmAnYCKXef3rTU4ngTzIaAHpNI83+Qln3swRmLvwO0zs2RaFOZHnG0hU4wqACd3b4IPrSyEiuO6Zxaafi+xzyqAqfLByG+YtWud0Uuzjg2CRjPFcDYCITBORu0TkbRHZKSJKRB6Nsc8IEZkvIttEZJ+IfCEiV4qI+fXgRDEcqK3DkvU7I64/d+7HNqaG/CY9LfZl3+lRZeK939mzvAC/n1aDI3u2sSQ9I7q0bvJaXNYpONXKdHa+n+5t2QmZ/MmLNQA3AqgBsBvAOgDdo20sIicCmAdgP4AnAWwDcDyA2wGMBHCylYklf3viwzUY3a0EFS1b4PEP1+CNrzfj1SW8w0+pz0iTpEjlaLMLgMkENP3btcS1x0T9mSEbuCzmIvI8LwYAs6AV/FcAGAvgjUgbikgBgL8BqAMwTin1sb78lwBeBzBNRE5VSj1heap9ZNf+Q/jFU1/gQG0dfj+tBiX5WU4nyTHXPrMYHVvnYu45g9lcgBzTKjcT2/YcxKQ+ZU4nxVP6Vhbi2YtHOp2MmJyusSFrmToKEAMp0nmuCZBS6g2l1DfK2BVvGoASAE8EC//6MfZDq0kAgJkWJNPXbnt1OV7+aiPeWLYFv3zuS6eT47jvftiD13jXnxz01EXD8ZspfTBnijUdgMMx0nEx0lXcLWWUWOngXWlr2PmxnjKoClnpWlHop8Pb23hmh7g4Vjy+przh/8M6tXIwJf7gxRqAeEzQn18Os24BgL0ARohIllLqgH3JSm3PffZ9w/9f/mqjgylxj+APDJFVot0T6VySh84WjI4TjR/GLnfLjXeXJMOTCnMy8NwlI/HFuu2Y1Lc89g4O0ILp1P+WJ/erwPJNu/H99n24js3uLJfqAUC1/rw8dIVSqlZEvgPQC0AnAFEHYBeRTyKsYi71mdq6evzf01/g++378NupfdDJQMEqLcAAgKwxtKP5d8pSv6hBXmNlsNWjrAA9ygqsOwEZEggI+9vYKNUDgEL9eUeE9cHlLW1Ii2+45a6YFW596Wvc99a3Da9nProIr8waw6YC5Jhe5YWxN3KAkSZArv+7cDiBKXwpJSKHpXoAYBql1MBwy/WagQE2J4dMtP9QHZZv2oU+FYVRh/dbv31fk8I/ACzbtMvq5BFFlWwZ1aqAPSWaADl8NyNg8MtN5ZsuRGSNVG+XELzDH+kWWXD5dhvSQi6klMKUexfihLvfxU3PfxV12627DyZ8nnr+QpNFgpNUeTGLeTHNdsrJjDxVzbjqkob/T+qb+qM7ub62iMhjUj0AWKY/dwtdISLpADoCqAWw0s5EkXt8sW4Hlm7QJuV65P3VUbdN5gfohmc5GhJZI3iX2I6y9NUTtUvpiM7FNpyNcrMiV9LfOrUvxlWXYFLfMlw2oUtS5+Ewou6WErVp5Dqp3gTodQBnADgawOMh68YAyAGwgCMA+deB2nqnk0CUFCtmpY1UHrx0QldMH9wOrfMy0fG6+aafl5qKNktt28JszD1niI2psQ+Lu0TWS/UagKcB/ADgVBEZFFwoItkAfq2//IsTCUtlXrqbFE/ZiVXQ5DaZaQFcMLqjrecsyc+yJOhIxJVHdnU6CaZ75LwhDbMPGxlhzAxu+T4pPCMd6o1ibQIFea4GQEQmA5isv2yrPw8Xkbn6/39QSl0NAEqpnSJyAbRA4E0ReQLANgAnQBsi9GkAT9qVdvI2My/CREE9ywqwRG+GFo87pvdDtzb5KM7TZtq2O+6+5cRe+GWMfjOxJFvuvHBM5+QOEIsDBePRXUswumtJ7A2JiJLguQAAQD8AZ4cs66Q/AGA1gKuDK5RSz4nIWAA3ADgJQDaAFQB+DuBOgzMKU4patjH8KD6L1+3AS19uwNQBFehSqlXD8yYZWeGonm0SCgAm96+wIDXG9a1MfvTkZK6+UwdUoEWUTrKUWvhLbQ7eyKIgzwUASqnZAGbHuc+7AI61Ij3kbTc+17xzbm1dPY6/+x0AwLxF6/DB9UfanSzyEbMCy0hV+1WtWphzApeZ1Mf4yDcJFx5Z6nQEi6hE1kv1PgBEcdu25/Bwn5t2Hmjo08AaAO8J+Og7a1+c02xZflY6/nrmoDBbR5dosXdqo1qJUwZVJXgUY23SJ3QvTfj41JQXKsJ5/TUH+wBQEAMAohCH6pteIAf++r94/rPvWXXqQXec2t/pJNhmYs+2TV4/f8lIfHDDEehZXmBbGn51fC/cOKkHHr9gGMpbJl7zEK5Aetdph7/LeTOHs+MqEVESPNcEiCgZD737Hf75wRrMHNcZUyK0oa6ra1r42LbnIK544jO8OmuMHUkkE/2kVxunkxBTa70Tb7JCy8PZGWnIybT3El+Yk4HzR3eKvWECJvUpQ15WOnIy0zCgXZEl5/ArBlNE/sMaAHIdq6qj9xyoxc0vLME3m3fj5//6PGLz3tr68HMD8CfSe7xQa3NUzzboV5V8h1q7WVlmDFcgDQQE47uXYminYvsKrCwYE1GKYgBApkum+P63BStRc/Or+NOry2JvHKc9B2qbvF747daw29XVh38Hd/z3G9PTRNabM6W300mISgR49uIReC3JGiYvBDtEROQODADIVebMX4qd+2tx5+srsO9gnbkHDykfnfnAB2E3q40QALy4eEPkQ/NOoSuJAMf0Nj5ajBMEAhFBYU6Gucc1mCXd2Ckw2gy4tvJA51i/4FdBZC4GAORaB2vDN8VJlNE7pJFqACLZuvsAduw7lEiSyGNa5mSgd4V9nWrjYWYM6sSoMP+6cDhK8rMwsH0Rzrd5dmO/c9soQO5KDVFqYgBAFKI+zh/DZxZ9jy27DliUGrLafWcOROu8TEPb5mam4z+Xjca3vzF/WhGnmvC4penQkI6t8P51R2DezBHISjd3gi831nJQfFjJag6XxXrkIAYARI1s2rk/7n0O1plbU0HmEcTuvH1077bo1sZYk5PgndI0D0wwYDSFThaOQwt1XvhciYhSAQMA8g0jd5Auf/zTiH0AIjnEAMDzrjumh2XHzs4IGCrY+vEOp+vvRvrkS3FbHyZ3pYYoNTEAIPdK4FfgoXe/w7g/vIF/frAmoVN+8N02/OKpz+PahwGA9/WpLLTs2GO6lmDhtRPw5tXjwq63quxldZnOLU2HiMg4l8V65CAGAOQ5h+rq8fKXG7B0w85my29+YQlWbd2L659d3Gw/o3cbv92yJ870uP02pn+JiCt+8NoUZCM3K/qkXC5IZlyKcs0dtciVXF9FYQ63dQIm6/CrpiAGAJQQpRReWrwBT3y4BgdqTR6us+Ek4Rf/7e2VuOjRRTjurnewYce+huWx7sRb1dbZ7NGKyFx+uVNtd6BTWZSDGSM6oCA7Hb+Z0sfek8eJhR7v43dIZC4GAJSQd1b8gJmPLcK1zyzGY+8n1tzGqLp6hUseW4Sf3L4AX6zbjt+/vKxh+R2vHZ6cK9IPxL6Dddh/qM6yseUizRxM7pCRbqxkPG/mCEzuV44HZwwy9fxGs5357bCtjwhmn9ALn900EacPbWf5uYiIyDzR66SJIvjV8181/P///WcJzh3VaNxukwvaT360tmESrtP/1nTyrmhDdtbW1WPFlt045b73sHN/LTLTrYl3D9Xy1pRbCYDMNGPf+8D2RRjYvsi6tKRom/wAR+4hG7ihKR9RKmEAQK4R2g71x70HUZiTgU9W/9iwbPeBWsPHGzTnv9i+9/AEXVY11Xn7my2WHJfMkW4wADDCylAvnvLNXaf1j308FpiS55MP0W2jABGR9RgAUEKsKAiF3syfePsCLL55IhK9wdi48G+l9TvinzuA7OGGck2stsvxJPG+Mwegtl7hmN5lSaUpFra3JiJKbewDQK4RWuY4WFeP+Ys3IGCwFMcyC82Z0tvpJEQUKxcbyeZH9y7DcX3Lw84rkGiwM6RjccP/W+b4YGSfePgkEnLbKEDuSg1RamIAQK4R7kfoUJ1CgLmUDDpjaPsmr4NZ6u7TYzeZaezeMwYgOyOAmqqWSaepc2lu0sdIhNF4YGr/CkwdUIGaykI8et5QbV8X1JwQkflcFuuRg9gEiBIS7Y5RoteXcPtJo3/DaVxQcdtdLHLG1AEVeGbR95jcr7yhg+pxfctRW6dw5ZOfGTrGsX3KML66FNkZAXS8bn7cafjnBUNx4cOfoKQgC5dP6Gpon2Q78Sa6dyAg+NMp/Zos458SOSlcXmaeJDIXAwByjXAX+ICI4T4A/H0gALjt5BrMOrIbKotaNFleVx9fDmmRmQYA6F1RgC+/1yadG965ONouDUZ0bo2PbjwSmWmBhiDE7o6W7Nh5WMKFR36GRJSi2LiCEmJF4SLcRF0i0X+Dl23afXh/RgAELW9WtcpplkfrEswgd582AF1K81BTWYhfTuppeL/sjDRDQ2Q2pNNFZU2rJs0jd/JCsOiBJBJ5CmsAKCFWNLcJd0gRRO0E/Pna7fh0zY+Y8+JSdGubb3qayDsuPyJ6U5vS/KyEjtuhdS5emzXG9YUkt6eP3IvNJ4n8hwEAuVpAYreMnnLvQgDAx43mCyD/mXVk9ABgZJfWCR/bi4XrZFLs1KRiXtOmILGgkojIaWwCRK4R6SaUFwtfZK8zhraLmU+czkVOnz8ebAKki3Fn/MIxnVFemI20gOBOA5OzERG5BWsAKCHJFA+Cs/nmZTXNfuEKHVonYC8VncgJ1x3bI+Y2bg8kk02eu9+ds6wKZ1pkpuGt/xuP7XsPoSTBJmZERE5gAEC2WrZxF6be+y5EBM9cPALd2mjt9j9Z/SOe/XRds+21PgB2p5K8JjSY9BKrsncyAQWbhOsMfIgZaQEW/m3APGkO1u5RkHd/Ncm1wnUo23uwFuc89BE++G5bw7Irn/gM868Yjd0HanHSXxaGPZZAOPoDmcLpbGR1PubfCSXKbbVjLKISWY99AMgWd72+oknhHwC+2bwLgHb3P5KAuO/HyQ9qKgudTgIlKZmOvPyT8xcvjALEPGkOdvCnIAYA1MyOvYdw75sr8NqSTaYd8/2VW5stC7btz0iLNtMvawCccKnB2WvtdtqQqoT3ZT4yzgPlQSIiSgIDAGrmN/OX4vcvL8MFD3+MZRt3mXLMcGWvtEAwAIicDZes32HK+Sk+2Rn2XBouGd/Z8Lbjq0twzdHd0au8wMIUWcfqO2+sKYuse6JzhDAScgRzsnXYB4CCGABQM09+vLbh/3MXrgq7Tby/i+EKJ8EagPQovXzvfH0FDtXygmU3u6qJLxprPACoqWqJljmZ+N1JfRM6l98KyD57u808+bNh6N+uJa48sit6V7BJGxFRY+wETFGZVYgId5jdB2qxa/+hqDUAAPDvz783JxHkOvnZGU4nwXF+L6hbZWinYjx78cjkDsIvh4hSFGsAyBaRxvK/7dXlSI/SBwAAfth90IokUYq478yBTifBFCxqEkXG1ljmYCdgCmIAQFElcqkIe52OcKBITYzIP6YNrAy73Gj/gKN7tzUzOdax+Xc3mQITy1pEqYl9ACjIFwGAiKwSERXhsdHp9PlBtLIP7+yY65qjuyd9DDtbPtwQYRbfX/wk+fdB7sa/fTKKrbGIzOWnPgA7ANwRZvluuxPiJZEuuvHcRbjrf99EvXjXsxRgqtZ5mUkfw86vpCi3eXpb5vivb4DZnZRZYDIBr01ElKL8FABsV0rNdjoRXmNGe8HbXlsedT1/Y90n2cJj/3Yt8ema7Qnvf/XE6uQS4EKRPlM/t8llkEJkL/7eUpAvmgARkb2e+NmwpPafPjjxCb+ITMMIxREsoxJZz081AFkiciaAdgD2APgCwAKlVJ2zyXK3SL9/a7ftM+0cvCNBoWINDZuKzC5qJtUJmH+TRCmJMS0F+elXti2ARwDMgdYX4HUA34jIWCM7i8gn4R4AUrqnYkKjAMVZePhu654EzkKRWF12Kw7TZr+x04ZU+bpZSySxPpHcrHQM7lAEABhXXWJ9glyAgQaFE+5vxc955fpjDxczrj0mpYscZCO/1AA8BOBtAF8B2AWgE4BLAfwMwEsiMlwp9bmD6fO1yx//1OkkpBYTfiijFVY7l+Zh63fbmi2f1LcMUMC1R4cf1Ydie+S8ofh41Y8Y3LEI1Te+nNSxeKePKDWcPrQ9DtbVo7ZeYcaIDkkdy8+BFDXliwBAKXVzyKIvAVwkIrsBXAVgNoApMY4RdrYhvRZggAnJdKVwI5M8+dEaB1JCRtUZuMJnpgdwsLY+4vqoRwizslVuJu45/fCfQbRjU2TZGWkY1bW108mgIJaWXMPPAW1megA/G2NsXhQio/zUBCic+/TnMY6mwmOumbfY6SRQFHX1sQstj18QvZNu1HkbwkQAt07t03R/C36svd6sKOIwn95+W0nxc6GOiMhJfg8AtujPuY6mgshEKsZdy4dmDI49V0CUglmbguxmy/KyzKtMvOXEXmGXBwMP3pQl2zBCIaIU5fcAIHgbdKWjqXCJunqF91dubbKMv3/eY6QGIF6/O6kPAgLkZqbhpuN6xtw+mWxTUdQiib3JDPFM9JfUeRjMuUL/dkUN/89I40WfyA9Svg+AiPQAsEYptSdkeQcAd+svH7U5Wa5043OL8fiHaxPat75e4a7XV2Djzn3Yd4gjqzopzYIhNMd3L8XCa49AXnY68rLS8eTPhmH6/e83rE+2HDdjRAfMXbgKFS1bYGy30rDbeL0JEJFbnVBTjreWb8HSDTvx25DmfESUmlI+AAAwHcBVIrIAwGpoowB1BjAJQDaA+QD+6Fzy3CNc4T9Y6FJKYdeBWhRkZ4Tdd+IdC7Bi825L00fGlORlYlx1Cd5ctiXsegUVd2FaIGhbmNXwemin4ibr65O8lXvTcT1xbJ8y9CjLR1ogNQv6dr8r3l0nowIBwe3T+zmdjKiYn4nM5YcA4A0A1QD6AxgJrb3/dgDvQJsX4BEVq9G0j23etR/19QrT7luIL9btwK8n9w67HQv/7vLg2YOxause3L9gJZ74qHlgZ3YTj9C/oIgdXiMIBARDOraKfo4UnR/Uz83s/PzeKbLU/EsncpeUDwCUUm8BeMvpdHjVf77YgONryrFozXYAwLXPcAQgt1NKK1B3KslL+BihNQQsqHkPvzNKJczP5mBwRUF+7wRMBvyw+4DTSaA4xLrAZ1jQR8COHxWv9wHwUgHGrjpR1r1SOB76UyHyLAYAFNNbEdqSkzeN6Gz+RFOhrej4A05E5D68NlMQAwCK6dUlm5xOAsUh2l3V8sLshDrZxtoj9JS8seu8ZO6u52ammZcQIiJyHQYARDY6uldbR88/trrE0HahHW5jliVDNkjRgXyS4qUmTKcOaYfiXG2yuKsndnM4NURkFt6coSAGAEQ2+v3JfZFv4qy54UQbLefaY3rYck4RwV/OGGDJuVKNVWFBMn0OsjPS8PrV4/DcJSNxyfgu5iUqhJf6RRARpRIGAAQA+NXzXzqdBF8oyM7A29eMt/Qc0Zp+FLYIP4+DFec8pk+ZJeciexS2yEC/qpZxD+lKRETuxwCAsGjNj/jHe6udToZvtMzJtPT4I7uY0Mk3znriRNqbzz1ncPw7UUrhKEBERM5gAED4eNU2p5NAADq1zk36GIM7FKFV7uEAw80FrHHVpQntl5HuzTvSvJFORERuwQCA8Jv5XzudhJTxu5P6RFw3oF3LiOtmjOiAB2Ykdkf8ztP6IyBAm4Is/OPcIQkdI1l2xhnVbfLRq7wAADB9UJWNZyYiIkoNKT8TMJGdpg9uh2vmNZ8t+dg+bXFdlA64v/hJNXIT7Bx8Qk05hnVshZY5mchMbxrT23XXOXQeACuJCObNHIGlG3aipjJyUOUVbGNPRER2Yw2Az9TW1aO+3sXtQlLAwPZFzZbde8ZAVLXKibhPomXA7AztT7i0ILtZ4R9IvAlQvLvZnaWyM9LQv10RAhxv1NMY+xDZy83NQsleDAB8ZNnGXRj5u9cx/rY3sX77PqeTk7ISKZMmMka8CPCPcxJr8hPvjwB/NKxjtPbk8QuGWZwS+zFfERE5gwGAj1z82CfYtPMAVm/dixG3vo5XvtrodJJSSmaa9ueUSKEmeCd0zpTehrbvVJKLBb8Yj6Gdig0d13osyVmpKCcDwztH/66JiGJhrRsFMQDwAaUUvvx+B77dsqfJ8gsf+cShFKWW8dUl6Ng6Fx/dcGTCxwhelE8ZVIXbp9fE3L6sMDtqk6KgSMFIrB8B1hCYjz+8RETkFuwEnAI+W7sdd7++Akf2KMWpQ9o1W3/zC0swd+Eq+xPmEw+FNMNJpCwcbAKUkRbAlP6VGN21BB+v2oaLHl1kQgqbi7uAH+Nd2VH+T9UCtJFOwOwoTERm4M0aCmIAkAIm3/MuAOC/SzdhVNfWqCxqemc4WuHfztFbUk1RTgbu/+mgZssT+UxDy3et87IwoXubiNsHXFYgtCMbpWpWtepv0Aufl8uyMRGRb7AJUIr5esOuuLa/6FH/NQNqXxy76YwRn940EYM7tGq2PCcz/rg6XIHeycJRrDv+ADB1QAUAoLBFBo7okdikXuRvXghSiIhSEQMAn3vlq01OJ8FUFS1bxNzGjELH+aM6Rlx3y2RjHXkb8+KN0Nkn9MLvTuqDeTNHIDsjzenkuF4iIz0ldT4vZiqXad1oVm0iolTCACDF+P1HlG4qAAAaJUlEQVRH/81fjIu5jZG725F0b5uPP55cg59P7BZxm46tczG+uqThdQsDheNw31u0r9LqNuHd2uTH3KYgOwPTB7dDl9I8S9MS5Oe87eO3bru/6c36MtMD+H8JBPNERF7APgCUUjLSAkgLCOoizEz1k15t8NX6nQkfv2dZAaYNrIy53V2nD8DUe9/F1t0Hw/YTCBVvgd6KO5PH9G6LunqFo3q2QZuCbNOPnywrmovMmznc/IOSpx3Vsw3e+sU4FGRnoIg1AI5gyzAi6zEASFGrt+7Bb+YvxeZdB5xOiqv8ZkofnHD3u5afJy8rHa9cOQa19QoZaYlVtEULCq47tkeiSYuoX1VLXDi2s+nHdav/XDYKvSsKbTufn2swvKZ9ca7TSSAishQDgBR18WOLkrrTnYqm9K9AcV5WcgeJoxAnIshIs6bUV5Kf5PvwIBagifyBf+pWYv0KadgHIEW5ofD/86Mit5N3Qr3ehiSeYReNzsxrBVf9CPI3w1EMfoiIyEwMAFKMmwoKlx/RFReO7RRxfU6mvSPHBMv98ZRlzxjavslru0dysZPby/hm9wGwewhKu3NOukW1T0RE5H0MAMgSN07S2qjXVLaMuM2Xs3+Cs4a1j7g+1MSekSfGMiJY3kum4GdngOWmYI68YfqgKgDA8E7FKCuMPSQuEfkNf1hIwz4AKWb5pt24838rnE4GMtO12DIQ5VoTCEhchdz7zhyIh99bhdkvLEkoTdVttOEqkxkGlJxjdkCUivng1pP64NxRHW0bmpWIvCb1rnuUGAYAKebWl752OgkAGt9lN6/UFggI+rcrirld6BmLczNR2SoH54/uFJK2+Nl578SMsf7NKuQWtMgw5ThkLRFBddvYczgQEZG/MQAgS0Uqww7p2Cqh4/WtjD1sY3qaoLbRPAAfXH8E0gLSUKDm/Q9jn8HjFwzDI++vwpT+lSk506/9M/Oy6p2IiNyBfQDIEsGRdvYfqmu27oLRHXHH9H76dvEdV0TQo6wg6jYZgabZOj0t0KTwdevUPg3/j3eUH6+V4eIp5IZ+F8M7F+PeMwbiqCT7XiTjpAGHJ107eVDsCdji4ZYmQMZS4bGMR0RErsYaALLU3oNNA4D/XTUWnUsOt09OpBCWHq1jAYCM9AAQZf6z8dWluOu0/th3sA6T+1fghme/NHxur40CFOnz9cq7uOm4nuhRlo8+FYXs1EpERGQS1gCQpfYcqG3yunHh3yqxJt8KBATH15TjlMFVyEwP4OYTeiEtIBiaYLMkL3LHve/YCnMycP7oThjaqdj0Y3tpGFCrJpQjIiJ/YgBAlgqtAbDCWcPao3NJLh45bwgAICMtvmx99ogOWHTjUXjiZ8OaLA837KjXmgCR+0XKUn85Y0DD/+8+vb89iSEiIl9gEyAyRY+yAizdcHj24eDN1foYt1nNuAt7y+Sm7fiHdSrG05+sAwDDwyEW5jQf5eb/jq5utowBAJkt0p/AT3q1xcPnDkFOZhoGGBj9ioiIyCjWAHjcZ2u3O50EAMCg9uELKGcNa49M/Y785RO6NFtvRSuMGyf1QPe2+agsaoF7G91FjVf4UVtSIwJIjXfhLfEGj4GAYEy3Egzq0IojCBGRKexu+kjuxRoAD/vnB2tw/bOLLT3HI+cNQXXbfAyZ87+o20XqbFqcl4X5V4zCis27Mb57aczznT+qI/7+zncJpTWoZU4mXrpiNJTSClGJCh1NKJWE+7amDTR3lB234+8gERH5VeqWcEKISKWIPCgi60XkgIisEpE7RMSzdetmF/6/mD0RX99ydJNlo7q0Rml+dsx9o91V6FKaj6N7lyErPfZY8lce1Q3njeqIi8Z2jrltNCKSUOH/+JpyAECfikJUtfLPqDPzZg5HSX6W08kgIiIiG/iiBkBEOgNYCKAUwPMAvgYwBMAVAI4WkZFKqa0OJjFu9fXm378syNbawb94+Sg89+n3OKGmwnDTg0RTExo45GWl45fH9QQA3PfWtw3LS20qnN5+Sg3OHNoONVUtw773VG2JMbC9f0ZAIiLyq1T9DaP4+aUG4F5ohf/LlVKTlVLXKqUmALgdQDWAOY6mLgFvLt9s2bF7lRfihkk90SfKrLuR2vzHz1jo0K1Nvknniy49LYChnYojznzLayclSkTwn8tG4WdjOjmdFNdoGabzPRFZh30AKCjlAwD97v9EAKsA3BOy+lcA9gA4S0RybU5aUn7176+S2r84N7PJ69Z5xu+wnz60HZ68cHiTZaEXFbMvMm6ZtZV3T1KHcuCXsHdFIa4/toft53WTOVN6oyA7HeeO7Ijylv5pZkdE5CZ+aAI0Xn9+VSlV33iFUmqXiLwLLUAYBiB6T1eXOFBbh7Xb9iV1jLaF2di652DD6/vOjD5aTrc2eVi+aTcA4OzhHZDWrH19YoWpRMpgbgkGiCh+Zwxtj9MGt0uqgz4RESXHDwFAcDD35RHWfwMtAOiGKAGAiHwSYVX3xJOWmPr62NvE0r44B1+tPzxu/6AO0duA33/WINz75goMbF+E6rbNm+PkZTXNSi0yY3f4BYD8bGNZMNg/AQCKcjKjbGmtvCxvNVnIzw6f3nQHCl/5WenYFTIztJPinTDOKn4sBrPwT9EEwlS15mT6obhivYIW3voNI+u44xfQWsGG7DsirA8ub2lDWmx3w7E98H9HV2PZr4/G+OoSAMDorq1x46SeDePz//nUfjGP06F1Ln4/rQbTB7drWHbLib0AADmZabhkfBdceWRXAFpzoin9Kwyl79LxXZGrBwvB4wUF05WRJrjp+J4Ny+dM7tNQgL3vzIGGzpOMm0/Q0pWXlY6Z45IbnShozpTDk5cd3attxO2O7KHNRjyofVFDM63g52zEZRO6ICf4+TaaMK2yqAVqqrQsf2K/cuMJj9PD52qzMwcEeHrmCNTo/UomW3jOaKYPqgIAdG+bj17lBUkdKzjHRHpA8I9zhzTUij1w9qCY+wZnmR7WqVXEII3IT47prV0Hh3RohSK9ier1x2r31wqy0zFjRAenkpZSrjumOzLTtd/+206ucTg15CRxoh2snUTkfgAXALhAKfX3MOvnALgewPVKqd8mcPxPBgwYMOCTTyJVEJivtq4er3y1CbsPHMKC5T+gS2kehnZqhZ37arFkw04c0b0UX6zbjpMHVTXpzHqorh6fr92OmqqWyEgLYPOu/di25yC6t02sIKSUwlfrd6K8ZQu0ys2EUgqfr9uBTiW5Te7Yx7Jtz0Gs374PvSuadzr+euNOtMrNbDYU6cYd+7Fz/yFbOgcH32dFyxYNP0xm+GT1Nvy45xAmdC+NeEc0+J31rWyJ/bV1WLllD2oqC+OaGCrS57v/UB2+Wr8D/aqKwjTpMs+3W3YjOyMNFS1b2HbOSOrrFT5dux29ygsidvSOx7KNu9AyJwNtCrKxYcc+7DlQiy6lsfNkbV09PtO/1+CPMZGfBf8m+lQWNgwZrZTC4u93oH2r3LCztVNituw6gB92H0CPsuRugpAzBg4ciEWLFi1SSiV1B9QPAcAfAFwN4Gql1G1h1t8N4BIAFyul/pLA8W0PAIiIiIjIf8wKAPxw62mZ/twtwvpge4pIfQSIiIiIiFKGHwKAN/TniSLS5P2KSD6AkQD2Anjf7oQREREREdkt5QMApdS3AF4F0AFaU5/GbgaQC+ARpdQem5NGRERERGQ7v4yrdTGAhQDuFJEjACwFMBTaHAHLAdzgYNqIiIiIiGyT8jUAQEMtwCAAc6EV/K8C0BnAnwEMU0ptdS51RERERET28UsNAJRSawGc43Q6iIiIiIic5IsaACIiIiIi0jAAICIiIiLyEQYAREREREQ+wgCAiIiIiMhHGAAQEREREfkIAwAiIiIiIh9hAEBERERE5CMMAIiIiIiIfIQBABERERGRjzAAICIiIiLyEVFKOZ0GTxORrS1atGjVo0cPp5NCRERERCls6dKl2Ldv3zalVHEyx2EAkCQR+Q5AAYBVNp+6u/78tc3npdTBPETJYh6iZDEPUbL8loc6ANiplOqYzEEYAHiUiHwCAEqpgU6nhbyJeYiSxTxEyWIeomQxDyWGfQCIiIiIiHyEAQARERERkY8wACAiIiIi8hEGAEREREREPsIAgIiIiIjIRzgKEBERERGRj7AGgIiIiIjIRxgAEBERERH5CAMAIiIiIiIfYQBAREREROQjDACIiIiIiHyEAQARERERkY8wACAiIiIi8hEGAB4jIpUi8qCIrBeRAyKySkTuEJEip9NG1hCRaSJyl4i8LSI7RUSJyKMx9hkhIvNFZJuI7BORL0TkShFJi7LPcSLypojsEJHdIvKBiJwd4zxni8iH+vY79P2PS/S9kvlEpFhEzheRZ0VkhZ4fdojIOyJynoiE/R1gHqLGROR3IvI/EVmr54dtIvKpiPxKRIoj7MM8RFGJyJn6b5oSkfMjbGN5nhCRNBGZpefRYP6eLyIjkn2PrqWU4sMjDwCdAWwCoAA8B+BWAK/rr78GUOx0Gvmw5Hv/TP+OdwFYqv//0SjbnwigFsBuAA8A+IOePxSApyLsc6m+/gcA9wC4HcBafdkfI+zzR339Wn37ewBs1Zdd6vTnxkfD93SR/p2sB/AYgN8CeBDAdn3509AnhWQe4iNKPjoI4H0979wK4C4AH+nf1fcAqpiH+IgzT1Xp16Fd+vd1vhN5AoAAeAqHy1J/0PPsbj0Pn+j0Z2XJ5+90AviI48sCXtEz6GUhy/+kL7/P6TTyYcn3Ph5AV/0iNQ5RAgAABQA2AzgAYFCj5dkAFur7nhqyTwcA+/ULZIdGy4sArND3GR6yzwh9+QoARSHH2qofr0My75sP0/LPBADHAwiELG8LYI3+PZ7EPMRHjHyUHWH5HP17vJd5iI848pMA+C+Ab6EVuJsFAHblCQCn6fu82zifAxis5+HNAPKd/szMfrAJkEeISGcAEwGsghbNNvYrAHsAnCUiuTYnjSymlHpDKfWN0q9IMUwDUALgCaXUx42OsR/AjfrLmSH7nAsgC8DdSqlVjfb5EcBv9JcXhewTfD1H3y64zypo+TMLwDkG0ksWU0q9rpR6QSlVH7J8I4D79JfjGq1iHqJm9O8/nH/pz10bLWMeolguh3Zz4hxo5Zdw7MoTwbx4Y+N8rpT6CMCT0PLyNCNvyksYAHjHeP351TA/5LugRa45AIbZnTBylQn688th1i0AsBfACBHJMrjPSyHbJLMPuc8h/bm20TLmIYrH8frzF42WMQ9RRCLSA1ozsj8rpRZE2dTyPCEi2dBqDfYCeDuO83geAwDvqNafl0dY/43+3M2GtJB7RcwnSqlaAN8BSAfQyeA+G6DdnakUkRwA0GuZKgDs1teHYl70ABFJB/BT/WXjH0vmIYpIRK4WkdkicruIvA3gFmiF/1sbbcY8RGHp151HoDU/vD7G5nbkic4A0gCs1POmkX1SQrrTCSDDCvXnHRHWB5e3tCEt5F6J5BMj++Tq2+1N8BzkPrcC6A1gvlLqlUbLmYcomqsBtGn0+mUAM5RSWxotYx6iSG4C0B/AKKXUvhjb2pEnfJuPWANAROQzInI5gKugjXhxlsPJIQ9RSrVVSgm0TuRTod3F/1REBjibMnI7ERkK7a7/bUqp95xOj98xAPCOYBRaGGF9cPl2G9JC7pVIPjG6z46QZ+ZFDxKRSwH8GcASAOOVUttCNmEeopiUUpuUUs9CG5yiGMDDjVYzD1ETetOfh6E15/mlwd3syBO+zUcMALxjmf4cqR1acASGSH0EyB8i5hP9AtwRWofPlQb3KYNWxbpOKbUXAJRSe6CN+52nrw/FvOhSInIltPHbv4RW+N8YZjPmITJMKbUaWjDZS0Ra64uZhyhUHrTvtgeA/Y0m/1LQRjIEgL/py+7QX9uRJ74FUAegk543jeyTEhgAeMcb+vPE0Jk7RSQfwEhobeDetzth5Cqv689Hh1k3BtpIUQuVUgcM7nNMyDbJ7EMOEpFroE2K8xm0wv/mCJsyD1G8yvXnOv2ZeYhCHYA2uVa4x6f6Nu/or4PNgyzPE/qwnwuh5cnRcZzH+5yeiIAP4w9wIjDfP2BsIrAtiG8Cno7gBDwp/YBW5a4AfAygVYxtmYf4CM0T3QAUhlkewOGJwN5lHuIjwfw1G+EnArMlT8DYRGAFTn9OZj9Ef5PkAfpkYAsBlAJ4HsBSAEOhzRGwHMAIpdRW51JIVhCRyQAm6y/bAvgJtKrz4JjFPyilrg7Z/mloF7onAGwDcAK0IdWeBnCKCvnDF5HLANwJ7QL5JICD0CY+qYTWYetqhBCR2wD8HMA6/biZAKZDaw98mVLq7mTfOyVPRM4GMBfa3dm7EH60i1VKqbmN9mEeogZ607HfQrtD+x2077gNgLHQOgFvBHCEUmpJo32Yh8gQEZkNrRnQBUqpv4esszxPiIhAm9BuGrSBEV7Qt50OLWg9SSn1vElv1z2cjkD4iO8BoArAQwA2QPtDWA3gDjSKdPlIrQcO3x2J9FgVZp+RAOYD+BHAPgCLAcwCkBblPMcDeAvALmjjK38E4OwYaZuhb7dH3+8tAMc5/ZnxEVf+UQDeZB7iI8p31BvA3dCaj/0Arf3+Dv17m40ItUrMQ3wYzF/Ba9T5EdZbniegDYs/S8+j+/Q8Ox/ajVXHPyMrHqwBICIiIiLyEXYCJiIiIiLyEQYAREREREQ+wgCAiIiIiMhHGAAQEREREfkIAwAiIiIiIh9hAEBERERE5CMMAIiIiIiIfIQBABERERGRjzAAICIiIiLyEQYAREREREQ+wgCAiIiIiMhHGAAQEVFcRORNEVFOpwMARGSGiCgRmeF0WoiIvIIBABFRCtELw7Ee45xOJxEROSfd6QQQEZElbo6yblWSx/4pgJwkj0FERA5hAEBElIKUUrMtPPYaq45NRETWYxMgIiIfE5HZwWZBInK2iHwqIvtEZLOIPCgibcPs06wPgGjOFpGFIrJFRPaLyFoReUVEpoc5xkARmaef54CIrBaRe0WkLEI6u4jIUyLyo4js0c8zKcZ7qxSRu0VkpX6OrSLybxEZHGbbfBH5pYh8KSI7RWSXiHwrIk+KyMDYnyQRkXewBoCIiABgFoCJAJ4E8DKAUQDOATBORIYqpbbE2H8OgOsAfAfgXwB2ACgDMBjAyfpxAQAichyAeQAEwNMAVgMYCGAmgBNFZJRS6rtG23cF8B6AYgAvAfgMQBcAz+mvmxGRAQBeBdAKwCsAngHQGsBkAO+IyBSl1Hx9W9Hf8wj9PH8HUAugEsB4AG8D+CTG+yci8gwGAEREKUhEZkdYtV8pdWuY5ccAGKqU+rTRMW4HcCWAWwGcF+OUFwL4HkBvpdTekLS0bvT/PAD/gPb7M04p9Xajddfo5/ortGAk6B5ohf8rlVJ/brT9idCCgCZEJB1aEJIHYLxS6q1G68oBfATgARHpoJQ6AKA3tML/c0qpKSHHCgAojPHeiYg8hQEAEVFq+lWE5TugFbJDPdK48K+bDa0W4HQRuVgvLEdzCEBd6EKl1A+NXp4I7a78440L/7rbAFwE4CgRaaeUWiMilQCOglazcHfIcZ8XkbcAjA05ziQAnQH8sXHhX99nvYj8HsAdAI4AML/R6n1h0l4P4McI75eIyJMYABARpSCllMS5y1uhC5RSO0TkM2gF7B7Qmt5E8hiAywAsEZF/6cd7Tym1I2S7Afrz62HOVysiCwB0ANAfwBr9GQDeUUo1Cy4AvInmAcBw/bl9hJqQrvpzD2gBwBJo7+00EWkP4HkA7wD4WCl1MMz+RESexgCAiIgAYFOE5Rv151jNYGYBWAmtxuBa/VErIvMBXKWUWhFynA0RjhNc3jJk+1jpa6xYfz45RprzAEApVSciEwDcBGAagN/p63eJyD8AXKeU2h3jWEREnsFRgIiICADaRFgeHAUo9E5+E0qpOqXUHUqpGv1YJwF4FsAJAF4WkayQ4zQbXUhXFrJd8DlW+hoL7nOiUkqiPBrmSlBK/aiUmqWUqoJWQ3A+gK8BXArgLxHfOBGRBzEAICIioHkzGohIIYB+APYDWGr0QEqpzUqpZ5RSp0Br6tMZWkdbAAj2MxgX5nzpAEbrLxeFbD9KRNLCnK7ZcQC8rz+PDrMuJqXUCqXUA9A+k93Q+i0QEaUMBgBERAQAZ4lI/5Bls6E1wXk8WgdgEckSkZFhlmdA6/ALAMGRgZ4DsA1ae/thIbtcCaAjgP8GJxtTSq0D8Jq+/NKQ45+IMIELtDb83wK4RESOjZDm4SKSo/+/o4h0CrNZEYAshOkcTETkZewDQESUgqIMAwpow12Gduh9CcC7egfeDdDmARgFYBW09vzRtIA2tv4KaOPlrwaQDW30nh4A/q2UWgoASqndInIugKcAvCUiT0Hr7DsQ2tCfG6ENKdrYJdDG579DRCYC+BzaPABTALwA4PjGGyulDonIVGjj/78oIguhdfLdC6AK2twEnaA1N9oLoAbAMyLyEbSajvUASqDd+c/A4T4BREQpQZRSsbciIiJPCJ2hN4JzlFJz9e1nQxsydDy00XeuBFANrenLfwBcr5Rq0mFXRN4EMDY40pB+p3+WfoxeAEoB7IJ2F34ugAdDR9PRZ+O9HlqQUQit4P8igFuUUuvDvK8u0IYvPRJaofwLAL+GVlB/qPF7arRPKYCfAzgOWoG/Hlpw8zm0mogn9JGHKgFcDK02oTO0O/9bACwGcKdSKuxkY0REXsUAgIjIxxoHAEqpN51NDRER2YF9AIiIiIiIfIQBABERERGRjzAAICIiIiLyEfYBICIiIiLyEdYAEBERERH5CAMAIiIiIiIfYQBAREREROQjDACIiIiIiHyEAQARERERkY8wACAiIiIi8hEGAEREREREPsIAgIiIiIjIRxgAEBERERH5CAMAIiIiIiIfYQBAREREROQjDACIiIiIiHyEAQARERERkY/8f9xc2SbME0krAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "image/png": {
       "height": 277,
       "width": 384
      },
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# plot the rewards curve\n",
    "fig = plt.figure()\n",
    "plt.plot(scores)\n",
    "plt.xlabel('Episodes')\n",
    "plt.ylabel('Reward')\n",
    "plt.title('Reward Curve: PPO')\n",
    "plt.savefig('images/reward_curve-ppo.png',dpi=300)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 6. Watch the trained RL agent!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# uncomment this cell to load the trained policy for Reacher Unity environment\n",
    "# load policy\n",
    "policy =  torch.load(model_save_path+'ppo-reacher.policy',map_location=device)\n",
    "agent = Agent(env)\n",
    "agent.policy = policy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total episode reward: 32.69999926909804\n"
     ]
    }
   ],
   "source": [
    "total_rewards = np.zeros(num_agents)\n",
    "\n",
    "# reset the environment\n",
    "env_info = env.reset(train_mode=True)[brain_name]\n",
    "states = np.array(env_info.vector_observations)\n",
    "while True:\n",
    "    prediction = agent.policy.act(states)\n",
    "    action  = prediction['a'].cpu().numpy()\n",
    "    v = prediction['v'].detach().cpu().numpy()\n",
    "    \n",
    "    # send all actions to tne environment\n",
    "    env_info = env.step(action)[brain_name]\n",
    "\n",
    "    next_states = np.array(env_info.vector_observations)         # get next state (for each agent)\n",
    "    rewards = np.array(env_info.rewards)                         # get reward (for each agent)\n",
    "    terminals = np.array(env_info.local_done)                    # see if episode finished\n",
    "    \n",
    "    states=next_states\n",
    "    total_rewards+= rewards\n",
    "    \n",
    "    if np.any(terminals):\n",
    "        for i,terminal in enumerate(terminals):\n",
    "            if terminal:\n",
    "                eps_reward = total_rewards[i]\n",
    "                break\n",
    "        break\n",
    "\n",
    "print (\"Total episode reward:\",eps_reward)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "reacher",
   "language": "python",
   "name": "reacher"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}