first commit

README.md

@@ -1,2 +1,148 @@
-# pdppo
-Post-Decision Proximal Policy Optimization with Dual Critic Networks for Accelerated Learning
+# Reinforcement Learning for Stochastic Discrete Lot-Sizing Problem and Frozen-Lake game
+
+This repository contains code and resources the research developed at University of São Paulo (USP) and Politecnico di Torino (Polito) on using reinforcement learning, particularly the Post-Decision Proximal Policy Optimization (PDPPO), for the Stochastic Discrete Lot-Sizing Problem and a Frozen-Lake game. 
+
+## Project Structure
+This repository consists of two main directories: `Lot-sizing` and `Lake application`, each containing the related files and folders. 
+
+### Lot-Sizing
+
+`Lot-sizing` directory holds the following subdirectories:
+
+- **agents**: Holds various versions of PDPPO agent implementations and utility functions.
+
+- **cfg_env**: Includes environment settings and configurations files for the project in JSON format. Additionally, `generate_setting.py` is used for generating new environment settings.
+
+- **cfg_sol**: Stores the solution settings in `sol_setting.json`.
+
+- **envs**: Contains different environment definitions for the problem, like `simplePlant.py` and `singleSequenceDependentMachinePlant.py`.
+
+- **logs**: Keeps the log files for the model training and evaluation.
+
+- **models**: Stores various optimization models.
+
+- **results**: After executing the experiments, the results are saved in this directory.
+
+- **scenarioManager**: Manages different scenario setups.
+
+- **test_functions**: Stores functions to validate the models and generate plots and tables.
+
+### Lake Application
+
+`Lake application` directory holds the following subdirectories:
+
+- **agents**: Contains various versions of PDPPO agent implementations for the Lake problem.
+
+- **envs**: Contains environment definitions, like `frozen_lake.py`.
+
+- **logs**: Contains the log files and results from the model training and evaluation for different scenarios.
+
+- **results**: Stores the output from experiments and relevant figures.
+
+- Root level scripts `experiments.py`, `generate_tables.py` and `plot_figure.py` are used for running experiments, generating output tables and plotting results respectively.
+
+
+## Repository structure:
+
+The main components of the repository are as follows:
+
+```graphql
+
+├───Lake application
+│   ├───agents        # contains the implementations of various agents
+│   ├───envs          # contains the FrozenLake environment implementation
+│   ├───logs          # contains the logs of the agent's performance
+│   │   ├───frozen_lake_PDPPO
+│   │   ├───frozen_lake_PPO
+│   │   ├───results_1
+│   │   └───results_2
+│   └───results       # contains the results of the agent's performance
+│       └───frozen_lake_PPO
+└───Lot-sizing
+    ├───.vscode
+    ├───agents        # contains the implementations of various agents
+    │   ├───utils     # utility functions for the agents
+    │   │   └───__pycache__
+    │   └───__pycache__
+    ├───cfg_env       # contains the settings for the Lot-sizing environment
+    │   └───setting file
+    ├───cfg_sol
+    ├───envs          # contains the Lot-sizing environment implementation
+    ├───logs          # contains the logs of the agent's performance
+    ├───models        # contains the models for the optimization problems
+    ├───results       # contains the results of the agent's performance
+    ├───scenarioManager # manages different scenarios for the Lot-sizing environment
+    └───test_functions # contains test functions for the Lot-sizing environment
+
+```
+
+## Requirements:
+
+This project uses the following main dependencies:
+
+- [Python 3.8](https://www.python.org/downloads/)
+- [numpy](https://numpy.org/)
+- [gym](https://gym.openai.com/)
+- [matplotlib](https://matplotlib.org/)
+- [torch](https://pytorch.org/)
+- gurobipy (not included in `requirements.txt` due to separate licensing)
+
+
+## How to Reproduce
+
+1. Clone the repository:
+
+```
+git clone https://github.com/username/repository.git
+
+```
+
+## Navigate into the project directory:
+
+```
+cd repository
+
+```
+
+Install the required Python packages. This project was developed with Python 3.8. Substitute requirements.txt with your actual requirements file:
+
+```
+pip install -r requirements.txt
+```
+
+NOTE: You might need to replace the frozen environment file in your environment path with the frozen_lake.py provided in this repository for the Lake application to work properly.
+
+
+Run the experiments:
+
+```
+python ./code/Lot-sizing/experiments.py
+python ./code/Lake application/experiments.py
+```
+
+Generate the tables:
+
+
+```
+python ./code/Lot-sizing/generate_tables.py
+python ./code/Lake application/generate_tables.py
+```
+
+Plot the figures:
+
+```
+python ./code/Lot-sizing/plot_figure.py
+python ./code/Lake application/plot_figure.py
+```
+
+
+You can find the results of the experiments in the results directories in both Lot-sizing and Lake application directories.
+
+
+## Reproducing Results
+To reproduce the results in the logs and results folders, you would need to run the experiments with the same hyperparameters and seeds.
+
+Please note that due to the stochastic nature of the environments and training process, the results might not be identical, but they should be within a similar range.
+
+## Contact
+For any additional questions, you can reach me at email@example.com

code/Lake application/agents/PDPPO one critic.py

@@ -0,0 +1,321 @@
+# -*- coding: utf-8 -*-
+"""
+Created on Wed Mar  1 00:43:49 2023
+
+@author: leona
+"""
+
+import numpy as np
+import torch
+import torch.nn as nn
+import torch.nn.init as init
+from torch.distributions import MultivariateNormal
+from torch.distributions import Categorical
+
+################################## set device ##################################
+print("============================================================================================")
+# set device to cpu or cuda
+device = torch.device('cpu')
+if(torch.cuda.is_available()): 
+    device = torch.device('cuda:0') 
+    torch.cuda.empty_cache()
+    print("Device set to : " + str(torch.cuda.get_device_name(device)))
+else:
+    print("Device set to : cpu")
+print("============================================================================================")
+
+
+################################## PDPPO Policy ##################################
+class RolloutBuffer:
+    def __init__(self):
+        self.actions = []
+        self.states = []
+        self.post_states = []
+        self.logprobs = []
+        self.rewards = []
+        self.state_values = []
+        self.state_values_post = []
+        self.is_terminals = []
+    
+    def clear(self):
+        del self.actions[:]
+        del self.states[:]
+        del self.post_states[:]
+        del self.logprobs[:]
+        del self.rewards[:]
+        del self.state_values[:]
+        del self.state_values_post[:]
+        del self.is_terminals[:]
+
+
+class ActorCritic(nn.Module):
+    def __init__(self, state_dim, action_dim, has_continuous_action_space, action_std_init):
+        super(ActorCritic, self).__init__()
+
+        self.has_continuous_action_space = has_continuous_action_space
+        
+        if has_continuous_action_space:
+            self.action_dim = action_dim
+            self.action_var = torch.full((action_dim,), action_std_init * action_std_init).to(device)
+        # actor
+        if has_continuous_action_space :
+            self.actor = nn.Sequential(
+                            nn.Linear(state_dim, 64),
+                            nn.Tanh(),
+                            nn.Linear(64, 64),
+                            nn.Tanh(),
+                            nn.Linear(64, action_dim),
+                            nn.Tanh()
+                        )
+        else:
+            
+            self.action_dim = action_dim
+            self.fc1 = nn.Linear(state_dim, 128)
+            self.fc2 = nn.Linear(128, 128)
+            self.actor = nn.Linear(128, self.action_dim.nvec.sum())
+            
+            
+        # critic
+        self.critic = nn.Sequential(
+                        nn.Linear(state_dim, 128),
+                        nn.Tanh(),
+                        nn.Linear(128, 128),
+                        nn.Tanh(),
+                        nn.Linear(128, 1)
+                    )
+        
+    
+    def forward(self, state):
+        raise NotImplementedError
+    
+    
+    
+    def set_action_std(self, new_action_std):
+        if self.has_continuous_action_space:
+            self.action_var = torch.full((self.action_dim,), new_action_std * new_action_std).to(device)
+        else:
+            print("--------------------------------------------------------------------------------------------")
+            print("WARNING : Calling ActorCritic::set_action_std() on discrete action space policy")
+            print("--------------------------------------------------------------------------------------------")
+
+
+    
+    def act(self, state,tau):
+
+        if self.has_continuous_action_space:
+            action_mean = self.actor(state)
+            cov_mat = torch.diag(self.action_var).unsqueeze(dim=0)
+            dist = MultivariateNormal(action_mean, cov_mat)
+        else:
+            x = nn.functional.relu(self.fc2(nn.functional.relu(self.fc1(state))))
+            logits = self.actor(x)
+            action_probs = nn.functional.softmax(logits, dim=-1)
+            dist = Categorical(action_probs.view(len(self.action_dim.nvec),-1))
+
+        action = dist.sample()
+        action_logprob = dist.log_prob(action)
+        
+        return action.detach(), action_logprob.detach()
+    
+    def evaluate(self, state,post_state, action,tau):
+
+        if self.has_continuous_action_space:
+            action_mean = self.actor(state)
+            
+            action_var = self.action_var.expand_as(action_mean)
+            cov_mat = torch.diag_embed(action_var).to(device)
+            dist = MultivariateNormal(action_mean, cov_mat)
+            
+            # For Single Action Environments.
+            if self.action_dim == 1:
+                action = action.reshape(-1, self.action_dim)
+        else:
+            x = nn.functional.relu(self.fc2(nn.functional.relu(self.fc1(state))))
+            logits = self.actor(x)
+            action_probs = nn.functional.softmax(logits, dim=-1)
+                
+            dist = Categorical(action_probs.view(state.shape[0],len(self.action_dim.nvec),-1))
+            # action_probs = self.actor(state)
+            # dist = Categorical(action_probs)
+        action_logprobs = dist.log_prob(action)
+        dist_entropy = dist.entropy()
+        state_values = self.critic(post_state)
+        
+        return action_logprobs, state_values, dist_entropy
+
+
+class PDPPO:
+    def __init__(self, state_dim, action_dim, lr_actor, lr_critic, gamma, K_epochs, eps_clip, env, has_continuous_action_space, tau, action_std_init=0.6):
+
+        self.has_continuous_action_space = has_continuous_action_space
+
+        if has_continuous_action_space:
+            self.action_std = action_std_init
+        
+        self.tau = tau
+        self.env = env
+        self.gamma = gamma
+        self.eps_clip = eps_clip
+        self.K_epochs = K_epochs
+        
+        self.buffer = RolloutBuffer()
+
+        self.policy = ActorCritic(state_dim, action_dim, has_continuous_action_space, action_std_init).to(device)
+        self.optimizer = torch.optim.Adam([
+                        {'params': self.policy.actor.parameters(), 'lr': lr_actor},
+                        {'params': self.policy.critic.parameters(), 'lr': lr_critic}
+                    ], weight_decay=0.001)
+
+        self.policy_old = ActorCritic(state_dim, action_dim, has_continuous_action_space, action_std_init).to(device)
+        self.policy_old.load_state_dict(self.policy.state_dict())
+        
+        self.MseLoss = nn.MSELoss()
+
+    def set_action_std(self, new_action_std):
+        if self.has_continuous_action_space:
+            self.action_std = new_action_std
+            self.policy.set_action_std(new_action_std)
+            self.policy_old.set_action_std(new_action_std)
+        else:
+            print("--------------------------------------------------------------------------------------------")
+            print("WARNING : Calling PDPPO::set_action_std() on discrete action space policy")
+            print("--------------------------------------------------------------------------------------------")
+
+    def decay_action_std(self, action_std_decay_rate, min_action_std):
+        print("--------------------------------------------------------------------------------------------")
+        if self.has_continuous_action_space:
+            self.action_std = self.action_std - action_std_decay_rate
+            self.action_std = round(self.action_std, 4)
+            if (self.action_std <= min_action_std):
+                self.action_std = min_action_std
+                print("setting actor output action_std to min_action_std : ", self.action_std)
+            else:
+                print("setting actor output action_std to : ", self.action_std)
+            self.set_action_std(self.action_std)
+
+        else:
+            print("WARNING : Calling PDPPO::decay_action_std() on discrete action space policy")
+        print("--------------------------------------------------------------------------------------------")
+    
+    def get_post_state(self, action, machine_setup, inventory_level):
+        setup_loss = np.zeros(self.env.n_machines, dtype=int)
+        setup_costs = np.zeros(self.env.n_machines)
+        # if we are just changing the setup, we use the setup cost matrix with the corresponding position given by the actual setup and the new setup
+        for m in range(self.env.n_machines):   
+            if action[m] != 0: # if the machine is not iddle
+                # 1. IF NEEDED CHANGE SETUP
+                if machine_setup[m] != action[m] and action[m] != 0:
+                    setup_costs[m] = self.env.setup_costs[m][action[m] - 1] 
+                    setup_loss[m] = self.env.setup_loss[m][action[m] - 1]
+                machine_setup[m] = action[m]
+                # 2. PRODUCTION
+                production = self.env.machine_production_matrix[m][action[m] - 1] - setup_loss[m]
+                inventory_level[action[m] - 1] += production
+            else:
+                machine_setup[m] = 0
+        # return the new machine_setup_inventory_level and the setup_cost        
+        return machine_setup, inventory_level, setup_costs 
+    
+    def select_action(self, state,tau):
+
+        if self.has_continuous_action_space:
+            with torch.no_grad():
+                state = torch.FloatTensor(state).to(device)
+                action, action_logprob, state_val = self.policy_old.act(state,tau)
+
+            self.buffer.states.append(state)
+            self.buffer.actions.append(action)
+            self.buffer.logprobs.append(action_logprob)
+            self.buffer.state_values.append(state_val)
+
+            return action.detach().cpu().numpy().flatten()
+        else:
+            with torch.no_grad():
+                state = torch.FloatTensor(state).to(device)
+                action, action_logprob = self.policy_old.act(state,tau)
+            
+            
+            machine_setup, inventory_level, setup_cost = self.get_post_state(action, state[self.env.n_items:self.env.n_items+self.env.n_machines].clone(), state[0:self.env.n_items].clone())
+            
+            post_state = state.clone()
+            post_state[self.env.n_items:self.env.n_items+self.env.n_machines] = machine_setup.clone()
+            post_state[0:self.env.n_items] = inventory_level.clone()
+            post_state = torch.FloatTensor(post_state).to(device)
+            
+            self.buffer.states.append(state)
+            self.buffer.post_states.append(post_state)
+            self.buffer.actions.append(action)
+            self.buffer.logprobs.append(action_logprob)
+            
+            with torch.no_grad():
+                #post_state = torch.cat([post_state.clone(),state.clone()])
+                state_val = self.policy_old.critic(post_state)     
+            
+            self.buffer.state_values.append(state_val)
+
+            return action.numpy()
+    
+    def update(self):
+        # Monte Carlo estimate of returns
+        rewards = []
+        discounted_reward = 0
+        for reward, is_terminal in zip(reversed(self.buffer.rewards), reversed(self.buffer.is_terminals)):
+            if is_terminal:
+                discounted_reward = 0
+            discounted_reward = reward + (self.gamma * discounted_reward)
+            rewards.insert(0, discounted_reward)
+            
+        # Normalizing the rewards
+        rewards = torch.tensor(rewards, dtype=torch.float32).to(device)
+        rewards = (rewards - rewards.mean()) / (rewards.std() + 1e-7)
+
+        # convert list to tensor
+        old_states = torch.squeeze(torch.stack(self.buffer.states, dim=0)).detach().to(device)
+        old_post_states = torch.squeeze(torch.stack(self.buffer.post_states, dim=0)).detach().to(device)
+        old_actions = torch.squeeze(torch.stack(self.buffer.actions, dim=0)).detach().to(device)
+        old_logprobs = torch.squeeze(torch.stack(self.buffer.logprobs, dim=0)).detach().to(device)
+        old_state_values = torch.squeeze(torch.stack(self.buffer.state_values, dim=0)).detach().to(device)
+
+        # calculate advantages
+        advantages = rewards.detach() - old_state_values.detach()
+
+        # Optimize policy for K epochs
+        for _ in range(self.K_epochs):
+
+            # Evaluating old actions and values
+            logprobs, state_values, dist_entropy = self.policy.evaluate(old_states,old_post_states, old_actions,self.tau)
+
+            # match state_values tensor dimensions with rewards tensor
+            state_values = torch.squeeze(state_values)
+            
+            # Finding the ratio (pi_theta / pi_theta__old)
+            ratios = torch.exp(logprobs - old_logprobs.detach())
+
+            # Finding Surrogate Loss  
+            surr1 = ratios * advantages.unsqueeze(1)
+            surr2 = torch.clamp(ratios, 1-self.eps_clip, 1+self.eps_clip) * advantages.unsqueeze(1)
+
+            # final loss of clipped objective PDPPO
+            loss = -torch.min(surr1, surr2) + 0.5 * self.MseLoss(state_values, rewards) - 0.012 * dist_entropy
+            
+            loss_numpy = loss.detach().numpy()
+            
+            # take gradient step
+            self.optimizer.zero_grad()
+            loss.mean().backward()
+            torch.nn.utils.clip_grad_norm_(self.policy.parameters(), max_norm=1)
+            self.optimizer.step()
+            
+        # Copy new weights into old policy
+        
+        self.policy_old.load_state_dict(self.policy.state_dict())
+
+        # clear buffer
+        self.buffer.clear()
+    
+    def save(self, checkpoint_path):
+        torch.save(self.policy_old.state_dict(), checkpoint_path)
+   
+    def load(self, checkpoint_path):
+        self.policy_old.load_state_dict(torch.load(checkpoint_path, map_location=lambda storage, loc: storage))
+        self.policy.load_state_dict(torch.load(checkpoint_path, map_location=lambda storage, loc: storage))

code/Lake application/agents/PDPPO two critics.py

@@ -0,0 +1,345 @@
+# -*- coding: utf-8 -*-
+"""
+Created on Wed Mar  1 00:43:49 2023
+
+@author: leona
+"""
+
+import numpy as np
+import torch
+import torch.nn as nn
+import torch.nn.init as init
+from torch.distributions import MultivariateNormal
+from torch.distributions import Categorical
+
+################################## set device ##################################
+print("============================================================================================")
+# set device to cpu or cuda
+device = torch.device('cpu')
+if(torch.cuda.is_available()): 
+    device = torch.device('cuda:0') 
+    torch.cuda.empty_cache()
+    print("Device set to : " + str(torch.cuda.get_device_name(device)))
+else:
+    print("Device set to : cpu")
+print("============================================================================================")
+
+
+################################## PDPPO Policy ##################################
+class RolloutBuffer:
+    def __init__(self):
+        self.actions = []
+        self.states = []
+        self.post_states = []
+        self.logprobs = []
+        self.rewards = []
+        self.state_values = []
+        self.state_values_post = []
+        self.is_terminals = []
+    
+    def clear(self):
+        del self.actions[:]
+        del self.states[:]
+        del self.post_states[:]
+        del self.logprobs[:]
+        del self.rewards[:]
+        del self.state_values[:]
+        del self.state_values_post[:]
+        del self.is_terminals[:]
+
+
+class ActorCritic(nn.Module):
+    def __init__(self, state_dim, action_dim, has_continuous_action_space, action_std_init):
+        super(ActorCritic, self).__init__()
+
+        self.has_continuous_action_space = has_continuous_action_space
+        
+        if has_continuous_action_space:
+            self.action_dim = action_dim
+            self.action_var = torch.full((action_dim,), action_std_init * action_std_init).to(device)
+        # actor
+        if has_continuous_action_space :
+            self.actor = nn.Sequential(
+                            nn.Linear(state_dim, 64),
+                            nn.Tanh(),
+                            nn.Linear(64, 64),
+                            nn.Tanh(),
+                            nn.Linear(64, action_dim),
+                            nn.Tanh()
+                        )
+        else:
+            
+            self.action_dim = action_dim
+            self.fc1 = nn.Linear(state_dim, 128)
+            self.fc2 = nn.Linear(128, 128)
+            self.actor = nn.Linear(128, self.action_dim.nvec.sum())
+            
+            
+        # critic
+        self.critic = nn.Sequential(
+                        nn.Linear(state_dim, 128),
+                        nn.Tanh(),
+                        nn.Linear(128, 128),
+                        nn.Tanh(),
+                        nn.Linear(128, 1)
+                    )
+        
+        self.critic_post = nn.Sequential(
+                        nn.Linear(state_dim, 128),
+                        nn.Tanh(),
+                        nn.Linear(128, 128),
+                        nn.Tanh(),
+                        nn.Linear(128, 1)
+                    )
+    
+    def forward(self, state):
+        raise NotImplementedError
+    
+    
+    
+    def set_action_std(self, new_action_std):
+        if self.has_continuous_action_space:
+            self.action_var = torch.full((self.action_dim,), new_action_std * new_action_std).to(device)
+        else:
+            print("--------------------------------------------------------------------------------------------")
+            print("WARNING : Calling ActorCritic::set_action_std() on discrete action space policy")
+            print("--------------------------------------------------------------------------------------------")
+
+
+    
+    def act(self, state,tau):
+
+        if self.has_continuous_action_space:
+            action_mean = self.actor(state)
+            cov_mat = torch.diag(self.action_var).unsqueeze(dim=0)
+            dist = MultivariateNormal(action_mean, cov_mat)
+        else:
+            #x = nn.functional.relu(self.fc(state))
+            x = nn.functional.relu(self.fc2(nn.functional.relu(self.fc1(state))))
+            logits = self.actor(x)
+            # x[torch.isnan(x)] = 0
+            action_probs = nn.functional.softmax(logits, dim=-1)
+            #action_probs = torch.nan_to_num(action_probs, nan=1e-6)
+            dist = Categorical(action_probs.view(len(self.action_dim.nvec),-1))
+            # action_probs = self.actor(state)
+            # dist = Categorical(action_probs)
+
+        action = dist.sample()
+        action_logprob = dist.log_prob(action)
+        
+        return action.detach(), action_logprob.detach()
+    
+    def evaluate(self, state,post_state, action,tau):
+
+        if self.has_continuous_action_space:
+            action_mean = self.actor(state)
+            
+            action_var = self.action_var.expand_as(action_mean)
+            cov_mat = torch.diag_embed(action_var).to(device)
+            dist = MultivariateNormal(action_mean, cov_mat)
+            
+            # For Single Action Environments.
+            if self.action_dim == 1:
+                action = action.reshape(-1, self.action_dim)
+        else:
+            #x = nn.functional.relu(self.fc(state))
+            x = nn.functional.relu(self.fc2(nn.functional.relu(self.fc1(state))))
+            # x[torch.isnan(x)] = 0
+            logits = self.actor(x)
+            action_probs = nn.functional.softmax(logits, dim=-1)
+            #action_probs = torch.nan_to_num(action_probs, nan=1e-6)
+            # mask = torch.isnan(action_probs)
+            # if torch.all(mask):
+            #     logits = torch.abs(logits)
+            #     action_probs = nn.functional.softmax(logits, dim=-1)
+                
+            dist = Categorical(action_probs.view(state.shape[0],len(self.action_dim.nvec),-1))
+            # action_probs = self.actor(state)
+            # dist = Categorical(action_probs)
+        action_logprobs = dist.log_prob(action)
+        dist_entropy = dist.entropy()
+        state_values = self.critic(state)
+        state_values_post = self.critic_post(post_state)
+        
+        return action_logprobs, state_values, state_values_post, dist_entropy
+
+
+class PDPPO:
+    def __init__(self, state_dim, action_dim, lr_actor, lr_critic, gamma, K_epochs, eps_clip, env, has_continuous_action_space, tau, action_std_init=0.6):
+
+        self.has_continuous_action_space = has_continuous_action_space
+
+        if has_continuous_action_space:
+            self.action_std = action_std_init
+        
+        self.tau = tau
+        self.env = env
+        self.gamma = gamma
+        self.eps_clip = eps_clip
+        self.K_epochs = K_epochs
+        
+        self.buffer = RolloutBuffer()
+
+        self.policy = ActorCritic(state_dim, action_dim, has_continuous_action_space, action_std_init).to(device)
+        self.optimizer = torch.optim.Adam([
+                        {'params': self.policy.actor.parameters(), 'lr': lr_actor},
+                        {'params': self.policy.critic.parameters(), 'lr': lr_critic},
+                        {'params': self.policy.critic_post.parameters(), 'lr': lr_critic}
+                    ], weight_decay=0.001)
+
+        self.policy_old = ActorCritic(state_dim, action_dim, has_continuous_action_space, action_std_init).to(device)
+        self.policy_old.load_state_dict(self.policy.state_dict())
+        
+        self.MseLoss = nn.MSELoss()
+
+    def set_action_std(self, new_action_std):
+        if self.has_continuous_action_space:
+            self.action_std = new_action_std
+            self.policy.set_action_std(new_action_std)
+            self.policy_old.set_action_std(new_action_std)
+        else:
+            print("--------------------------------------------------------------------------------------------")
+            print("WARNING : Calling PDPPO::set_action_std() on discrete action space policy")
+            print("--------------------------------------------------------------------------------------------")
+
+    def decay_action_std(self, action_std_decay_rate, min_action_std):
+        print("--------------------------------------------------------------------------------------------")
+        if self.has_continuous_action_space:
+            self.action_std = self.action_std - action_std_decay_rate
+            self.action_std = round(self.action_std, 4)
+            if (self.action_std <= min_action_std):
+                self.action_std = min_action_std
+                print("setting actor output action_std to min_action_std : ", self.action_std)
+            else:
+                print("setting actor output action_std to : ", self.action_std)
+            self.set_action_std(self.action_std)
+
+        else:
+            print("WARNING : Calling PDPPO::decay_action_std() on discrete action space policy")
+        print("--------------------------------------------------------------------------------------------")
+    
+    def get_post_state(self, action, machine_setup, inventory_level):
+        setup_loss = np.zeros(self.env.n_machines, dtype=int)
+        setup_costs = np.zeros(self.env.n_machines)
+        # if we are just changing the setup, we use the setup cost matrix with the corresponding position given by the actual setup and the new setup
+        for m in range(self.env.n_machines):   
+            if action[m] != 0: # if the machine is not iddle
+                # 1. IF NEEDED CHANGE SETUP
+                if machine_setup[m] != action[m] and action[m] != 0:
+                    setup_costs[m] = self.env.setup_costs[m][action[m] - 1] 
+                    setup_loss[m] = self.env.setup_loss[m][action[m] - 1]
+                machine_setup[m] = action[m]
+                # 2. PRODUCTION
+                production = self.env.machine_production_matrix[m][action[m] - 1] - setup_loss[m]
+                inventory_level[action[m] - 1] += production
+            else:
+                machine_setup[m] = 0
+        # return the new machine_setup_inventory_level and the setup_cost        
+        return machine_setup, inventory_level, setup_costs 
+    
+    def select_action(self, state,tau):
+
+        if self.has_continuous_action_space:
+            with torch.no_grad():
+                state = torch.FloatTensor(state).to(device)
+                action, action_logprob, state_val = self.policy_old.act(state,tau)
+
+            self.buffer.states.append(state)
+            self.buffer.actions.append(action)
+            self.buffer.logprobs.append(action_logprob)
+            self.buffer.state_values.append(state_val)
+
+            return action.detach().cpu().numpy().flatten()
+        else:
+            with torch.no_grad():
+                state = torch.FloatTensor(state).to(device)
+                action, action_logprob = self.policy_old.act(state,tau)
+            
+            
+            machine_setup, inventory_level, setup_cost = self.get_post_state(action, state[self.env.n_items:self.env.n_items+self.env.n_machines].clone(), state[0:self.env.n_items].clone())
+            
+            post_state = state.clone()
+            post_state[self.env.n_items:self.env.n_items+self.env.n_machines] = machine_setup.clone()
+            post_state[0:self.env.n_items] = inventory_level.clone()
+            post_state = torch.FloatTensor(post_state).to(device)
+            
+            self.buffer.states.append(state)
+            self.buffer.post_states.append(post_state)
+            self.buffer.actions.append(action)
+            self.buffer.logprobs.append(action_logprob)
+            
+            with torch.no_grad():
+                #post_state = torch.cat([post_state.clone(),state.clone()])
+                state_val = self.policy_old.critic(state)
+                state_val_post = self.policy_old.critic_post(post_state)            
+            
+            self.buffer.state_values.append(state_val)
+            self.buffer.state_values_post.append(state_val_post)
+
+            return action.numpy()
+    
+    def update(self):
+        # Monte Carlo estimate of returns
+        rewards = []
+        discounted_reward = 0
+        for reward, is_terminal in zip(reversed(self.buffer.rewards), reversed(self.buffer.is_terminals)):
+            if is_terminal:
+                discounted_reward = 0
+            discounted_reward = reward + (self.gamma * discounted_reward)
+            rewards.insert(0, discounted_reward)
+            
+        # Normalizing the rewards
+        rewards = torch.tensor(rewards, dtype=torch.float32).to(device)
+        rewards = (rewards - rewards.mean()) / (rewards.std() + 1e-7)
+
+        # convert list to tensor
+        old_states = torch.squeeze(torch.stack(self.buffer.states, dim=0)).detach().to(device)
+        old_post_states = torch.squeeze(torch.stack(self.buffer.post_states, dim=0)).detach().to(device)
+        old_actions = torch.squeeze(torch.stack(self.buffer.actions, dim=0)).detach().to(device)
+        old_logprobs = torch.squeeze(torch.stack(self.buffer.logprobs, dim=0)).detach().to(device)
+        old_state_values = torch.squeeze(torch.stack(self.buffer.state_values, dim=0)).detach().to(device)
+        old_state_values_post = torch.squeeze(torch.stack(self.buffer.state_values_post, dim=0)).detach().to(device)
+
+        # calculate advantages
+        advantages = rewards.detach() - torch.min(old_state_values.detach(), old_state_values_post.detach()).detach()
+
+        # Optimize policy for K epochs
+        for _ in range(self.K_epochs):
+
+            # Evaluating old actions and values
+            logprobs, state_values, state_values_post, dist_entropy = self.policy.evaluate(old_states,old_post_states, old_actions,self.tau)
+
+            # match state_values tensor dimensions with rewards tensor
+            state_values = torch.squeeze(state_values)
+            
+            # Finding the ratio (pi_theta / pi_theta__old)
+            ratios = torch.exp(logprobs - old_logprobs.detach())
+
+            # Finding Surrogate Loss  
+            surr1 = ratios * advantages.unsqueeze(1)
+            surr2 = torch.clamp(ratios, 1-self.eps_clip, 1+self.eps_clip) * advantages.unsqueeze(1)
+
+            # final loss of clipped objective PDPPO
+            loss = -torch.min(surr1, surr2) + 0.5 * self.MseLoss(torch.min(state_values,state_values_post.squeeze()), rewards) - 0.012 * dist_entropy
+            
+            loss_numpy = loss.detach().numpy()
+            
+            # take gradient step
+            self.optimizer.zero_grad()
+            loss.mean().backward()
+            torch.nn.utils.clip_grad_norm_(self.policy.parameters(), max_norm=1)
+            self.optimizer.step()
+            
+        # Copy new weights into old policy
+        
+        self.policy_old.load_state_dict(self.policy.state_dict())
+
+        # clear buffer
+        self.buffer.clear()
+    
+    def save(self, checkpoint_path):
+        torch.save(self.policy_old.state_dict(), checkpoint_path)
+   
+    def load(self, checkpoint_path):
+        self.policy_old.load_state_dict(torch.load(checkpoint_path, map_location=lambda storage, loc: storage))
+        self.policy.load_state_dict(torch.load(checkpoint_path, map_location=lambda storage, loc: storage))

code/Lake application/agents/PDPPO.py

@@ -0,0 +1,301 @@
+# -*- coding: utf-8 -*-
+"""
+Created on Wed Mar  1 00:43:49 2023
+
+@author: leona
+"""
+
+import numpy as np
+import torch
+import torch.nn as nn
+import torch.nn.init as init
+from torch.distributions import MultivariateNormal
+from torch.distributions import Categorical
+
+################################## set device ##################################
+print("============================================================================================")
+# set device to cpu or cuda
+device = torch.device('cpu')
+if(torch.cuda.is_available()): 
+    device = torch.device('cuda:0') 
+    torch.cuda.empty_cache()
+    print("Device set to : " + str(torch.cuda.get_device_name(device)))
+else:
+    print("Device set to : cpu")
+print("============================================================================================")
+
+
+################################## PDPPO Policy ##################################
+class RolloutBuffer:
+    def __init__(self):
+        self.actions = []
+        self.states = []
+        self.post_states = []
+        self.logprobs = []
+        self.rewards = []
+        self.state_values = []
+        self.state_values_post = []
+        self.is_terminals = []
+    
+    def clear(self):
+        del self.actions[:]
+        del self.states[:]
+        del self.post_states[:]
+        del self.logprobs[:]
+        del self.rewards[:]
+        del self.state_values[:]
+        del self.state_values_post[:]
+        del self.is_terminals[:]
+
+
+class ActorCritic(nn.Module):
+    def __init__(self, state_dim, action_dim, has_continuous_action_space, action_std_init):
+        super(ActorCritic, self).__init__()
+
+        self.has_continuous_action_space = has_continuous_action_space
+        
+        if has_continuous_action_space:
+            self.action_dim = action_dim
+            self.action_var = torch.full((action_dim,), action_std_init * action_std_init).to(device)
+        # actor
+        if has_continuous_action_space :
+            self.actor = nn.Sequential(
+                            nn.Linear(state_dim, 64),
+                            nn.Tanh(),
+                            nn.Linear(64, 64),
+                            nn.Tanh(),
+                            nn.Linear(64, action_dim),
+                            nn.Tanh()
+                        )
+        else:
+            
+            self.actor = nn.Sequential(
+                            nn.Linear(state_dim, 128),
+                            nn.Tanh(),
+                            nn.Linear(128, 128),
+                            nn.Tanh(),
+                            nn.Linear(128, action_dim)
+                        )
+            
+            
+        # critic
+        self.critic = nn.Sequential(
+                        nn.Linear(state_dim, 128),
+                        nn.Tanh(),
+                        nn.Linear(128, 128),
+                        nn.Tanh(),
+                        nn.Linear(128, 1)
+                    )
+        
+        self.critic_post = nn.Sequential(
+                        nn.Linear(state_dim, 128),
+                        nn.Tanh(),
+                        nn.Linear(128, 128),
+                        nn.Tanh(),
+                        nn.Linear(128, 1)
+                    )
+    
+    def forward(self, state):
+        raise NotImplementedError
+    
+    
+    
+    def set_action_std(self, new_action_std):
+        if self.has_continuous_action_space:
+            self.action_var = torch.full((self.action_dim,), new_action_std * new_action_std).to(device)
+        else:
+            print("--------------------------------------------------------------------------------------------")
+            print("WARNING : Calling ActorCritic::set_action_std() on discrete action space policy")
+            print("--------------------------------------------------------------------------------------------")
+
+
+    
+    def act(self, state,tau):
+
+        if self.has_continuous_action_space:
+            action_mean = self.actor(state)
+            cov_mat = torch.diag(self.action_var).unsqueeze(dim=0)
+            dist = MultivariateNormal(action_mean, cov_mat)
+        else:
+            #x = nn.functional.relu(self.fc2(nn.functional.relu(self.fc1(state))))
+            logits = self.actor(state)
+            action_probs = nn.functional.softmax(logits, dim=-1)  
+            dist = Categorical(action_probs)
+
+        action = dist.sample()
+        action_logprob = dist.log_prob(action)
+        
+        return action.detach(), action_logprob.detach()
+    
+    def evaluate(self, state,post_state, action,tau):
+
+        #x = nn.functional.relu(self.fc2(nn.functional.relu(self.fc1(state))))
+        logits = self.actor(state)
+        action_probs = nn.functional.softmax(logits, dim=-1)  
+        dist = Categorical(action_probs)
+
+        action_logprobs = dist.log_prob(action.T).T
+        dist_entropy = dist.entropy()
+        state_values = self.critic(state)
+        state_values_post = self.critic_post(post_state)
+        
+        return action_logprobs, state_values, state_values_post, dist_entropy
+
+
+class PDPPO:
+    def __init__(self, state_dim, action_dim, lr_actor, lr_critic, gamma, K_epochs, eps_clip, env, has_continuous_action_space, tau, action_std_init=0.6):
+
+        self.has_continuous_action_space = has_continuous_action_space
+
+        if has_continuous_action_space:
+            self.action_std = action_std_init
+        
+        self.tau = tau
+        self.env = env
+        self.gamma = gamma
+        self.eps_clip = eps_clip
+        self.K_epochs = K_epochs
+        
+        self.buffer = RolloutBuffer()
+
+        self.policy = ActorCritic(state_dim, action_dim, has_continuous_action_space, action_std_init).to(device)
+        self.optimizer = torch.optim.Adam([
+                        {'params': self.policy.actor.parameters(), 'lr': lr_actor},
+                        {'params': self.policy.critic.parameters(), 'lr': lr_critic},
+                        {'params': self.policy.critic_post.parameters(), 'lr': lr_critic}
+                    ], weight_decay=0.0002) #, weight_decay=0.001
+
+        self.policy_old = ActorCritic(state_dim, action_dim, has_continuous_action_space, action_std_init).to(device)
+        self.policy_old.load_state_dict(self.policy.state_dict())
+        
+        self.MseLoss = nn.MSELoss()
+
+    def set_action_std(self, new_action_std):
+        if self.has_continuous_action_space:
+            self.action_std = new_action_std
+            self.policy.set_action_std(new_action_std)
+            self.policy_old.set_action_std(new_action_std)
+        else:
+            print("--------------------------------------------------------------------------------------------")
+            print("WARNING : Calling PDPPO::set_action_std() on discrete action space policy")
+            print("--------------------------------------------------------------------------------------------")
+
+    def decay_action_std(self, action_std_decay_rate, min_action_std):
+        print("--------------------------------------------------------------------------------------------")
+        if self.has_continuous_action_space:
+            self.action_std = self.action_std - action_std_decay_rate
+            self.action_std = round(self.action_std, 4)
+            if (self.action_std <= min_action_std):
+                self.action_std = min_action_std
+                print("setting actor output action_std to min_action_std : ", self.action_std)
+            else:
+                print("setting actor output action_std to : ", self.action_std)
+            self.set_action_std(self.action_std)
+
+        else:
+            print("WARNING : Calling PDPPO::decay_action_std() on discrete action space policy")
+        print("--------------------------------------------------------------------------------------------")
+    
+    def select_action(self, state, tau):
+
+        state_int = state.copy()
+
+        with torch.no_grad():
+            state = torch.tensor(state).to(device)
+            state = state.float() 
+            state = torch.unsqueeze(state, 1).T
+            action, action_logprob = self.policy_old.act(state,tau)
+        
+        post_state = self.env.get_post_decision_state(np.argmax(state_int),action.clone())
+
+        binary_array = np.zeros(state.shape[1], dtype=int)
+        binary_array[post_state] = 1
+        
+
+        post_state = torch.tensor(binary_array).to(device)
+        post_state = post_state.float() 
+        post_state = torch.unsqueeze(post_state, 1).T
+        
+
+        self.buffer.states.append(state)
+        self.buffer.post_states.append(post_state)
+        self.buffer.actions.append(action)
+        self.buffer.logprobs.append(action_logprob)
+        
+        with torch.no_grad():
+            state_val = self.policy_old.critic(state)
+            state_val_post = self.policy_old.critic(post_state)            
+        
+        self.buffer.state_values.append(state_val)
+        self.buffer.state_values_post.append(state_val_post)
+
+        return action.cpu().numpy()
+    
+    def update(self):
+        # Monte Carlo estimate of returns
+        rewards = []
+        discounted_reward = 0
+        for reward, is_terminal in zip(reversed(self.buffer.rewards), reversed(self.buffer.is_terminals)):
+            if is_terminal:
+                discounted_reward = 0
+            discounted_reward = reward + (self.gamma * discounted_reward)
+            rewards.insert(0, discounted_reward)
+            
+        # Normalizing the rewards
+        rewards = torch.tensor(rewards, dtype=torch.float32).to(device)
+        rewards = (rewards - rewards.mean()) / (rewards.std() + 1e-7)
+
+        # convert list to tensor
+        
+        old_states = torch.squeeze(torch.stack(self.buffer.states, dim=1)).detach().to(device)
+        old_post_states = torch.squeeze(torch.stack(self.buffer.post_states, dim=0).detach().to(device),1)
+        old_actions = torch.stack(self.buffer.actions, dim=0).detach().to(device)
+        old_logprobs = torch.stack(self.buffer.logprobs, dim=0).detach().to(device)
+        old_state_values = torch.squeeze(torch.stack(self.buffer.state_values, dim=0)).detach().to(device)
+        old_state_values_post = torch.squeeze(torch.stack(self.buffer.state_values_post, dim=0)).detach().to(device)
+
+        # calculate advantages
+        advantages = rewards.detach() - torch.min(old_state_values.detach(), old_state_values_post.detach()).detach()
+
+        # Optimize policy for K epochs
+        for _ in range(self.K_epochs):
+
+            # Evaluating old actions and values
+            logprobs, state_values, state_values_post, dist_entropy = self.policy.evaluate(old_states,old_post_states, old_actions,self.tau)
+
+            # match state_values tensor dimensions with rewards tensor
+            state_values = torch.squeeze(state_values)
+            
+            state_values_post = torch.squeeze(state_values_post)
+
+            # Finding the ratio (pi_theta / pi_theta__old)
+            ratios = torch.exp(logprobs - old_logprobs.detach())
+
+            # Finding Surrogate Loss  
+            surr1 = ratios * advantages.unsqueeze(1)
+            surr2 = torch.clamp(ratios, 1-self.eps_clip, 1+self.eps_clip) * advantages.unsqueeze(1)
+
+            # final loss of clipped objective PDPPO
+            loss = -torch.min(surr1, surr2) + 0.5 * self.MseLoss(torch.min(state_values,state_values_post), rewards) - 0.012 * dist_entropy
+            
+            loss_numpy = loss.detach().cpu().numpy()
+            
+            # take gradient step
+            self.optimizer.zero_grad()
+            loss.mean().backward()
+            torch.nn.utils.clip_grad_norm_(self.policy.parameters(), max_norm=1)
+            self.optimizer.step()
+            
+        # Copy new weights into old policy
+        
+        self.policy_old.load_state_dict(self.policy.state_dict())
+
+        # clear buffer
+        self.buffer.clear()
+    
+    def save(self, checkpoint_path):
+        torch.save(self.policy_old.state_dict(), checkpoint_path)
+   
+    def load(self, checkpoint_path):
+        self.policy_old.load_state_dict(torch.load(checkpoint_path, map_location=lambda storage, loc: storage))
+        self.policy.load_state_dict(torch.load(checkpoint_path, map_location=lambda storage, loc: storage))

code/Lake application/agents/PDPPOAgent two critics.py

@@ -0,0 +1,394 @@
+import os # Provides a way of interacting with the file system
+import sys
+import glob # Helps find all the pathnames matching a specified pattern according to the rules used by the Unix shell
+import time # Provides various time-related functions
+from datetime import datetime # Module that supplies classes for working with dates and times
+
+import numpy as np # A library for the Python programming language, adding support for large, multi-dimensional arrays and matrices
+import gym # Provides a collection of test problems — environments — that you can use to work out your reinforcement learning algorithms
+import torch # A machine learning framework that provides tensor computation (like NumPy) with strong acceleration on GPUs
+import copy # Provides a module for shallow and deep copying operations
+import matplotlib.pyplot as plt # A plotting library for the Python programming language and its numerical mathematics extension NumPy
+import matplotlib.patches as mpatches # Provides a way of adding a colored patch to the plot, for example to create a legend
+BASE_DIR = os.path.dirname(os.path.abspath('__file__'))
+AGENTS_DIR = os.path.join(BASE_DIR,'agents')
+sys.path.append(AGENTS_DIR)
+from PDPPO import PDPPO
+from envs import *
+import copy
+
+
+class SimplePlantSB(SimplePlant):
+    def __init__(self, settings, stoch_model):
+        super().__init__(settings, stoch_model)
+        try:self.dict_obs = settings['dict_obs']
+        except:self.dict_obs = False
+        self.last_inventory = copy.copy(self.inventory_level)
+        self.action_space = gym.spaces.MultiDiscrete(
+            [self.n_items+1] * self.n_machines
+        )
+        
+        if self.dict_obs:
+            self.observation_space = gym.spaces.Dict({
+                'inventory_level': gym.spaces.Box(low = np.zeros(self.n_items),high = np.ones(self.n_items)*(settings['max_inventory_level'][0]+1)*self.n_items),
+                'machine_setup': gym.spaces.MultiDiscrete([self.n_items+1] * self.n_machines)
+                #'last_inventory_level':gym.spaces.Box(low = np.zeros(self.n_items),high = np.ones(self.n_items)*(settings['max_inventory_level'][0]+1)*self.n_items)
+            })
+        else:
+            self.observation_space = gym.spaces.Box(
+                low=np.zeros(self.n_items+self.n_machines),# high for the inventory level
+                high=np.concatenate(
+                    [
+                        np.array(self.max_inventory_level),
+                        np.ones(self.n_machines) * (self.n_items+1), #high for the machine setups 
+                        #np.array(self.max_inventory_level) # high for the inventory level
+                    ]),
+                dtype=np.int32
+            )
+
+    def step(self, action):
+        """
+        Step method: Execute one time step within the environment
+
+        Parameters
+        ----------
+        action : action given by the agent
+
+        Returns
+        -------
+        obs : Observation of the state give the method _next_observation
+        reward : Cost given by the _reward method
+        done : returns True or False given by the _done method
+        dict : possible information for control to environment monitoring
+
+        """
+        self.last_inventory = copy.copy(self.inventory_level)
+            
+        self.total_cost = self._take_action(action, self.machine_setup, self.inventory_level, self.demand)
+        
+        # self.total_cost['setup_costs'] = 0
+        # self.total_cost['holding_costs'] = 0
+        
+        reward = -sum([ele for key, ele in self.total_cost.items()])
+        #reward = -self.total_cost['lost_sales']
+        
+        #reward = np.abs(action)
+        
+        self.current_step += 1
+        done = self.current_step == self.T
+        obs = self._next_observation()
+
+        return obs, reward, done, self.total_cost
+
+    def _next_observation(self):
+        """
+        Returns the next demand
+        """
+        obs = SimplePlant._next_observation(self)
+        #obs['last_inventory_level'] = copy.copy(self.last_inventory)
+        if isinstance(obs, dict):    
+            if not self.dict_obs:
+                obs = np.concatenate(
+                    (
+                        obs['inventory_level'], # n_items size
+                        obs['machine_setup'], # n_machine size
+                        #obs['last_inventory_level']# n_items size
+                    )
+                )
+        else:
+            if self.dict_obs:
+                raise('Change dict_obst to False')
+        return obs
+
+
+class PDPPOAgent():
+    def __init__(self, env: SimplePlant, settings: dict):
+        self.env = SimplePlantSB(env.settings, env.stoch_model)
+        self.last_inventory = env.inventory_level
+        self.model_name = settings['model_name']
+        self.experiment_name = settings['experiment_name']
+        self.parallelization = settings['parallelization']
+        try:self.dict_obs = settings['dict_obs']
+        except:self.dict_obs = False
+        
+        self.POSSIBLE_STATES = self.env.n_items + 1
+        self.env.cost_to_reward = True
+        self.epsilon = 0
+
+        BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))    
+        # Use the logs file in the root path of the main.
+        self.LOG_DIR = os.path.join(BASE_DIR,'logs')
+        
+        print("============================================================================================")
+    
+        ####### initialize environment hyperparameters ######
+   
+        self.has_continuous_action_space = False  # continuous action space; else discrete
+    
+        self.max_ep_len = 1000                  # max timesteps in one episode
+        self.tau = 1
+        self.tau_start = 1.0  # initial value of tau
+        self.tau_end = 2.0  # final value of tau
+        
+        self.print_freq = self.max_ep_len * 4        # print avg reward in the interval (in num timesteps)
+        self.log_freq = self.max_ep_len * 4           # log avg reward in the interval (in num timesteps)
+        self.save_model_freq = int(4999)          # save model frequency (in num timesteps)
+    
+        self.action_std = 0.6                    # starting std for action distribution (Multivariate Normal)
+        self.action_std_decay_rate = 0.05        # linearly decay self.action_std (self.action_std = self.action_std - self.action_std_decay_rate)
+        self.min_action_std = 0.1                # minimum self.action_std (stop decay after self.action_std <= min_self.action_std)
+        self.action_std_decay_freq = int(2.5e5)  # self.action_std decay frequency (in num timesteps)
+        #####################################################
+    
+        ## Note : print/log frequencies should be > than self.max_ep_len
+    
+        ################ PDPPO hyperparameters ################
+        self.update_timestep = self.max_ep_len * 4      # update policy every n timesteps
+        self.K_epochs = 60               # update policy for K epochs in one PDPPO update
+    
+        self.eps_clip = 0.2          # clip parameter for PDPPO
+        self.gamma = 0.99           # discount factor
+    
+        self.lr_actor = 0.00055      # learning rate for actor network
+        self.lr_critic = 0.001       # learning rate for critic network
+    
+        self.random_seed = 0         # set random seed if required (0 = no random seed)
+        #####################################################
+        self.run_num_pretrained = 0      #### change this to prevent overwriting weights in same self.experiment_name folder
+        
+        print("training environment name : " + self.experiment_name + '_PDPPO')
+        
+        
+    
+        # state space dimension
+        self.state_dim = self.env.observation_space.shape[0]
+    
+        # action space dimension
+        if self.has_continuous_action_space:
+            self.action_dim = self.env.action_space.shape[0]
+        else:
+            self.action_dim = self.env.action_space
+
+        self.pdppo_agent = PDPPO(self.state_dim, self.action_dim, self.lr_actor, self.lr_critic, self.gamma, self.K_epochs, self.eps_clip, copy.copy(self.env), self.has_continuous_action_space,self.tau, self.action_std)
+        
+       
+    ################################### Training ###################################
+    def learn(self,n_episodes = 100000):
+
+    
+        ###################### logging ######################
+        
+        self.max_training_timesteps = n_episodes   # break training loop if timeteps > self.max_training_timesteps
+        
+        env = self.env
+        
+        #### log files for multiple runs are NOT overwritten
+        log_dir = self.LOG_DIR
+        if not os.path.exists(log_dir):
+              os.makedirs(log_dir)
+    
+        log_dir = log_dir + '/' + self.experiment_name + '_PDPPO/'
+        if not os.path.exists(log_dir):
+              os.makedirs(log_dir)
+    
+        #### get number of log files in log directory
+        run_num = 0
+        current_num_files = next(os.walk(log_dir))[2]
+        run_num = len(current_num_files)
+    
+        #### create new log file for each run
+        log_f_name = log_dir + '/PDPPO_' + self.experiment_name + "_log_" + str(run_num) + ".csv"
+    
+        print("current logging run number for " + self.experiment_name + " : ", run_num)
+        print("logging at : " + log_f_name)
+        #####################################################
+    
+        ################### checkpointing ###################
+        
+    
+        directory = self.LOG_DIR
+        if not os.path.exists(directory):
+              os.makedirs(directory)
+    
+        directory = directory + '/' + self.experiment_name + '_PDPPO' + '/'
+        if not os.path.exists(directory):
+              os.makedirs(directory)
+    
+        
+        checkpoint_path = directory + "PDPPO_{}_{}_{}.pth".format(self.experiment_name, self.random_seed, self.run_num_pretrained)
+        print("save checkpoint path : " + checkpoint_path)
+        #####################################################
+    
+    
+        ############# print all hyperparameters #############
+        print("--------------------------------------------------------------------------------------------")
+        print("max training timesteps : ", self.max_training_timesteps)
+        print("max timesteps per episode : ", self.max_ep_len)
+        print("model saving frequency : " + str(self.save_model_freq) + " timesteps")
+        print("log frequency : " + str(self.log_freq) + " timesteps")
+        print("printing average reward over episodes in last : " + str(self.print_freq) + " timesteps")
+        print("--------------------------------------------------------------------------------------------")
+        print("state space dimension : ", self.state_dim)
+        print("action space dimension : ", self.action_dim)
+        print("--------------------------------------------------------------------------------------------")
+        if self.has_continuous_action_space:
+            print("Initializing a continuous action space policy")
+            print("--------------------------------------------------------------------------------------------")
+            print("starting std of action distribution : ", self.action_std)
+            print("decay rate of std of action distribution : ", self.action_std_decay_rate)
+            print("minimum std of action distribution : ", min_self.action_std)
+            print("decay frequency of std of action distribution : " + str(self.action_std_decay_freq) + " timesteps")
+        else:
+            print("Initializing a discrete action space policy")
+        print("--------------------------------------------------------------------------------------------")
+        print("PDPPO update frequency : " + str(self.update_timestep) + " timesteps")
+        print("PDPPO K epochs : ", self.K_epochs)
+        print("PDPPO epsilon clip : ", self.eps_clip)
+        print("discount factor (self.gamma) : ", self.gamma)
+        print("--------------------------------------------------------------------------------------------")
+        print("optimizer learning rate actor : ", self.lr_actor)
+        print("optimizer learning rate critic : ", self.lr_critic)
+        if self.random_seed:
+            print("--------------------------------------------------------------------------------------------")
+            print("setting random seed to ", self.random_seed)
+
+        #####################################################
+    
+        print("============================================================================================")
+    
+        ################# training procedure ################
+    
+        # initialize a PDPPO agent
+        self.PDPPO_agent = PDPPO(self.state_dim, self.action_dim, self.lr_actor, self.lr_critic, self.gamma, self.K_epochs, self.eps_clip, copy.copy(self.env), self.has_continuous_action_space, self.action_std)
+    
+        # track total training time
+        start_time = datetime.now().replace(microsecond=0)
+        print("Started training at (GMT) : ", start_time)
+    
+        print("============================================================================================")
+    
+        # logging file
+        log_f = open(log_f_name,"w+")
+        log_f.write('episode,timestep,reward\n')
+    
+        # printing and logging variables
+        print_running_reward = 0
+        print_running_episodes = 0
+    
+        log_running_reward = 0
+        log_running_episodes = 0
+    
+        time_step = 0
+        i_episode = 0
+        
+        annealing_steps = self.max_training_timesteps  # total number of training steps
+        
+        # training loop
+        while time_step <= self.max_training_timesteps:
+            
+            anneal_rate = (self.tau_end - self.tau_start) / annealing_steps  # rate of tau increase per step
+            
+            self.tau = max(self.tau_end, self.tau_start + anneal_rate * time_step)
+            
+            state = env.reset()
+            current_ep_reward = 0
+    
+            for t in range(1, self.max_ep_len+1):
+    
+                # select action with policy
+                action = self.pdppo_agent.select_action(state,self.tau)
+                state, reward, done, _ = env.step(action)
+    
+                # saving reward and is_terminals
+                self.pdppo_agent.buffer.rewards.append(reward)
+                self.pdppo_agent.buffer.is_terminals.append(done)
+    
+                time_step +=1
+                current_ep_reward += reward
+    
+                # update PDPPO agent
+                if time_step % self.update_timestep == 0:
+                    self.pdppo_agent.update()
+    
+                # if continuous action space; then decay action std of ouput action distribution
+                if self.has_continuous_action_space and time_step % self.action_std_decay_freq == 0:
+                    self.pdppo_agent.decay_self.action_std(self.action_std_decay_rate, self.action_std)
+    
+                # log in logging file
+                if time_step % self.log_freq == 0:
+    
+                    # log average reward till last episode
+                    log_avg_reward = log_running_reward / log_running_episodes
+                    log_avg_reward = round(log_avg_reward, 4)
+    
+                    log_f.write('{},{},{}\n'.format(i_episode, time_step, log_avg_reward))
+                    log_f.flush()
+    
+                    log_running_reward = 0
+                    log_running_episodes = 0
+    
+                # printing average reward
+                if time_step % self.print_freq == 0:
+    
+                    # print average reward till last episode
+                    print_avg_reward = print_running_reward / print_running_episodes
+                    print_avg_reward = round(print_avg_reward, 2)
+    
+                    print("Episode : {} \t\t Timestep : {} \t\t Average Reward : {}".format(i_episode, time_step, print_avg_reward))
+    
+                    print_running_reward = 0
+                    print_running_episodes = 0
+    
+                # save model weights
+                if time_step % self.save_model_freq == 0:
+                    print("--------------------------------------------------------------------------------------------")
+                    #print("saving model at : " + checkpoint_path)
+                    self.pdppo_agent.save(checkpoint_path)
+                    #print("model saved")
+                    print("Elapsed Time  : ", datetime.now().replace(microsecond=0) - start_time)
+                    print("--------------------------------------------------------------------------------------------")
+    
+                # break; if the episode is over
+                if done:
+                    break
+    
+            print_running_reward += current_ep_reward
+            print_running_episodes += 1
+    
+            log_running_reward += current_ep_reward
+            log_running_episodes += 1
+    
+            i_episode += 1
+    
+        log_f.close()
+        #env.close()
+    
+        # print total training time
+        print("============================================================================================")
+        end_time = datetime.now().replace(microsecond=0)
+        print("Started training at (GMT) : ", start_time)
+        print("Finished training at (GMT) : ", end_time)
+        print("Total training time  : ", end_time - start_time)
+        print("============================================================================================")
+
+    def get_action(self,state):
+        if isinstance(state, dict):    
+            if not self.dict_obs:
+                state = np.concatenate(
+                    (
+                        state['inventory_level'], # n_items size
+                        state['machine_setup'], # n_machine size
+                    )
+                )
+        else:
+            if self.dict_obs:
+                raise('Change dict_obst to False')
+        return self.pdppo_agent.select_action(state,self.tau)
+    
+    def load_agent(self,path):
+        #directory = "PDPPO_preTrained" + '/' + env_name + '/'
+        directory = self.LOG_DIR
+        directory = directory + '/' + self.experiment_name + '_PDPPO' + '/'
+        checkpoint_path = directory + "PDPPO_{}_{}_{}.pth".format(self.experiment_name, self.random_seed, self.run_num_pretrained)
+        print("loading network from : " + checkpoint_path)
+        self.pdppo_agent.load(checkpoint_path)

code/Lake application/agents/PDPPOAgent.py

@@ -0,0 +1,402 @@
+import os # Provides a way of interacting with the file system
+import sys
+import glob # Helps find all the pathnames matching a specified pattern according to the rules used by the Unix shell
+import time # Provides various time-related functions
+from datetime import datetime # Module that supplies classes for working with dates and times
+
+import numpy as np # A library for the Python programming language, adding support for large, multi-dimensional arrays and matrices
+import gym # Provides a collection of test problems — environments — that you can use to work out your reinforcement learning algorithms
+import torch # A machine learning framework that provides tensor computation (like NumPy) with strong acceleration on GPUs
+import copy # Provides a module for shallow and deep copying operations
+import matplotlib.pyplot as plt # A plotting library for the Python programming language and its numerical mathematics extension NumPy
+import matplotlib.patches as mpatches # Provides a way of adding a colored patch to the plot, for example to create a legend
+BASE_DIR = os.path.dirname(os.path.abspath('__file__'))
+AGENTS_DIR = os.path.join(BASE_DIR,'agents')
+sys.path.append(AGENTS_DIR)
+from agents.PDPPO import PDPPO
+from envs import *
+import copy
+
+
+class SimplePlantSB(SimplePlant):
+    def __init__(self, settings, stoch_model):
+        super().__init__(settings, stoch_model)
+        try:self.dict_obs = settings['dict_obs']
+        except:self.dict_obs = False
+        self.last_inventory = copy.copy(self.inventory_level)
+        self.action_space = gym.spaces.MultiDiscrete(
+            [self.n_items+1] * self.n_machines
+        )
+        
+        if self.dict_obs:
+            self.observation_space = gym.spaces.Dict({
+                'inventory_level': gym.spaces.Box(low = np.zeros(self.n_items),high = np.ones(self.n_items)*(settings['max_inventory_level'][0]+1)*self.n_items),
+                'machine_setup': gym.spaces.MultiDiscrete([self.n_items+1] * self.n_machines)
+                #'last_inventory_level':gym.spaces.Box(low = np.zeros(self.n_items),high = np.ones(self.n_items)*(settings['max_inventory_level'][0]+1)*self.n_items)
+            })
+        else:
+            self.observation_space = gym.spaces.Box(
+                low=np.zeros(self.n_items+self.n_machines),# high for the inventory level
+                high=np.concatenate(
+                    [
+                        np.array(self.max_inventory_level),
+                        np.ones(self.n_machines) * (self.n_items+1), #high for the machine setups 
+                        #np.array(self.max_inventory_level) # high for the inventory level
+                    ]),
+                dtype=np.int32
+            )
+
+    def step(self, action):
+        """
+        Step method: Execute one time step within the environment
+
+        Parameters
+        ----------
+        action : action given by the agent
+
+        Returns
+        -------
+        obs : Observation of the state give the method _next_observation
+        reward : Cost given by the _reward method
+        done : returns True or False given by the _done method
+        dict : possible information for control to environment monitoring
+
+        """
+        self.last_inventory = copy.copy(self.inventory_level)
+            
+        self.total_cost = self._take_action(action, self.machine_setup, self.inventory_level, self.demand)
+        
+        # self.total_cost['setup_costs'] = 0
+        # self.total_cost['holding_costs'] = 0
+        
+        reward = -sum([ele for key, ele in self.total_cost.items()])
+        #reward = -self.total_cost['lost_sales']
+        
+        #reward = np.abs(action)
+        
+        self.current_step += 1
+        done = self.current_step == self.T
+        obs = self._next_observation()
+
+        return obs, reward, done, self.total_cost
+
+    def _next_observation(self):
+        """
+        Returns the next demand
+        """
+        obs = SimplePlant._next_observation(self)
+        #obs['last_inventory_level'] = copy.copy(self.last_inventory)
+        if isinstance(obs, dict):    
+            if not self.dict_obs:
+                obs = np.concatenate(
+                    (
+                        obs['inventory_level'], # n_items size
+                        obs['machine_setup'], # n_machine size
+                        #obs['last_inventory_level']# n_items size
+                    )
+                )
+        else:
+            if self.dict_obs:
+                raise('Change dict_obst to False')
+        return obs
+
+
+class PDPPOAgent():
+    def __init__(self, env: SimplePlant, settings: dict):
+        self.env = env
+
+        self.model_name = settings['model_name']
+        self.experiment_name = settings['experiment_name']
+        self.parallelization = settings['parallelization']
+        try:self.dict_obs = settings['dict_obs']
+        except:self.dict_obs = False
+        
+        self.POSSIBLE_STATES = self.env.observation_space.n
+        self.env.cost_to_reward = True
+        self.epsilon = 0
+
+        BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))    
+        # Use the logs file in the root path of the main.
+        self.LOG_DIR = os.path.join(BASE_DIR,'logs')
+        
+        print("============================================================================================")
+    
+        ####### initialize environment hyperparameters ######
+   
+        self.has_continuous_action_space = False  # continuous action space; else discrete
+    
+        self.max_ep_len = 100                  # max timesteps in one episode
+        self.tau = 1
+        self.tau_start = 1.0  # initial value of tau
+        self.tau_end = 2.0  # final value of tau
+        
+        self.print_freq = self.max_ep_len * 4        # print avg reward in the interval (in num timesteps)
+        self.log_freq = self.max_ep_len * 4           # log avg reward in the interval (in num timesteps)
+        self.save_model_freq = int(4999)          # save model frequency (in num timesteps)
+    
+        self.action_std = 0.6                    # starting std for action distribution (Multivariate Normal)
+        self.action_std_decay_rate = 0.05        # linearly decay self.action_std (self.action_std = self.action_std - self.action_std_decay_rate)
+        self.min_action_std = 0.1                # minimum self.action_std (stop decay after self.action_std <= min_self.action_std)
+        self.action_std_decay_freq = int(2.5e5)  # self.action_std decay frequency (in num timesteps)
+        #####################################################
+    
+        ## Note : print/log frequencies should be > than self.max_ep_len
+    
+        ################ PDPPO hyperparameters ################
+        self.update_timestep = self.max_ep_len * 6      # update policy every n timesteps
+        self.K_epochs = 40               # update policy for K epochs in one PDPPO update
+    
+        self.eps_clip = 0.21          # clip parameter for PDPPO
+        self.gamma = 0.991            # discount factor
+    
+        self.lr_actor = 0.0004      # learning rate for actor network
+        self.lr_critic = 0.0012       # learning rate for critic network
+    
+        self.random_seed = 0         # set random seed if required (0 = no random seed)
+        #####################################################
+        self.run_num_pretrained = 0      #### change this to prevent overwriting weights in same self.experiment_name folder
+        
+        print("training environment name : " + self.experiment_name + '_PDPPO')
+        
+        
+    
+        # state space dimension
+        self.state_dim = self.env.observation_space.n
+    
+        # action space dimension
+        if self.has_continuous_action_space:
+            self.action_dim = self.env.action_space.n
+        else:
+            self.action_dim = self.env.action_space.n
+
+        self.pdppo_agent = PDPPO(self.state_dim, self.action_dim, self.lr_actor, self.lr_critic, self.gamma, self.K_epochs, self.eps_clip, copy.copy(self.env), self.has_continuous_action_space,self.tau, self.action_std)
+        
+       
+    ################################### Training ###################################
+    def learn(self,n_episodes = 100000):
+
+    
+        ###################### logging ######################
+        
+        self.max_training_timesteps = n_episodes   # break training loop if timeteps > self.max_training_timesteps
+        
+        env = self.env
+        
+        #### log files for multiple runs are NOT overwritten
+        log_dir = self.LOG_DIR
+        if not os.path.exists(log_dir):
+              os.makedirs(log_dir)
+    
+        log_dir = log_dir + '/' + self.experiment_name + '_PDPPO/'
+        if not os.path.exists(log_dir):
+              os.makedirs(log_dir)
+    
+        #### get number of log files in log directory
+        run_num = 0
+        current_num_files = next(os.walk(log_dir))[2]
+        run_num = len(current_num_files)
+    
+        #### create new log file for each run
+        log_f_name = log_dir + '/PDPPO_' + self.experiment_name + "_log_" + str(run_num) + ".csv"
+    
+        print("current logging run number for " + self.experiment_name + " : ", run_num)
+        print("logging at : " + log_f_name)
+        #####################################################
+    
+        ################### checkpointing ###################
+        
+    
+        directory = self.LOG_DIR
+        if not os.path.exists(directory):
+              os.makedirs(directory)
+    
+        directory = directory + '/' + self.experiment_name + '_PDPPO' + '/'
+        if not os.path.exists(directory):
+              os.makedirs(directory)
+    
+        
+        checkpoint_path = directory + "PDPPO_{}_{}_{}.pth".format(self.experiment_name, self.random_seed, self.run_num_pretrained)
+        print("save checkpoint path : " + checkpoint_path)
+        #####################################################
+    
+    
+        ############# print all hyperparameters #############
+        print("--------------------------------------------------------------------------------------------")
+        print("max training timesteps : ", self.max_training_timesteps)
+        print("max timesteps per episode : ", self.max_ep_len)
+        print("model saving frequency : " + str(self.save_model_freq) + " timesteps")
+        print("log frequency : " + str(self.log_freq) + " timesteps")
+        print("printing average reward over episodes in last : " + str(self.print_freq) + " timesteps")
+        print("--------------------------------------------------------------------------------------------")
+        print("state space dimension : ", self.state_dim)
+        print("action space dimension : ", self.action_dim)
+        print("--------------------------------------------------------------------------------------------")
+        if self.has_continuous_action_space:
+            print("Initializing a continuous action space policy")
+            print("--------------------------------------------------------------------------------------------")
+            print("starting std of action distribution : ", self.action_std)
+            print("decay rate of std of action distribution : ", self.action_std_decay_rate)
+            print("minimum std of action distribution : ", min_self.action_std)
+            print("decay frequency of std of action distribution : " + str(self.action_std_decay_freq) + " timesteps")
+        else:
+            print("Initializing a discrete action space policy")
+        print("--------------------------------------------------------------------------------------------")
+        print("PDPPO update frequency : " + str(self.update_timestep) + " timesteps")
+        print("PDPPO K epochs : ", self.K_epochs)
+        print("PDPPO epsilon clip : ", self.eps_clip)
+        print("discount factor (self.gamma) : ", self.gamma)
+        print("--------------------------------------------------------------------------------------------")
+        print("optimizer learning rate actor : ", self.lr_actor)
+        print("optimizer learning rate critic : ", self.lr_critic)
+        if self.random_seed:
+            print("--------------------------------------------------------------------------------------------")
+            print("setting random seed to ", self.random_seed)
+
+        #####################################################
+    
+        print("============================================================================================")
+    
+        ################# training procedure ################
+    
+        # initialize a PDPPO agent
+        self.PDPPO_agent = PDPPO(self.state_dim, self.action_dim, self.lr_actor, self.lr_critic, self.gamma, self.K_epochs, self.eps_clip, copy.copy(self.env), self.has_continuous_action_space, self.action_std)
+    
+        # track total training time
+        start_time = datetime.now().replace(microsecond=0)
+        print("Started training at (GMT) : ", start_time)
+    
+        print("============================================================================================")
+    
+        # logging file
+        log_f = open(log_f_name,"w+")
+        log_f.write('episode,timestep,reward\n')
+    
+        # printing and logging variables
+        print_running_reward = 0
+        print_running_episodes = 0
+    
+        log_running_reward = 0
+        log_running_episodes = 0
+    
+        time_step = 0
+        i_episode = 0
+        
+        annealing_steps = self.max_training_timesteps  # total number of training steps
+        
+        # training loop
+        while time_step <= self.max_training_timesteps:
+            
+            anneal_rate = (self.tau_end - self.tau_start) / annealing_steps  # rate of tau increase per step
+            
+            self.tau = max(self.tau_end, self.tau_start + anneal_rate * time_step)
+            
+            state = env.reset()
+            current_ep_reward = 0
+    
+            binary_array = np.zeros(self.state_dim, dtype=int)
+            binary_array[state] = 1
+            state = binary_array
+
+            for t in range(1, self.max_ep_len+1):
+    
+                # select action with policy
+                action = self.pdppo_agent.select_action(state,self.tau)
+                state, reward, done, _ = env.step(action.item())
+
+                binary_array = np.zeros(self.state_dim, dtype=int)
+                binary_array[state] = 1
+                state = binary_array
+
+                # saving reward and is_terminals
+                self.pdppo_agent.buffer.rewards.append(reward)
+                self.pdppo_agent.buffer.is_terminals.append(done)
+    
+                time_step +=1
+                current_ep_reward += reward
+    
+                # update PDPPO agent
+                if time_step % self.update_timestep == 0:
+                    self.pdppo_agent.update()
+    
+                # if continuous action space; then decay action std of ouput action distribution
+                if self.has_continuous_action_space and time_step % self.action_std_decay_freq == 0:
+                    self.pdppo_agent.decay_self.action_std(self.action_std_decay_rate, self.action_std)
+    
+                # log in logging file
+                if time_step % self.log_freq == 0:
+    
+                    # log average reward till last episode
+                    log_avg_reward = log_running_reward / log_running_episodes
+                    log_avg_reward = round(log_avg_reward, 4)
+    
+                    log_f.write('{},{},{}\n'.format(i_episode, time_step, log_avg_reward))
+                    log_f.flush()
+    
+                    log_running_reward = 0
+                    log_running_episodes = 0
+    
+                # printing average reward
+                if time_step % self.print_freq == 0:
+    
+                    # print average reward till last episode
+                    print_avg_reward = print_running_reward / print_running_episodes
+                    print_avg_reward = round(print_avg_reward, 2)
+    
+                    print("Episode : {} \t\t Timestep : {} \t\t Average Reward : {}".format(i_episode, time_step, print_avg_reward))
+    
+                    print_running_reward = 0
+                    print_running_episodes = 0
+    
+                # save model weights
+                if time_step % self.save_model_freq == 0:
+                    print("--------------------------------------------------------------------------------------------")
+                    #print("saving model at : " + checkpoint_path)
+                    self.pdppo_agent.save(checkpoint_path)
+                    #print("model saved")
+                    print("PDPPO Elapsed Time  : ", datetime.now().replace(microsecond=0) - start_time)
+                    print("--------------------------------------------------------------------------------------------")
+    
+                # break; if the episode is over
+                if done:
+                    break
+    
+            print_running_reward += current_ep_reward
+            print_running_episodes += 1
+    
+            log_running_reward += current_ep_reward
+            log_running_episodes += 1
+    
+            i_episode += 1
+    
+        log_f.close()
+        #env.close()
+    
+        # print total training time
+        print("============================================================================================")
+        end_time = datetime.now().replace(microsecond=0)
+        print("Started training at (GMT) : ", start_time)
+        print("Finished training at (GMT) : ", end_time)
+        print("Total training time  : ", end_time - start_time)
+        print("============================================================================================")
+
+    def get_action(self,state):
+        if isinstance(state, dict):    
+            if not self.dict_obs:
+                state = np.concatenate(
+                    (
+                        state['inventory_level'], # n_items size
+                        state['machine_setup'], # n_machine size
+                    )
+                )
+        else:
+            if self.dict_obs:
+                raise('Change dict_obst to False')
+        return self.pdppo_agent.select_action(state,self.tau)
+    
+    def load_agent(self,path):
+        #directory = "PDPPO_preTrained" + '/' + env_name + '/'
+        directory = self.LOG_DIR
+        directory = directory + '/' + self.experiment_name + '_PDPPO' + '/'
+        checkpoint_path = directory + "PDPPO_{}_{}_{}.pth".format(self.experiment_name, self.random_seed, self.run_num_pretrained)
+        print("loading network from : " + checkpoint_path)
+        self.pdppo_agent.load(checkpoint_path)

code/Lake application/agents/PDPPO_two_actors.py

@@ -0,0 +1,353 @@
+# -*- coding: utf-8 -*-
+"""
+Created on Wed Mar  1 00:43:49 2023
+
+@author: leona
+"""
+
+import torch
+import numpy as np
+import torch.nn as nn
+from torch.distributions import MultivariateNormal
+from torch.distributions import Categorical
+
+################################## set device ##################################
+print("============================================================================================")
+# set device to cpu or cuda
+device = torch.device('cpu')
+if(torch.cuda.is_available()): 
+    device = torch.device('cuda:0') 
+    torch.cuda.empty_cache()
+    print("Device set to : " + str(torch.cuda.get_device_name(device)))
+else:
+    print("Device set to : cpu")
+print("============================================================================================")
+
+
+################################## PDPPO Policy ##################################
+class RolloutBuffer:
+    def __init__(self):
+        self.actions = []
+        self.actions_pre = []
+        self.actions_post = []
+        self.states = []
+        self.pre_states = []
+        self.post_states = []
+        self.logprobs = []
+        self.logprobs_pre = []
+        self.logprobs_post = []
+        self.rewards = []
+        self.rewards_pre = []
+        self.rewards_post = []
+        self.state_values = []
+        self.state_values_post = []
+        self.is_terminals = []
+    
+    def clear(self):
+        del self.actions[:]
+        del self.actions_pre[:]
+        del self.actions_post[:]
+        del self.states[:]
+        del self.pre_states[:]
+        del self.post_states[:]
+        del self.logprobs[:]
+        del self.logprobs_pre[:]
+        del self.logprobs_post[:]
+        del self.rewards[:]
+        del self.rewards_pre[:]
+        del self.rewards_post[:]
+        del self.state_values[:]
+        del self.state_values_post[:]
+        del self.is_terminals[:]
+
+class ActorCritic(nn.Module):
+    def __init__(self, state_dim,state_dim_pre,state_dim_post, action_dim, has_continuous_action_space, action_std_init):
+        super(ActorCritic, self).__init__()
+        # actor - multidiscrete
+        self.action_dim = action_dim
+        
+        
+        self.actor = nn.Sequential(
+                        nn.Linear(state_dim, 128),
+                        nn.Linear(128, 128),
+                        nn.Linear(128, self.action_dim.nvec.sum())
+                    )
+        
+        self.actor_pre = nn.Sequential(
+                        nn.Linear(state_dim_pre, 128),
+                        nn.Linear(128, 128),
+                        nn.Linear(128, self.action_dim.nvec.sum())
+                    )
+        self.actor_post = nn.Sequential(
+                        nn.Linear(state_dim_post, 128),
+                        nn.Linear(128, 128),
+                        nn.Linear(128, self.action_dim.nvec.sum())
+                    )
+        
+            
+        # critic
+        self.critic_pre = nn.Sequential(
+                        nn.Linear(state_dim_pre, 128),
+                        # nn.Tanh(),
+                        # nn.Linear(64, 64),
+                        nn.Tanh(),
+                        nn.Linear(128, 1)
+                    )
+        
+        self.critic_post = nn.Sequential(
+                        nn.Linear(state_dim_post, 128),
+                        # nn.Tanh(),
+                        # nn.Linear(64, 64),
+                        nn.Tanh(),
+                        nn.Linear(128, 1)
+                    )
+    
+    def forward(self, state):
+        raise NotImplementedError
+    
+    
+    
+    def set_action_std(self, new_action_std):
+        if self.has_continuous_action_space:
+            self.action_var = torch.full((self.action_dim,), new_action_std * new_action_std).to(device)
+        else:
+            print("--------------------------------------------------------------------------------------------")
+            print("WARNING : Calling ActorCritic::set_action_std() on discrete action space policy")
+            print("--------------------------------------------------------------------------------------------")
+
+
+    
+    def act(self, state):
+
+        # x = nn.functional.relu(self.fc(state))
+        x = nn.functional.relu(self.fc2(nn.functional.relu(self.fc1(state))))
+        logits = self.actor(x)
+        action_probs = nn.functional.softmax(logits, dim=-1)
+        dist = Categorical(action_probs.view(len(self.action_dim.nvec),-1))
+
+        action = dist.sample()
+        action_logprob = dist.log_prob(action)
+
+        return action.detach(), action_logprob.detach()
+    
+    def evaluate(self, state, pre_state, post_state, action):
+
+        # x = nn.functional.relu(self.fc(state))
+        x = nn.functional.relu(self.fc2(nn.functional.relu(self.fc1(state))))
+        logits = self.actor(x)
+        action_probs = nn.functional.softmax(logits, dim=-1)
+        dist = Categorical(action_probs.view(state.shape[0],len(self.action_dim.nvec),-1))
+        # action_probs = self.actor(state)
+        # dist = Categorical(action_probs)
+            
+        action_logprobs = dist.log_prob(action)
+        dist_entropy = dist.entropy()
+        state_values_pre = self.critic_pre(pre_state)
+        state_values_post = self.critic_post(post_state)
+        
+        return action_logprobs, state_values_pre,state_values_post, dist_entropy
+
+
+class PDPPO:
+    def __init__(self, state_dim, action_dim, lr_actor, lr_critic, gamma, K_epochs, eps_clip, has_continuous_action_space, env, action_std_init=0.6):
+
+        self.has_continuous_action_space = has_continuous_action_space
+
+        if has_continuous_action_space:
+            self.action_std = action_std_init
+        
+        self.env = env
+        
+        self.reward_old_pre = -np.inf
+        self.reward_old_post = -np.inf
+        
+        self.gamma = gamma
+        self.eps_clip = eps_clip
+        self.K_epochs = K_epochs
+        
+        self.buffer = RolloutBuffer()
+        
+        state_dim_pre = self.env.n_machines
+        state_dim_post = self.env.n_items
+        
+        self.policy = ActorCritic(state_dim,state_dim_pre,state_dim_post, action_dim, has_continuous_action_space, action_std_init).to(device)
+        self.optimizer = torch.optim.Adam([
+                        {'params': self.policy.actor.parameters(), 'lr': lr_actor},
+                        {'params': self.policy.critic.parameters(), 'lr': lr_critic*10},
+                        {'params': self.policy.critic_post.parameters(), 'lr': lr_critic*1}
+                     ])
+    
+        self.policy_old = ActorCritic(state_dim,state_dim_pre,state_dim_post, action_dim, has_continuous_action_space, action_std_init).to(device)
+        self.policy_old.load_state_dict(self.policy.state_dict())
+        
+        self.MseLoss = nn.MSELoss()
+
+    
+    def get_post_state(self, action, machine_setup, inventory_level):
+        setup_loss = np.zeros(self.env.n_machines, dtype=int)
+        setup_costs = np.zeros(self.env.n_machines)
+        # if we are just changing the setup, we use the setup cost matrix with the corresponding position given by the actual setup and the new setup
+        for m in range(self.env.n_machines):   
+            if action[m] != 0: # if the machine is not iddle
+                # 1. IF NEEDED CHANGE SETUP
+                if machine_setup[m] != action[m] and action[m] != 0:
+                    setup_costs[m] = self.env.setup_costs[m][action[m] - 1] 
+                    setup_loss[m] = self.env.setup_loss[m][action[m] - 1]
+                machine_setup[m] = action[m]
+                # 2. PRODUCTION
+                production = self.env.machine_production_matrix[m][action[m] - 1] - setup_loss[m]
+                inventory_level[action[m] - 1] += production
+            else:
+                machine_setup[m] = 0
+        # return the new machine_setup_inventory_level and the setup_cost        
+        return machine_setup, inventory_level, setup_costs    
+    
+    def select_action(self, state):
+        with torch.no_grad():
+            state = torch.FloatTensor(state).to(device)
+            action, action_logprob = self.policy_old.act(state)
+            
+        
+        pre_state = state[self.env.n_items:self.env.n_items+self.env.n_machines].clone()
+        
+        machine_setup, inventory_level, setup_cost = self.get_post_state(action, state[self.env.n_items:self.env.n_items+self.env.n_machines], state[0:self.env.n_items])
+        
+        post_state = inventory_level.clone()
+        
+        with torch.no_grad():
+            action_pre, action_logprob_pre = self.policy_old.act_pre(pre_state)
+            action_post, action_logprob_post = self.policy_old.act_post(post_state)
+        
+        
+        self.buffer.states.append(state)
+        self.buffer.pre_states.append(pre_state)
+        self.buffer.post_states.append(post_state)
+        self.buffer.actions.append(action)
+        self.buffer.actions_pre.append(action_pre)
+        self.buffer.actions_post.append(action_post)
+        self.buffer.logprobs.append(action_logprob)
+        self.buffer.logprobs.append(action_logprob_pre)
+        self.buffer.logprobs.append(action_logprob_post)
+        
+        with torch.no_grad():
+            state_val = self.policy_old.critic(pre_state).detach()
+            state_val_post = self.policy_old.critic_post(post_state).detach()
+        
+        self.buffer.state_values.append(state_val)
+        self.buffer.state_values_post.append(state_val_post)
+        
+        if self.has_continuous_action_space:
+            return action.detach().cpu().numpy().flatten() 
+        
+        else:
+            return action.numpy()
+    
+    def update(self):
+        
+        
+        # Monte Carlo estimate of returns
+        rewards = []
+        discounted_reward = 0
+        for reward, is_terminal in zip(reversed(self.buffer.rewards), reversed(self.buffer.is_terminals)):
+            if is_terminal:
+                discounted_reward = 0
+            discounted_reward = reward + (self.gamma * discounted_reward)
+            rewards.insert(0, discounted_reward)
+            
+        # Normalizing the rewards
+        rewards = torch.tensor(rewards, dtype=torch.float32).to(device)
+        rewards = rewards/(-rewards).max()
+        #rewards = rewards - rewards.min()
+        rewards = (rewards - rewards.mean()) / (rewards.std() + 1e-7)
+        
+        # Monte Carlo estimate of returns (pre decision)
+        rewards_pre = []
+        discounted_reward = 0
+        for reward_pre, is_terminal in zip(reversed(self.buffer.rewards_pre), reversed(self.buffer.is_terminals)):
+            if is_terminal:
+                discounted_reward = 0
+            discounted_reward = reward_pre + (self.gamma * discounted_reward)
+            rewards_pre.insert(0, discounted_reward)
+            
+        # Normalizing the rewards
+        rewards_pre = torch.tensor(rewards_pre, dtype=torch.float32).to(device)
+        #rewards_pre = rewards_pre/(-rewards_pre).max()
+        #rewards_pre = rewards_pre - rewards_pre.min()
+        rewards_pre = (rewards_pre - rewards_pre.mean()) / (rewards_pre.std() + 1e-7)
+        
+        # Monte Carlo estimate of returns (post decision)
+        rewards_post = []
+        discounted_reward = 0
+        for reward_post, is_terminal in zip(reversed(self.buffer.rewards_post), reversed(self.buffer.is_terminals)):
+            if is_terminal:
+                discounted_reward = 0
+            discounted_reward = reward_post + (self.gamma * discounted_reward)
+            rewards_post.insert(0, discounted_reward)
+            
+        # Normalizing the rewards
+        rewards_post = torch.tensor(rewards_post, dtype=torch.float32).to(device)
+        #rewards_post = rewards_post/(-rewards_post).max()
+        #rewards_post = rewards_post - rewards_post.min()
+        rewards_post = (rewards_post - rewards_post.mean()) / (rewards_post.std() + 1e-7)
+        
+        # rewards_post = -rewards_post/(rewards_pre + rewards_post).min()
+        
+        # rewards_pre = -rewards_pre/(rewards_pre + rewards_post).min()
+        
+        # convert list to tensor
+        old_states = torch.squeeze(torch.stack(self.buffer.states, dim=0)).detach().to(device)
+        old_pre_states = torch.squeeze(torch.stack(self.buffer.pre_states, dim=0)).detach().to(device)
+        old_post_states = torch.squeeze(torch.stack(self.buffer.post_states, dim=0)).detach().to(device)
+        old_actions = torch.squeeze(torch.stack(self.buffer.actions, dim=0)).detach().to(device)
+        old_actions_pre = torch.squeeze(torch.stack(self.buffer.actions, dim=0)).detach().to(device)
+        old_actions_post = torch.squeeze(torch.stack(self.buffer.actions, dim=0)).detach().to(device)
+        old_logprobs = torch.squeeze(torch.stack(self.buffer.logprobs, dim=0)).detach().to(device)
+        old_logprobs_pre = torch.squeeze(torch.stack(self.buffer.logprobs, dim=0)).detach().to(device)
+        old_logprobs_post = torch.squeeze(torch.stack(self.buffer.logprobs, dim=0)).detach().to(device)
+        old_state_values = torch.squeeze(torch.stack(self.buffer.state_values, dim=0)).detach().to(device)
+        old_state_values_post = torch.squeeze(torch.stack(self.buffer.state_values_post, dim=0)).detach().to(device)
+
+        # calculate advantages
+        advantages_post = rewards_post.detach() - old_state_values_post.detach()
+        advantages_pre = rewards_pre.detach() - old_state_values.detach()
+        advantages = rewards.detach() - old_state_values_post.detach() - old_state_values.detach()
+        
+        sum_loss = 0
+        
+        # Optimize policy for K epochs
+        for i in range(self.K_epochs):
+            
+            # Evaluating old actions and values
+            logprobs, logprobs_pre, logprobs_post, state_values, state_values_post, dist_entropy, dist_entropy_pre, dist_entropy_post = self.policy.evaluate(old_states, old_pre_states, old_post_states, old_actions, old_actions_pre, old_actions_post)
+
+            # Finding the ratio (pi_theta / pi_theta__old)
+            ratios = torch.exp(logprobs - old_logprobs)
+            
+            # Finding Surrogate Loss
+            surr1 = ratios * advantages.unsqueeze(1)
+            surr2 = torch.clamp(ratios, 1 - self.eps_clip, 1 + self.eps_clip) * advantages.unsqueeze(1)
+            
+            surr = -torch.min(surr1, surr2)
+        
+            loss = surr + 0.5 * self.MseLoss(old_state_values_post, old_state_values) - 0.01*dist_entropy
+            
+            # Optmization - gradient backpropagation
+            self.optimizer.zero_grad()
+            loss.mean().backward(retain_graph=True)
+            self.optimizer.step()
+
+            
+        # print('Avg Loss: {}'.format(sum_loss.mean().item()))
+        print('Last Loss {}'.format(loss.sum().item()))
+        # Copy new weights into old policy
+        self.policy_old.load_state_dict(self.policy.state_dict())
+
+        # clear buffer
+        self.buffer.clear()
+    
+    def save(self, checkpoint_path):
+        torch.save(self.policy_old.state_dict(), checkpoint_path)
+   
+    def load(self, checkpoint_path):
+        self.policy_old.load_state_dict(torch.load(checkpoint_path, map_location=lambda storage, loc: storage))
+        self.policy.load_state_dict(torch.load(checkpoint_path, map_location=lambda storage, loc: storage))

code/Lake application/agents/PDPPO_two_critics_two_actors.py

@@ -0,0 +1,377 @@
+# -*- coding: utf-8 -*-
+"""
+Created on Wed Mar  1 00:43:49 2023
+
+@author: leona
+"""
+
+import torch
+import numpy as np
+import torch.nn as nn
+from torch.distributions import MultivariateNormal
+from torch.distributions import Categorical
+
+################################## set device ##################################
+print("============================================================================================")
+# set device to cpu or cuda
+device = torch.device('cpu')
+if(torch.cuda.is_available()): 
+    device = torch.device('cuda:0') 
+    torch.cuda.empty_cache()
+    print("Device set to : " + str(torch.cuda.get_device_name(device)))
+else:
+    print("Device set to : cpu")
+print("============================================================================================")
+
+
+################################## PDPPO Policy ##################################
+class RolloutBuffer:
+    def __init__(self):
+        self.actions = []
+        self.states = []
+        self.pre_states = []
+        self.post_states = []
+        self.logprobs = []
+        self.rewards = []
+        self.rewards_pre = []
+        self.rewards_post = []
+        self.state_values = []
+        self.state_values_post = []
+        self.is_terminals = []
+    
+    def clear(self):
+        del self.actions[:]
+        del self.states[:]
+        del self.pre_states[:]
+        del self.post_states[:]
+        del self.logprobs[:]
+        del self.rewards[:]
+        del self.rewards_pre[:]
+        del self.rewards_post[:]
+        del self.state_values[:]
+        del self.state_values_post[:]
+        del self.is_terminals[:]
+
+class ActorCritic(nn.Module):
+    def __init__(self, state_dim,state_dim_pre,state_dim_post, action_dim, has_continuous_action_space, action_std_init):
+        super(ActorCritic, self).__init__()
+
+        self.has_continuous_action_space = has_continuous_action_space
+        
+        if has_continuous_action_space:
+            self.action_dim = action_dim
+            self.action_var = torch.full((action_dim,), action_std_init * action_std_init).to(device)
+        # actor
+        if has_continuous_action_space :
+            self.actor = nn.Sequential(
+                            nn.Linear(state_dim, 64),
+                            nn.Tanh(),
+                            nn.Linear(64, 64),
+                            nn.Tanh(),
+                            nn.Linear(64, action_dim),
+                            nn.Tanh()
+                        )
+        else:
+            # actor - multidiscrete
+            self.action_dim = action_dim
+            # self.fc = nn.Linear(state_dim, 64)
+            self.fc1 = nn.Linear(state_dim, 128)
+            self.fc2 = nn.Linear(128, 128)
+            self.actor = nn.Linear(128, self.action_dim.nvec.sum())
+            
+        # critic
+        self.critic = nn.Sequential(
+                        nn.Linear(state_dim, 128),
+                        nn.Tanh(),
+                        nn.Linear(128, 128),
+                        nn.Tanh(),
+                        nn.Linear(128, 1)
+                    )
+        
+        self.critic_post = nn.Sequential(
+                        nn.Linear(state_dim, 128),
+                        nn.Tanh(),
+                        nn.Linear(128, 128),
+                        nn.Tanh(),
+                        nn.Linear(128, 1)
+                    )
+    
+    def forward(self, state):
+        raise NotImplementedError
+    
+    
+    
+    def set_action_std(self, new_action_std):
+        if self.has_continuous_action_space:
+            self.action_var = torch.full((self.action_dim,), new_action_std * new_action_std).to(device)
+        else:
+            print("--------------------------------------------------------------------------------------------")
+            print("WARNING : Calling ActorCritic::set_action_std() on discrete action space policy")
+            print("--------------------------------------------------------------------------------------------")
+
+
+    
+    def act(self, state):
+
+        if self.has_continuous_action_space:
+            action_mean = self.actor(state)
+            cov_mat = torch.diag(self.action_var).unsqueeze(dim=0)
+            dist = MultivariateNormal(action_mean, cov_mat)
+        else:
+            # x = nn.functional.relu(self.fc(state))
+            x = nn.functional.relu(self.fc2(nn.functional.relu(self.fc1(state))))
+            logits = self.actor(x)
+            action_probs = nn.functional.softmax(logits, dim=-1)
+            dist = Categorical(action_probs.view(len(self.action_dim.nvec),-1))
+
+        action = dist.sample()
+        action_logprob = dist.log_prob(action)
+
+        return action.detach(), action_logprob.detach()
+    
+    def evaluate(self, state, pre_state, post_state, action):
+
+        if self.has_continuous_action_space:
+            action_mean = self.actor(state)
+            
+            action_var = self.action_var.expand_as(action_mean)
+            cov_mat = torch.diag_embed(action_var).to(device)
+            dist = MultivariateNormal(action_mean, cov_mat)
+            
+            # For Single Action Environments.
+            if self.action_dim == 1:
+                action = action.reshape(-1, self.action_dim)
+        else:
+            # x = nn.functional.relu(self.fc(state))
+            x = nn.functional.relu(self.fc2(nn.functional.relu(self.fc1(state))))
+            logits = self.actor(x)
+            action_probs = nn.functional.softmax(logits, dim=-1)
+            dist = Categorical(action_probs.view(state.shape[0],len(self.action_dim.nvec),-1))
+            # action_probs = self.actor(state)
+            # dist = Categorical(action_probs)
+        action_logprobs = dist.log_prob(action)
+        dist_entropy = dist.entropy()
+        state_values = self.critic(pre_state)
+        state_values_post = self.critic_post(post_state)
+        
+        return action_logprobs, state_values,state_values_post, dist_entropy
+
+
+class PDPPO:
+    def __init__(self, state_dim, action_dim, lr_actor, lr_critic, gamma, K_epochs, eps_clip, has_continuous_action_space, env, action_std_init=0.6):
+
+        self.has_continuous_action_space = has_continuous_action_space
+
+        if has_continuous_action_space:
+            self.action_std = action_std_init
+        
+        self.env = env
+        
+        self.reward_old_pre = -np.inf
+        self.reward_old_post = -np.inf
+        
+        self.gamma = gamma
+        self.eps_clip = eps_clip
+        self.K_epochs = K_epochs
+        
+        self.buffer = RolloutBuffer()
+        
+        state_dim_pre = self.env.n_machines
+        state_dim_post = self.env.n_items
+        
+        self.policy = ActorCritic(state_dim,state_dim_pre,state_dim_post, action_dim, has_continuous_action_space, action_std_init).to(device)
+        self.optimizer = torch.optim.Adam([
+                        {'params': self.policy.actor.parameters(), 'lr': lr_actor},
+                        {'params': self.policy.critic.parameters(), 'lr': lr_critic*10},
+                        {'params': self.policy.critic_post.parameters(), 'lr': lr_critic*1}
+                     ])
+    
+        self.policy_old = ActorCritic(state_dim,state_dim_pre,state_dim_post, action_dim, has_continuous_action_space, action_std_init).to(device)
+        self.policy_old.load_state_dict(self.policy.state_dict())
+        
+        self.MseLoss = nn.MSELoss()
+
+    def set_action_std(self, new_action_std):
+        if self.has_continuous_action_space:
+            self.action_std = new_action_std
+            self.policy.set_action_std(new_action_std)
+            self.policy_old.set_action_std(new_action_std)
+        else:
+            print("--------------------------------------------------------------------------------------------")
+            print("WARNING : Calling PDPPO::set_action_std() on discrete action space policy")
+            print("--------------------------------------------------------------------------------------------")
+
+    def decay_action_std(self, action_std_decay_rate, min_action_std):
+        print("--------------------------------------------------------------------------------------------")
+        if self.has_continuous_action_space:
+            self.action_std = self.action_std - action_std_decay_rate
+            self.action_std = round(self.action_std, 4)
+            if (self.action_std <= min_action_std):
+                self.action_std = min_action_std
+                print("setting actor output action_std to min_action_std : ", self.action_std)
+            else:
+                print("setting actor output action_std to : ", self.action_std)
+            self.set_action_std(self.action_std)
+
+        else:
+            print("WARNING : Calling PDPPO::decay_action_std() on discrete action space policy")
+        print("--------------------------------------------------------------------------------------------")
+    
+    def get_post_state(self, action, machine_setup, inventory_level):
+        setup_loss = np.zeros(self.env.n_machines, dtype=int)
+        setup_costs = np.zeros(self.env.n_machines)
+        # if we are just changing the setup, we use the setup cost matrix with the corresponding position given by the actual setup and the new setup
+        for m in range(self.env.n_machines):   
+            if action[m] != 0: # if the machine is not iddle
+                # 1. IF NEEDED CHANGE SETUP
+                if machine_setup[m] != action[m] and action[m] != 0:
+                    setup_costs[m] = self.env.setup_costs[m][action[m] - 1] 
+                    setup_loss[m] = self.env.setup_loss[m][action[m] - 1]
+                machine_setup[m] = action[m]
+                # 2. PRODUCTION
+                production = self.env.machine_production_matrix[m][action[m] - 1] - setup_loss[m]
+                inventory_level[action[m] - 1] += production
+            else:
+                machine_setup[m] = 0
+        # return the new machine_setup_inventory_level and the setup_cost        
+        return machine_setup, inventory_level, setup_costs    
+    
+    def select_action(self, state):
+        with torch.no_grad():
+            state = torch.FloatTensor(state).to(device)
+            action, action_logprob = self.policy_old.act(state)
+        
+        #pre_state = state[self.env.n_items:self.env.n_items+self.env.n_machines].clone()
+        
+        machine_setup, inventory_level, setup_cost = self.get_post_state(action, state[self.env.n_items:self.env.n_items+self.env.n_machines], state[0:self.env.n_items])
+        
+        post_state = state.clone()
+        post_state[self.env.n_items:self.env.n_items+self.env.n_machines] = machine_setup.clone()
+        post_state[0:self.env.n_items] = inventory_level.clone()
+        
+        pre_state = state.clone()
+        
+        self.buffer.states.append(state)
+        self.buffer.pre_states.append(pre_state)
+        self.buffer.post_states.append(post_state)
+        self.buffer.actions.append(action)
+        self.buffer.logprobs.append(action_logprob)
+        
+        with torch.no_grad():
+            state_val = self.policy_old.critic(pre_state).detach()
+            state_val_post = self.policy_old.critic_post(post_state).detach()
+        
+        self.buffer.state_values.append(state_val)
+        self.buffer.state_values_post.append(state_val_post)
+        
+        if self.has_continuous_action_space:
+            return action.detach().cpu().numpy().flatten() 
+        
+        else:
+            return action.numpy()
+    
+    def update(self):
+        
+        
+        # Monte Carlo estimate of returns
+        rewards = []
+        discounted_reward = 0
+        for reward, is_terminal in zip(reversed(self.buffer.rewards), reversed(self.buffer.is_terminals)):
+            if is_terminal:
+                discounted_reward = 0
+            discounted_reward = reward + (self.gamma * discounted_reward)
+            rewards.insert(0, discounted_reward)
+            
+        # Normalizing the rewards
+        rewards = torch.tensor(rewards, dtype=torch.float32).to(device)
+        #rewards = rewards/(-rewards).max()
+        #rewards = rewards - rewards.min()
+        #rewards = (rewards - rewards.mean()) / (rewards.std() + 1e-7)
+        
+        # Monte Carlo estimate of returns (pre decision)
+        rewards_pre = []
+        discounted_reward = 0
+        for reward_pre, is_terminal in zip(reversed(self.buffer.rewards_pre), reversed(self.buffer.is_terminals)):
+            if is_terminal:
+                discounted_reward = 0
+            discounted_reward = reward_pre + (self.gamma * discounted_reward)
+            rewards_pre.insert(0, discounted_reward)
+            
+        # Normalizing the rewards
+        rewards_pre = torch.tensor(rewards_pre, dtype=torch.float32).to(device)
+        #rewards_pre = rewards_pre/(-rewards_pre).max()
+        #rewards_pre = rewards_pre - rewards_pre.min()
+        rewards_pre = (rewards_pre - rewards_pre.mean()) / (rewards_pre.std() + 1e-7)
+        
+        # Monte Carlo estimate of returns (post decision)
+        rewards_post = []
+        discounted_reward = 0
+        for reward_post, is_terminal in zip(reversed(self.buffer.rewards_post), reversed(self.buffer.is_terminals)):
+            if is_terminal:
+                discounted_reward = 0
+            discounted_reward = reward_post + (self.gamma * discounted_reward)
+            rewards_post.insert(0, discounted_reward)
+            
+        # Normalizing the rewards
+        rewards_post = torch.tensor(rewards_post, dtype=torch.float32).to(device)
+        #rewards_post = rewards_post/(-rewards_post).max()
+        #rewards_post = rewards_post - rewards_post.min()
+        rewards_post = (rewards_post - rewards_post.mean()) / (rewards_post.std() + 1e-7)
+        
+        #rewards_post = -rewards_post/(rewards_pre + rewards_post).min()
+        
+        #rewards_pre = -rewards_pre/(rewards_pre + rewards_post).min()
+        
+        # convert list to tensor
+        old_states = torch.squeeze(torch.stack(self.buffer.states, dim=0)).detach().to(device)
+        old_pre_states = torch.squeeze(torch.stack(self.buffer.pre_states, dim=0)).detach().to(device)
+        old_post_states = torch.squeeze(torch.stack(self.buffer.post_states, dim=0)).detach().to(device)
+        old_actions = torch.squeeze(torch.stack(self.buffer.actions, dim=0)).detach().to(device)
+        old_logprobs = torch.squeeze(torch.stack(self.buffer.logprobs, dim=0)).detach().to(device)
+        old_state_values = torch.squeeze(torch.stack(self.buffer.state_values, dim=0)).detach().to(device)
+        old_state_values_post = torch.squeeze(torch.stack(self.buffer.state_values_post, dim=0)).detach().to(device)
+
+        # calculate advantages
+        #advantages_post = rewards_post.detach() - old_state_values_post.detach()
+        #advantages_pre = rewards_pre.detach() - old_state_values.detach()
+        advantages = rewards - (old_state_values.detach()+old_state_values_post.detach())#(rewards_post + rewards_pre) - old_state_values_post.detach() - old_state_values.detach()# torch.min(advantages_pre,advantages_post)
+        
+        sum_loss = 0
+        
+        # Optimize policy for K epochs
+        for i in range(self.K_epochs):
+            
+            # Evaluating old actions and values
+            logprobs, state_values, state_values_post, dist_entropy = self.policy.evaluate(old_states, old_pre_states, old_post_states, old_actions)
+
+            # Finding the ratio (pi_theta / pi_theta__old)
+            ratios = torch.exp(logprobs - old_logprobs)
+            
+            # Finding Surrogate Loss
+            surr1 = ratios * advantages.unsqueeze(1)
+            surr2 = torch.clamp(ratios, 1 - self.eps_clip, 1 + self.eps_clip) * advantages.unsqueeze(1)
+            
+            surr = -torch.min(surr1, surr2)
+        
+            loss = surr + 0.5 * self.MseLoss(state_values_post+state_values,rewards) - 0.01*dist_entropy
+            
+            # Optmization - gradient backpropagation
+            self.optimizer.zero_grad()
+            loss.mean().backward(retain_graph=True)
+            self.optimizer.step()
+
+            
+        # print('Avg Loss: {}'.format(sum_loss.mean().item()))
+        print('Last Loss {}'.format(loss.sum().item()))
+        # Copy new weights into old policy
+        self.policy_old.load_state_dict(self.policy.state_dict())
+
+        # clear buffer
+        self.buffer.clear()
+    
+    def save(self, checkpoint_path):
+        torch.save(self.policy_old.state_dict(), checkpoint_path)
+   
+    def load(self, checkpoint_path):
+        self.policy_old.load_state_dict(torch.load(checkpoint_path, map_location=lambda storage, loc: storage))
+        self.policy.load_state_dict(torch.load(checkpoint_path, map_location=lambda storage, loc: storage))

code/Lake application/agents/PDPPO_v0.py

@@ -0,0 +1,328 @@
+import os
+import copy
+import numpy as np
+import torch
+import torch.nn as nn
+import torch.optim as optim
+import torch.nn.functional as F
+from torch.distributions import Categorical
+from envs import *
+import gym
+
+
+
+class SimplePlantSB(SimplePlant):
+    def __init__(self, settings, stoch_model):
+        super().__init__(settings, stoch_model)
+        try:self.dict_obs = settings['dict_obs']
+        except:self.dict_obs = False
+        self.last_inventory = copy.copy(self.inventory_level)
+        self.action_space = gym.spaces.MultiDiscrete(
+            [self.n_items+1] * self.n_machines
+        )
+        
+        if self.dict_obs:
+            self.observation_space = gym.spaces.Dict({
+                'inventory_level': gym.spaces.Box(low = np.zeros(self.n_items),high = np.ones(self.n_items)*(settings['max_inventory_level'][0]+1)*self.n_items),
+                'machine_setup': gym.spaces.MultiDiscrete([self.n_items+1] * self.n_machines)
+            })
+        else:
+            self.observation_space = gym.spaces.Box(
+                low=np.zeros(self.n_items+self.n_machines),# high for the inventory level
+                high=np.concatenate(
+                    [
+                        np.array(self.max_inventory_level),
+                        np.ones(self.n_machines) * (self.n_items+1), #high for the machine setups 
+                    ]),
+                dtype=np.int32
+            )
+
+    def step(self, action):
+        """
+        Step method: Execute one time step within the environment
+
+        Parameters
+        ----------
+        action : action given by the agent
+
+        Returns
+        -------
+        obs : Observation of the state give the method _next_observation
+        reward : Cost given by the _reward method
+        done : returns True or False given by the _done method
+        dict : possible information for control to environment monitoring
+
+        """
+        self.last_inventory = copy.copy(self.inventory_level)
+            
+        self.total_cost = self._take_action(action, self.machine_setup, self.inventory_level, self.demand)
+        
+        # self.total_cost['setup_costs'] = 0
+        # self.total_cost['holding_costs'] = 0
+        
+        reward = -sum([ele for key, ele in self.total_cost.items()])
+        #reward = -self.total_cost['lost_sales']
+        
+        #reward = np.abs(action)
+        
+        self.current_step += 1
+        done = self.current_step == self.T
+        obs = self._next_observation()
+
+        return obs, reward, done, self.total_cost
+
+    def _next_observation(self):
+        """
+        Returns the next demand
+        """
+        obs = SimplePlant._next_observation(self)
+        #obs['last_inventory_level'] = copy.copy(self.last_inventory)
+        if isinstance(obs, dict):    
+            if not self.dict_obs:
+                obs = np.concatenate(
+                    (
+                        obs['inventory_level'], # n_items size
+                        obs['machine_setup'], # n_machine size
+                        #obs['last_inventory_level']# n_items size
+                    )
+                )
+        else:
+            if self.dict_obs:
+                raise('Change dict_obst to False')
+        return obs
+
+# Define the policy network
+class Policy(nn.Module):
+    def __init__(self, input_size, output_shape):
+        super(Policy, self).__init__()
+        self.fc1 = nn.Linear(input_size, 128)
+        self.fc_list = nn.ModuleList([nn.Linear(128, output_shape[0]) for list(output_shape)[1] in range(0,output_shape[1])])
+    
+    def forward(self, x):
+        x = F.relu(self.fc1(x)).requires_grad_()
+        outputs = [F.softmax(fc(x), dim=1)for fc in self.fc_list]
+        return outputs
+
+# Define the value network for deterministic components
+class Value(nn.Module):
+    def __init__(self,input_size,output_size):
+        super(Value, self).__init__()
+        self.fc1 = nn.Linear(input_size, 128)
+        self.fc2 = nn.Linear(128, output_size)
+
+    def forward(self, x):
+        x = F.relu(self.fc1(x)).requires_grad_()
+        x = self.fc2(x)
+        return x
+
+# Define the value network for stochastic components
+class ValueStochastic(nn.Module):
+    def __init__(self,input_size,output_size):
+        super(ValueStochastic, self).__init__()
+        self.fc1 = nn.Linear(input_size, 128)
+        self.fc2 = nn.Linear(128, output_size)
+
+    def forward(self, x):
+        x = F.relu(self.fc1(x)).requires_grad_()
+        x = F.softmax(self.fc2(x), dim=1)
+        return x
+
+# Define the PPO agent
+class PDPPO:
+    def __init__(self, env: SimplePlant, settings: dict):
+        
+        self.env = SimplePlantSB(env.settings, env.stoch_model)
+        self.last_inventory = env.inventory_level
+        self.experiment_name = settings['experiment_name']
+        try:self.dict_obs = settings['dict_obs']
+        except:self.dict_obs = False
+        
+        self.POSSIBLE_STATES = self.env.n_items + 1
+        self.env.cost_to_reward = True
+        self.epsilon = 0
+
+        BASE_DIR = os.path.dirname(os.path.dirname(os.path.abspath(__file__)))    
+        # Use the logs file in the root path of the main.
+        self.LOG_DIR = os.path.join(BASE_DIR,'logs')
+        
+        
+        if self.dict_obs == False:
+            input_size = self.env.observation_space.shape[0]
+        output_size_policy = (self.env.n_items+1, self.env.action_space.shape[0]) # we add 1 for the idle state
+        output_size_value = self.env.action_space.shape[0]
+        self.policy = Policy(input_size,output_size_policy)
+        self.value = Value(input_size,output_size_value)
+        self.value_post = ValueStochastic(input_size,output_size_value)
+        self.optimizer_policy = optim.Adam(self.policy.parameters(), lr=1e-3)
+        self.optimizer_value = optim.Adam(self.value.parameters(), lr=1e-3)
+        self.optimizer_value_post = optim.Adam(self.value_post.parameters(), lr=1e-3)
+        self.eps_clip = 0.2
+        self.gamma = 0.99
+        self.lmbda = 0.95
+
+    def get_post_state(self, action, machine_setup, inventory_level):
+        setup_loss = np.zeros(self.env.n_machines, dtype=int)
+        setup_costs = np.zeros(self.env.n_machines)
+        # if we are just changing the setup, we use the setup cost matrix with the corresponding position given by the actual setup and the new setup
+        for m in range(self.env.n_machines):   
+            if action[m] != 0: # if the machine is not iddle
+                # 1. IF NEEDED CHANGE SETUP
+                if machine_setup[m] != action[m] and action[m] != 0:
+                    setup_costs[m] = self.env.setup_costs[m][action[m] - 1] 
+                    setup_loss[m] = self.env.setup_loss[m][action[m] - 1]
+                machine_setup[m] = action[m]
+                # 2. PRODUCTION
+                production = self.env.machine_production_matrix[m][action[m] - 1] - setup_loss[m]
+                inventory_level[action[m] - 1] += production
+            else:
+                machine_setup[m] = 0
+        # return the new machine_setup_inventory_level and the setup_cost        
+        return machine_setup, inventory_level, setup_costs
+
+    def get_action(self, state):
+        state = torch.from_numpy(state).float().unsqueeze(0)
+        probs = self.policy(state)
+        probs_concat = torch.stack(probs, dim=1)
+        m = Categorical(probs_concat)
+        action = m.sample()
+        value = self.value(state)
+        machine_setup, inventory_level, setup_cost = self.get_post_state(action.numpy()[0], state[0][self.env.n_items:self.env.n_items+self.env.n_machines].numpy(), state[0][0:self.env.n_items].numpy())
+        value_post = self.value_post(state)
+        
+        return action, m.log_prob(action), probs_concat, value, value_post
+    
+    
+    def update(self, rewards, rewards_pre_state, rewards_post_state, states, post_states, actions, probs, next_states):
+        # Update deterministic value function
+        for epoch in range(10):
+            for i in range(len(actions)):
+                state = torch.from_numpy(states[i]).float().unsqueeze(0)
+                value = self.value(state)
+                next_state = torch.from_numpy(next_states[i]).float().unsqueeze(0)
+                next_value = self.value(next_state)
+                target = rewards_pre_state[i] + self.gamma * next_value
+                advantage = target - value
+                loss = advantage.pow(2).mean()
+                self.optimizer_value.zero_grad()
+                loss.backward()
+                self.optimizer_value.step()
+
+        # Update stochastic value function
+        for epoch in range(10):
+            for i in range(len(actions)):
+                state = torch.from_numpy(states[i]).float().unsqueeze(0)
+                value = self.value_post(state)
+                post_state = torch.from_numpy(post_states[i]).float().unsqueeze(0)
+                value_post = self.value_post(post_state)
+                target = rewards_post_state[i] + self.gamma * value_post
+                advantage = target - value
+                loss = advantage.pow(2).mean()
+                self.optimizer_value_post.zero_grad()
+                loss.backward()
+                self.optimizer_value_post.step()
+
+        # Update policy network
+        states = torch.from_numpy(np.vstack(states)).float()
+        actions = torch.cat(actions).unsqueeze(1)
+        old_probs = torch.cat(probs)
+        old_probs = torch.gather(old_probs.clone(),2, actions)
+
+        policy_epochs = 10
+        for epoch in range(policy_epochs):
+            probs = self.policy(states)
+            probs = torch.stack(probs, dim=1).clone()
+            m = Categorical(probs)
+            action = m.sample()
+            probs = torch.gather(probs, 2, actions)
+            kl_div = (old_probs * (torch.log(old_probs) - torch.log(probs))).sum()
+
+            for state,post_state, action, old_prob, prob, next_state, reward_pre_state, reward_post_state in zip(states,post_states, actions, old_probs, probs, next_states,rewards_pre_state,rewards_post_state):
+                state = state.unsqueeze(0)
+                next_state = torch.from_numpy(next_state).unsqueeze(0).float()
+                post_state = torch.from_numpy(post_state).unsqueeze(0).float()
+                action = action.unsqueeze(0)
+                old_prob = old_prob.unsqueeze(0)
+                prob = prob.unsqueeze(0)
+                value = self.value(state)
+                value_post = self.value_post(post_state)
+                advantage = reward_pre_state + self.gamma * self.value(next_state) - self.value(state)
+                advantage_post = reward_post_state + self.gamma * self.value_post(post_state) - self.value_post(state)
+                
+                ratio = (prob / old_prob)
+                surr1 = ratio * advantage
+                surr2 = torch.clamp(ratio, 1 - self.eps_clip, 1 + self.eps_clip) * advantage
+                policy_loss = -torch.min(surr1, surr2) - 0.01 * m.entropy()
+
+                ratio_post = ratio
+                surr1_post = ratio_post * advantage_post
+                surr2_post = torch.clamp(ratio_post, 1 - self.eps_clip, 1 + self.eps_clip) * advantage_post
+                policy_loss_post = -torch.min(surr1_post, surr2_post) - 0.01 * m.entropy()
+
+                self.optimizer_policy.zero_grad()
+                (policy_loss.pow(2).mean() + policy_loss_post.pow(2).mean() + 0.5 * value.pow(2).mean() + 0.5 * value_post.pow(2).mean()).backward(retain_graph=True)
+                self.optimizer_policy.step()
+    
+    def learn(self, n_episodes=1000, save_interval=100):
+        # Train the agent
+        for episode in range(n_episodes):
+            state = self.env.reset()
+            rewards = []
+            rewards_pre_state = []
+            rewards_post_state = []
+            states = []
+            next_states = []
+            actions = []
+            probs = []
+            post_states = []
+            # next_post_states = []
+            done = False
+            while not done:
+                action, log_prob, prob, value, value_post = self.get_action(state)
+                next_state, reward, done, info = self.env.step(action[0].detach().numpy())
+                machine_setup, inventory_level, setup_cost = self.get_post_state(action[0].detach().numpy(), state[self.env.n_items:self.env.n_items+self.env.n_machines], state[0:self.env.n_items])
+                post_state = state.copy()
+                post_state[self.env.n_items:self.env.n_items+self.env.n_machines] = machine_setup
+                post_state[0:self.env.n_items] = inventory_level
+                post_states.append(post_state)
+                post_state = torch.from_numpy(post_state).float().unsqueeze(0)
+                rewards.append(reward)
+                reward_pre_state = -(self.env.total_cost['holding_costs'] + self.env.total_cost['lost_sales'])
+                reward_post_state = -setup_cost.sum()
+                rewards_pre_state.append(reward_pre_state)
+                rewards_post_state.append(reward_post_state)
+                states.append(state)
+                next_states.append(next_state)
+                actions.append(action)
+                probs.append(prob)
+                
+                state = next_state
+                if done:
+                    self.update(rewards, rewards_pre_state, rewards_post_state, states, post_states, actions, probs, next_states)
+                    print('Episode:', episode, 'Reward:', sum(rewards))
+                    if episode % save_interval == 0:
+                        self.save(f'policy_{episode}.pt')            
+            self.save(self.LOG_DIR)
+    
+    
+    def save(self, filepath):
+        torch.save({
+            'policy_state_dict': self.policy.state_dict(),
+            'value_state_dict': self.value.state_dict(),
+            'value_post_state_dict': self.value_post.state_dict(),
+            'optimizer_policy_state_dict': self.optimizer_policy.state_dict(),
+            'optimizer_value_state_dict': self.optimizer_value.state_dict(),
+            'optimizer_value_post_state_dict': self.optimizer_value_post.state_dict()
+        }, filepath)
+
+    
+    
+    def load(self, filepath):
+        checkpoint = torch.load(filepath)
+        self.policy.load_state_dict(checkpoint['policy_state_dict'])
+        self.value.load_state_dict(checkpoint['value_state_dict'])
+        self.value_post.load_state_dict(checkpoint['value_post_state_dict'])
+        self.optimizer_policy.load_state_dict(checkpoint['optimizer_policy_state_dict'])
+        self.optimizer_value.load_state_dict(checkpoint['optimizer_value_state_dict'])
+        self.optimizer_value_post.load_state_dict(checkpoint['optimizer_value_post_state_dict'])
+    
+    

code/Lake application/agents/PPO.py

@@ -0,0 +1,248 @@
+# -*- coding: utf-8 -*-
+"""
+Created on Wed Mar  1 00:43:49 2023
+
+@author: leona
+"""
+
+import torch
+import torch.nn as nn
+from torch.distributions import MultivariateNormal
+from torch.distributions import Categorical
+
+################################## set device ##################################
+print("============================================================================================")
+# set device to cpu or cuda
+device = torch.device('cpu')
+if(torch.cuda.is_available()): 
+    device = torch.device('cuda:0') 
+    torch.cuda.empty_cache()
+    print("Device set to : " + str(torch.cuda.get_device_name(device)))
+else:
+    print("Device set to : cpu")
+print("============================================================================================")
+
+
+################################## PPO Policy ##################################
+class RolloutBuffer:
+    def __init__(self):
+        self.actions = []
+        self.states = []
+        self.logprobs = []
+        self.rewards = []
+        self.state_values = []
+        self.is_terminals = []
+    
+    def clear(self):
+        del self.actions[:]
+        del self.states[:]
+        del self.logprobs[:]
+        del self.rewards[:]
+        del self.state_values[:]
+        del self.is_terminals[:]
+
+
+class ActorCritic(nn.Module):
+    def __init__(self, state_dim, action_dim, has_continuous_action_space, action_std_init):
+        super(ActorCritic, self).__init__()
+
+        self.has_continuous_action_space = has_continuous_action_space
+        self.action_dim = action_dim
+
+        if has_continuous_action_space:
+            self.action_dim = action_dim
+            self.action_var = torch.full((action_dim,), action_std_init * action_std_init).to(device)
+        # actor
+        if has_continuous_action_space :
+            self.actor = nn.Sequential(
+                            nn.Linear(state_dim, 64),
+                            nn.Tanh(),
+                            nn.Linear(64, 64),
+                            nn.Tanh(),
+                            nn.Linear(64, action_dim),
+                            nn.Tanh()
+                        )
+        else:
+            
+            self.actor = nn.Sequential(
+                            nn.Linear(state_dim, 128),
+                            nn.Tanh(),
+                            nn.Linear(128, 128),
+                            nn.Tanh(),
+                            nn.Linear(128, action_dim)
+                        )
+            
+        # critic
+        self.critic = nn.Sequential(
+                        nn.Linear(state_dim, 128),
+                        nn.Tanh(),
+                        nn.Linear(128, 128),
+                        nn.Tanh(),
+                        nn.Linear(128, 1)
+                    )
+    
+    def forward(self, state):
+        raise NotImplementedError
+    
+    
+    
+    def set_action_std(self, new_action_std):
+        if self.has_continuous_action_space:
+            self.action_var = torch.full((self.action_dim,), new_action_std * new_action_std).to(device)
+        else:
+            print("--------------------------------------------------------------------------------------------")
+            print("WARNING : Calling ActorCritic::set_action_std() on discrete action space policy")
+            print("--------------------------------------------------------------------------------------------")
+
+    def act(self, state):
+
+        logits = self.actor(state)
+        action_probs = nn.functional.softmax(logits, dim=-1)
+        dist = Categorical(action_probs)
+
+        action = dist.sample()
+        action_logprob = dist.log_prob(action)
+        state_val = self.critic(state)
+
+        return action.detach(), action_logprob.detach(), state_val.detach()
+    
+    def evaluate(self, state, action):
+
+        logits = self.actor(state)
+        action_probs = nn.functional.softmax(logits, dim=-1)
+        dist = Categorical(action_probs)
+
+        action_logprobs = dist.log_prob(action.T).T
+        dist_entropy = dist.entropy()
+        state_values = self.critic(state)
+        
+        return action_logprobs, state_values, dist_entropy
+
+
+class PPO:
+    def __init__(self, state_dim, action_dim, lr_actor, lr_critic, gamma, K_epochs, eps_clip, has_continuous_action_space, action_std_init=0.6):
+
+        self.has_continuous_action_space = has_continuous_action_space
+
+        if has_continuous_action_space:
+            self.action_std = action_std_init
+
+        self.gamma = gamma
+        self.eps_clip = eps_clip
+        self.K_epochs = K_epochs
+        
+        self.buffer = RolloutBuffer()
+
+        self.policy = ActorCritic(state_dim, action_dim, has_continuous_action_space, action_std_init).to(device)
+        self.optimizer = torch.optim.Adam([
+                        {'params': self.policy.actor.parameters(), 'lr': lr_actor},
+                        {'params': self.policy.critic.parameters(), 'lr': lr_critic}
+                    ])
+
+        self.policy_old = ActorCritic(state_dim, action_dim, has_continuous_action_space, action_std_init).to(device)
+        self.policy_old.load_state_dict(self.policy.state_dict())
+        
+        self.MseLoss = nn.MSELoss()
+
+    def set_action_std(self, new_action_std):
+        if self.has_continuous_action_space:
+            self.action_std = new_action_std
+            self.policy.set_action_std(new_action_std)
+            self.policy_old.set_action_std(new_action_std)
+        else:
+            print("--------------------------------------------------------------------------------------------")
+            print("WARNING : Calling PPO::set_action_std() on discrete action space policy")
+            print("--------------------------------------------------------------------------------------------")
+
+    def decay_action_std(self, action_std_decay_rate, min_action_std):
+        print("--------------------------------------------------------------------------------------------")
+        if self.has_continuous_action_space:
+            self.action_std = self.action_std - action_std_decay_rate
+            self.action_std = round(self.action_std, 4)
+            if (self.action_std <= min_action_std):
+                self.action_std = min_action_std
+                print("setting actor output action_std to min_action_std : ", self.action_std)
+            else:
+                print("setting actor output action_std to : ", self.action_std)
+            self.set_action_std(self.action_std)
+
+        else:
+            print("WARNING : Calling PPO::decay_action_std() on discrete action space policy")
+        print("--------------------------------------------------------------------------------------------")
+
+    def select_action(self, state):
+
+        with torch.no_grad():
+            state = torch.tensor(state).to(device)
+            state = state.float() 
+            state = torch.unsqueeze(state, 1).T
+            action, action_logprob, state_val = self.policy_old.act(state)
+        
+        self.buffer.states.append(state)
+        self.buffer.actions.append(action)
+        self.buffer.logprobs.append(action_logprob)
+        self.buffer.state_values.append(state_val)
+
+        return action.cpu().numpy()
+    
+    def update(self):
+        # Monte Carlo estimate of returns
+        rewards = []
+        discounted_reward = 0
+        for reward, is_terminal in zip(reversed(self.buffer.rewards), reversed(self.buffer.is_terminals)):
+            if is_terminal:
+                discounted_reward = 0
+            discounted_reward = reward + (self.gamma * discounted_reward)
+            rewards.insert(0, discounted_reward)
+            
+        # Normalizing the rewards
+        rewards = torch.tensor(rewards, dtype=torch.float32).to(device)
+        rewards = (rewards - rewards.mean()) / (rewards.std() + 1e-7)
+
+        # convert list to tensor
+        old_states = torch.squeeze(torch.stack(self.buffer.states, dim=1)).detach().to(device)
+        old_actions = torch.stack(self.buffer.actions, dim=0).detach().to(device)
+        old_logprobs = torch.stack(self.buffer.logprobs, dim=0).detach().to(device)
+        old_state_values = torch.squeeze(torch.stack(self.buffer.state_values, dim=0)).detach().to(device)
+
+        # calculate advantages
+        advantages = rewards.detach() - old_state_values.detach()
+
+        # Optimize policy for K epochs
+        for _ in range(self.K_epochs):
+
+            # Evaluating old actions and values
+            logprobs, state_values, dist_entropy = self.policy.evaluate(old_states, old_actions)
+
+            # match state_values tensor dimensions with rewards tensor
+            state_values = torch.squeeze(state_values)
+            
+            # Finding the ratio (pi_theta / pi_theta__old)
+            ratios = torch.exp(logprobs - old_logprobs.detach())
+
+            # Finding Surrogate Loss  
+            surr1 = ratios * advantages.unsqueeze(1)
+            surr2 = torch.clamp(ratios, 1-self.eps_clip, 1+self.eps_clip) * advantages.unsqueeze(1)
+
+            # final loss of clipped objective PPO
+            loss = -torch.min(surr1, surr2) + 0.5 * self.MseLoss(state_values, rewards) - 0.012 * dist_entropy
+            
+            loss_numpy = loss.detach().cpu().numpy()
+            
+            # take gradient step
+            self.optimizer.zero_grad()
+            loss.mean().backward()
+            self.optimizer.step()
+            
+        # Copy new weights into old policy
+        self.policy_old.load_state_dict(self.policy.state_dict())
+
+        # clear buffer
+        self.buffer.clear()
+    
+    def save(self, checkpoint_path):
+        torch.save(self.policy_old.state_dict(), checkpoint_path)
+   
+    def load(self, checkpoint_path):
+        self.policy_old.load_state_dict(torch.load(checkpoint_path, map_location=lambda storage, loc: storage))
+        self.policy.load_state_dict(torch.load(checkpoint_path, map_location=lambda storage, loc: storage))

code/Lake application/agents/__init__.py

@@ -0,0 +1,8 @@
+from .PPOAgent import PPOAgent
+from .PDPPOAgent import PDPPOAgent
+
+
+__all__ = [
+    "PPOAgent",
+    "PDPPOAgent"
+]

code/Lake application/envs/frozen_lake.py

@@ -0,0 +1,301 @@
+import sys
+from contextlib import closing
+
+import numpy as np
+from io import StringIO
+
+from gym import utils
+from gym.envs.toy_text import discrete
+
+LEFT = 0
+DOWN = 1
+RIGHT = 2
+UP = 3
+
+MAPS = {
+    "4x4": ["SFFF", "FHFH", "FFFH", "HFFG"],
+    "8x8": [
+        "SFFFFFFF",
+        "FFFFFFFF",
+        "FFFHFFFF",
+        "FFFFFHFF",
+        "FFFHFFFF",
+        "FHHFFFHF",
+        "FHFFHFHF",
+        "FFFHFFFG",
+    ],
+}
+
+
+def generate_random_map(size=8, p=0.8):
+    """Generates a random valid map (one that has a path from start to goal)
+    :param size: size of each side of the grid
+    :param p: probability that a tile is frozen
+    """
+    valid = False
+
+    # DFS to check that it's a valid path.
+    def is_valid(res):
+        frontier, discovered = [], set()
+        frontier.append((0, 0))
+        while frontier:
+            r, c = frontier.pop()
+            if not (r, c) in discovered:
+                discovered.add((r, c))
+                directions = [(1, 0), (0, 1), (-1, 0), (0, -1)]
+                for x, y in directions:
+                    r_new = r + x
+                    c_new = c + y
+                    if r_new < 0 or r_new >= size or c_new < 0 or c_new >= size:
+                        continue
+                    if res[r_new][c_new] == "G":
+                        return True
+                    if res[r_new][c_new] != "H":
+                        frontier.append((r_new, c_new))
+        return False
+
+    while not valid:
+        p = min(1, p)
+        res = np.random.choice(["F", "H"], (size, size), p=[p, 1 - p])
+        res[0][0] = "S"
+        res[-1][-1] = "G"
+        valid = is_valid(res)
+    return ["".join(x) for x in res]
+
+
+class FrozenLakeEnv(discrete.DiscreteEnv):
+    """
+    Winter is here. You and your friends were tossing around a frisbee at the
+    park when you made a wild throw that left the frisbee out in the middle of
+    the lake. The water is mostly frozen, but there are a few holes where the
+    ice has melted. If you step into one of those holes, you'll fall into the
+    freezing water. At this time, there's an international frisbee shortage, so
+    it's absolutely imperative that you navigate across the lake and retrieve
+    the disc. However, the ice is slippery, so you won't always move in the
+    direction you intend.
+    The surface is described using a grid like the following
+
+        SFFF
+        FHFH
+        FFFH
+        HFFG
+
+    S : starting point, safe
+    F : frozen surface, safe
+    H : hole, fall to your doom
+    G : goal, where the frisbee is located
+
+    The episode ends when you reach the goal or fall in a hole.
+    You receive a reward of 1 if you reach the goal, and zero otherwise.
+    """
+
+    metadata = {"render.modes": ["human", "ansi"]}
+
+    def __init__(self, desc=None, map_name="4x4", is_slippery=True):
+        if desc is None and map_name is None:
+            desc = generate_random_map()
+        elif desc is None:
+            desc = MAPS[map_name]
+        self.desc = desc = np.asarray(desc, dtype="c")
+        self.nrow, self.ncol = nrow, ncol = desc.shape
+        self.reward_range = (0, 1)
+
+        nA = 4
+        nS = nrow * ncol
+
+        isd = np.array(desc == b"S").astype("float64").ravel()
+        isd /= isd.sum()
+
+        P = {s: {a: [] for a in range(nA)} for s in range(nS)}
+
+        def to_s(row, col):
+            return row * ncol + col
+
+        def inc(row, col, a):
+            if a == LEFT:
+                col = max(col - 1, 0)
+            elif a == DOWN:
+                row = min(row + 1, nrow - 1)
+            elif a == RIGHT:
+                col = min(col + 1, ncol - 1)
+            elif a == UP:
+                row = max(row - 1, 0)
+            return (row, col)
+        
+        goal_position = None
+        for row in range(nrow):
+            for col in range(ncol):
+                if desc[row, col] == b'G':
+                    goal_position = (row, col)
+                    break
+            if goal_position:
+                break
+
+        def proximity_reward(current_row, current_col):
+            goal_row, goal_col = goal_position
+            distance = abs(goal_row - current_row) + abs(goal_col - current_col)
+            return 1.0 / (1.0 + distance)
+
+        def update_probability_matrix(row, col, a):
+            newrow, newcol = inc(row, col, a)
+            newstate = to_s(newrow, newcol)
+            newletter = desc[newrow, newcol]
+            terminated = bytes(newletter) in b"GH"
+            reward = float(newletter == b"G")
+            if not terminated:
+                reward = proximity_reward(newrow, newcol)
+            return newstate, reward, terminated
+
+        # def update_probability_matrix(row, col, action):
+        #     newrow, newcol = inc(row, col, action)
+        #     newstate = to_s(newrow, newcol)
+        #     newletter = desc[newrow, newcol]
+        #     done = bytes(newletter) in b"GH"
+        #     reward = float(newletter == b"G")
+        #     return newstate, reward, done
+
+        # for row in range(nrow):
+        #     for col in range(ncol):
+        #         s = to_s(row, col)
+        #         for a in range(4):
+        #             li = P[s][a]
+        #             letter = desc[row, col]
+        #             if letter in b"GH":
+        #                 li.append((1.0, s, 0, True))
+        #             else:
+        #                 if is_slippery:
+        #                     for b in [(a - 1) % 4, a, (a + 1) % 4]:
+        #                         li.append(
+        #                             (1.0 / 3.0, *update_probability_matrix(row, col, b))
+        #                         )
+        #                 else:
+        #                     li.append((1.0, *update_probability_matrix(row, col, a)))
+        
+        np.random.seed(42)  # Set a seed for reproducibility
+        tile_probabilities = np.random.dirichlet(np.ones(4), size=(nrow, ncol))
+
+        def to_row_col(s):
+            return divmod(s, ncol)
+
+        # for row in range(nrow):
+        #     for col in range(ncol):
+        #         s = to_s(row, col)
+        #         for a in range(4):
+        #             li = P[s][a]
+        #             letter = desc[row, col]
+        #             if letter in b"GH":
+        #                 li.append((1.0, s, 0, True))
+        #             else:
+        #                 if is_slippery:
+        #                     # First, the agent moves in the desired direction
+        #                     newstate, reward, terminated = update_probability_matrix(row, col, a)
+        #                     if terminated:
+        #                         li.append((1.0, newstate, reward, terminated))
+        #                     else:
+        #                         # After the first move, slippery condition causes a random additional movement
+        #                         row2, col2 = to_row_col(newstate)
+        #                         for b in range(4):
+        #                             li.append(
+        #                                 (1.0 / 4.0, *update_probability_matrix(row2, col2, b))
+        #                             )
+        #                 else:
+        #                     li.append((1.0, *update_probability_matrix(row, col, a)))
+
+        # for row in range(nrow):
+        #     for col in range(ncol):
+        #         s = to_s(row, col)
+        #         for a in range(4):
+        #             li = P[s][a]
+        #             letter = desc[row, col]
+        #             if letter in b"GH":
+        #                 li.append((1.0, s, 0, True))
+        #             else:
+        #                 if is_slippery:
+        #                     # First, the agent moves in the desired direction
+        #                     newstate, reward, terminated = update_probability_matrix(row, col, a)
+        #                     if terminated:
+        #                         li.append((1.0, newstate, reward, terminated))
+        #                     else:
+        #                         # After the first move, slippery condition causes an additional movement
+        #                         row2, col2 = to_row_col(newstate)
+        #                         for b, prob in enumerate(tile_probabilities[row2, col2]):
+        #                             li.append(
+        #                                 (prob, *update_probability_matrix(row2, col2, b))
+        #                             )
+        #                 else:
+        #                     li.append((1.0, *update_probability_matrix(row, col, a)))
+
+        base_slip_prob=0.3
+        
+        for row in range(nrow):
+            for col in range(ncol):
+                s = to_s(row, col)
+                for a in range(4):
+                    li = P[s][a]
+                    letter = desc[row, col]
+                    if letter in b"GH":
+                        li.append((1.0, s, 0, True))
+                    else:
+                        if is_slippery:
+                            # First, the agent moves in the desired direction
+                            newstate, reward, terminated = update_probability_matrix(row, col, a)
+                            if terminated:
+                                li.append((1.0, newstate, reward, terminated))
+                            else:
+                                # After the first move, slippery condition causes an additional movement
+                                row2, col2 = to_row_col(newstate)
+                                for b, prob in enumerate(tile_probabilities[row2, col2]):
+                                    li.append(
+                                        (base_slip_prob * prob, *update_probability_matrix(row2, col2, b))
+                                    )
+                                # Add the remaining probability for staying at the newstate
+                                li.append((1.0 - base_slip_prob, newstate, reward, False))
+                        else:
+                            li.append((1.0, *update_probability_matrix(row, col, a)))
+
+
+        super(FrozenLakeEnv, self).__init__(nS, nA, P, isd)
+
+    def get_post_decision_state(self, s, a):
+            def inc(row, col, a):
+                if a == LEFT:
+                    col = max(col - 1, 0)
+                elif a == DOWN:
+                    row = min(row + 1, self.nrow - 1)
+                elif a == RIGHT:
+                    col = min(col + 1, self.ncol - 1)
+                elif a == UP:
+                    row = max(row - 1, 0)
+                return (row, col)
+
+            def to_s(row, col):
+                return row * self.ncol + col
+
+            def to_row_col(s):
+                row = s // self.ncol
+                col = s % self.ncol
+                return row, col
+
+            row, col = to_row_col(s)
+            next_row, next_col = inc(row, col, a)
+            next_s = to_s(next_row, next_col)
+            return next_s
+    
+    def render(self, mode="human"):
+        outfile = StringIO() if mode == "ansi" else sys.stdout
+
+        row, col = self.s // self.ncol, self.s % self.ncol
+        desc = self.desc.tolist()
+        desc = [[c.decode("utf-8") for c in line] for line in desc]
+        desc[row][col] = utils.colorize(desc[row][col], "red", highlight=True)
+        if self.lastaction is not None:
+            outfile.write(
+                "  ({})\n".format(["Left", "Down", "Right", "Up"][self.lastaction])
+            )
+        else:
+            outfile.write("\n")
+        outfile.write("\n".join("".join(line) for line in desc) + "\n")
+
+        if mode != "human":
+            with closing(outfile):
+                return outfile.getvalue()

code/Lake application/experiments.py

@@ -0,0 +1,117 @@
+# 
+# -*- coding: utf-8 -*-
+import os
+import gc
+import sys
+import json
+import random
+import gym
+
+BASE_DIR = os.path.dirname(os.path.abspath('__file__'))
+AGENTS_DIR = os.path.join(BASE_DIR,'agents')
+sys.path.append(AGENTS_DIR)
+
+from agents.PPO import PPO
+from agents.PDPPO import PDPPO
+
+import numpy as np
+from agents import *
+from agents import StochasticProgrammingAgent, AdpAgentHD3
+from agents import StableBaselineAgent, MultiAgentRL, EnsembleAgent, PerfectInfoAgent,PSOagent,AdpAgentHD, PPOAgent
+from test_functions import *
+from scenarioManager.stochasticDemandModel import StochasticDemandModel
+
+
+#'15items_5machines_i100','25items_10machines'
+
+if __name__ == '__main__':
+    for i in range(0,5):
+        
+        experiment_name = 'frozen_lake'
+        
+        # Setting the seeds
+        np.random.seed(1)
+        random.seed(10)
+        
+        from gym.envs.toy_text.frozen_lake import generate_random_map
+
+        # Models setups:
+        env = gym.make('FrozenLake-v1', desc=generate_random_map(size=8), is_slippery=True)
+
+        setting_sol_method = {
+            'discount_rate': 0.99,
+            'experiment_name': experiment_name,
+            'parallelization': False,
+            'model_name': 'PPO',
+            'branching_factors': [4, 2, 2],
+            'dict_obs': False # To be employed if dictionary observations are necessary
+        }
+        # Parameters for the RL:
+        
+        training_epochs_RL = 200000
+        
+        setting_sol_method['parallelization'] = False
+        
+        # Number of test execution (number of complet environment iterations)
+        nreps = 100
+        
+        ###########################################################################
+        # #PPO
+        ###########################################################################
+        
+        base_model_name = 'PPO'
+        ppo_agent = PPOAgent(
+            env,
+            setting_sol_method
+        )
+        ppo_agent.learn(n_episodes=training_epochs_RL) # Each ep with 200 steps
+        
+        #load best agent before appending in the test list
+        BEST_MODEL_DIR = os.path.join(os.path.dirname(os.path.abspath('__file__')),'logs',f'best_{base_model_name}_{experiment_name}','best_model')
+        ppo_agent.load_agent(BEST_MODEL_DIR) # For training purposes
+
+        
+        
+        ###########################################################################
+        # Post-decision PPO
+        ###########################################################################
+        
+        base_model_name = 'PDPPO'
+        pdppo_agent = PDPPOAgent(
+            env,
+            setting_sol_method
+        )
+        pdppo_agent.learn(n_episodes=training_epochs_RL) # Each ep with 200 steps
+        
+        #load best agent before appending in the test list
+        BEST_MODEL_DIR = os.path.join(os.path.dirname(os.path.abspath('__file__')),'logs',f'best_{base_model_name}_{experiment_name}','best_model')
+        pdppo_agent.load_agent(BEST_MODEL_DIR) # For training purposes
+        
+        
+        ###########################################################################
+        #TESTING
+        # settings['dict_obs'] = False
+        # setting_sol_method['multiagent'] = False
+        # setting_sol_method['dict_obs'] = False
+        # env = SimplePlant(settings, stoch_model)
+        # setting_sol_method['experiment_name'] = experiment_name
+        # dict_res = test_agents(
+        #     env,
+        #     agents=agents,
+        #     n_reps=nreps,
+        #     setting_sol_method = setting_sol_method,
+        #     use_benchmark_PI=False
+        # )
+        
+        # for key,_ in agents:
+        #     cost = dict_res[key,'costs']
+        #     print(f'\n Cost in {nreps} iterations for the model {key}: {cost}')
+        # try:
+        #     cost = dict_res['PI','costs']
+        #     print(f'\n Cost in {nreps} repetitions for the model PI: {cost}')
+        # except:
+        #     pass
+                
+        #del multiagent
+        del env
+        gc.collect()

code/Lake application/generate_tables.py

@@ -0,0 +1,78 @@
+import os
+import pandas as pd
+import numpy as np
+import matplotlib.pyplot as plt
+import seaborn as sns
+
+main_folder = 'logs/results_2'
+
+def get_max_rewards():
+    experiment_names = ['frozen_lake']
+    methods = ['PDPPO', 'PPO']
+    results = pd.DataFrame(columns=['Environment', 'Method', 'Max Reward', 'Max Reward Standard Deviation'])
+    main_folder = 'logs/results_2'
+    for experiment_name in experiment_names:
+        for method in methods:
+            env_name = experiment_name
+            max_rewards = []
+            for run_num in range(1, 6):
+                log_f_name = f'{main_folder}/{method}_{env_name}_log_{run_num}.csv'
+                data = pd.read_csv(log_f_name)
+                max_reward = data['reward'].max()
+                max_rewards.append(max_reward)
+            mean_max_reward = np.mean(max_rewards)
+            std_max_reward = np.std(max_rewards)
+            results = results.append({'Environment': env_name, 'Method': method, 'Max Reward': mean_max_reward, 'Max Reward Standard Deviation': std_max_reward}, ignore_index=True)
+
+    return results
+    
+def get_first_rewards():
+    experiment_names = ['frozen_lake']
+    methods = ['PDPPO', 'PPO']
+    results = pd.DataFrame(columns=['Environment', 'Method', 'First Reward', 'First Reward Standard Deviation'])
+
+    for experiment_name in experiment_names:
+        for method in methods:
+            env_name = experiment_name
+            first_rewards = []
+            for run_num in range(1, 6):
+                log_f_name = f'{main_folder}/{method}_{env_name}_log_{run_num}.csv'
+                data = pd.read_csv(log_f_name)
+                reward_50000 = data[data['timestep'] == 50000]['reward'].values
+                first_rewards.append(reward_50000)
+            mean_first_reward = np.mean(first_rewards)
+            std_first_reward = np.std(first_rewards)
+            results = results.append({'Environment': env_name, 'Method': method, 'First Reward': mean_first_reward, 'First Reward Standard Deviation': std_first_reward}, ignore_index=True)
+
+    return results
+
+def get_steps_reward_threshold():
+    experiment_names = ['frozen_lake']
+    methods = ['PDPPO', 'PPO']
+    reward_thresholds = [10]
+    results = pd.DataFrame(columns=['Environment', 'Method', 'Steps', 'Steps Standard Deviation'])
+
+    for i, experiment_name in enumerate(experiment_names):
+        for j, method in enumerate(methods):
+            env_name = experiment_name
+            reward_steps = []
+            for run_num in range(1, 6):
+                log_f_name = f'logs/results_1/{method}_{env_name}_log_{run_num}.csv'
+                data = pd.read_csv(log_f_name)
+                reward_threshold_value = reward_thresholds[i]
+                reward_steps.append(data[data['reward'] >= reward_threshold_value]['timestep'].iloc[0])
+            mean_reward_steps = np.mean(reward_steps) if reward_steps else np.nan
+            std_reward_steps = np.std(reward_steps) if reward_steps else np.nan
+            results = results.append({'Environment': env_name, 'Method': method, 'Steps': mean_reward_steps, 'Steps Standard Deviation': std_reward_steps}, ignore_index=True)
+
+    return results
+    
+
+if __name__ == '__main__':
+    max_rewards_df = get_max_rewards()
+    first_rewards_df = get_first_rewards()
+    steps_rewards_df = get_steps_reward_threshold()
+
+    final_results = pd.merge(max_rewards_df, first_rewards_df, on=['Environment', 'Method'])
+    final_results = pd.merge(final_results, steps_rewards_df, on=['Environment', 'Method'])
+    print(final_results)

code/Lake application/logs/frozen_lake_PDPPO/PDPPO_frozen_lake_0_0.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:e1c9e2a71c5d15566400c1716b2450770bb0b1e02e290ba4d6a0e6d866e97a1a
+size 307271

code/Lake application/logs/frozen_lake_PDPPO/PDPPO_frozen_lake_log_1.csv

@@ -0,0 +1,501 @@
+episode,timestep,reward
+22,400,1.2526
+48,800,1.1762
+66,1200,1.698
+86,1600,1.5699
+100,2000,2.2196
+114,2400,2.2243
+130,2800,1.9494
+144,3200,2.1269
+161,3600,1.8416
+173,4000,2.4108
+186,4400,2.3831
+195,4800,3.7798
+204,5200,3.3905
+212,5600,3.7893
+221,6000,2.6417
+230,6400,3.9088
+245,6800,2.3282
+256,7200,2.7405
+263,7600,4.6528
+273,8000,3.0146
+281,8400,3.2375
+291,8800,3.553
+297,9200,4.7911
+307,9600,3.4636
+314,10000,4.4748
+320,10400,4.6664
+329,10800,3.4853
+336,11200,4.7085
+341,11600,6.6859
+349,12000,3.6321
+355,12400,5.3428
+362,12800,4.1236
+367,13200,6.1173
+373,13600,5.3324
+377,14000,7.2656
+381,14400,7.7223
+386,14800,5.9847
+390,15200,7.6658
+395,15600,5.8334
+400,16000,7.1939
+404,16400,7.1478
+408,16800,7.5988
+414,17200,6.1732
+419,17600,6.1007
+423,18000,7.2673
+427,18400,7.41
+432,18800,7.0193
+438,19200,5.3807
+443,19600,5.9541
+447,20000,7.6451
+452,20400,7.0583
+457,20800,7.3468
+461,21200,7.6243
+466,21600,6.6661
+473,22000,4.9625
+479,22400,4.5857
+484,22800,6.9212
+488,23200,6.1686
+493,23600,7.897
+497,24000,7.9243
+501,24400,8.608
+506,24800,6.6952
+511,25200,6.2059
+516,25600,7.1384
+522,26000,5.7323
+527,26400,5.9145
+534,26800,5.6198
+538,27200,7.6158
+542,27600,7.7017
+546,28000,7.8191
+551,28400,8.2935
+556,28800,7.3137
+562,29200,5.8557
+570,29600,4.3915
+576,30000,4.9182
+581,30400,6.3954
+585,30800,7.2091
+590,31200,9.0975
+594,31600,5.9126
+600,32000,5.1805
+606,32400,6.9075
+611,32800,6.2059
+616,33200,4.6794
+621,33600,7.0498
+626,34000,7.4649
+631,34400,6.6183
+635,34800,8.8603
+639,35200,6.275
+643,35600,9.1126
+648,36000,6.9308
+653,36400,6.7373
+657,36800,7.6857
+662,37200,7.836
+666,37600,6.3725
+671,38000,6.2922
+676,38400,6.6979
+680,38800,7.6388
+684,39200,7.0079
+688,39600,7.5892
+692,40000,7.7355
+697,40400,6.5238
+702,40800,6.181
+706,41200,6.5753
+711,41600,6.499
+716,42000,6.1642
+721,42400,6.3709
+725,42800,6.5148
+730,43200,6.1071
+734,43600,7.2231
+739,44000,6.9884
+743,44400,7.6752
+747,44800,7.645
+751,45200,7.6253
+756,45600,6.9888
+760,46000,6.551
+764,46400,8.8765
+768,46800,7.644
+772,47200,7.7078
+776,47600,7.7402
+780,48000,7.7096
+786,48400,6.4542
+790,48800,6.8511
+795,49200,6.7727
+799,49600,7.6417
+803,50000,7.6993
+807,50400,7.7043
+812,50800,6.8261
+816,51200,6.92
+820,51600,7.9919
+825,52000,6.4103
+830,52400,5.7618
+834,52800,7.0424
+838,53200,7.6928
+842,53600,7.7354
+847,54000,7.3411
+852,54400,6.3078
+856,54800,7.4612
+860,55200,6.6696
+865,55600,6.7569
+869,56000,6.9279
+874,56400,6.5996
+878,56800,7.7271
+882,57200,7.6932
+886,57600,7.5902
+890,58000,7.1683
+897,58400,4.5812
+903,58800,6.1682
+908,59200,6.1325
+912,59600,7.8386
+916,60000,6.3943
+921,60400,6.8943
+925,60800,7.8821
+932,61200,4.9096
+937,61600,6.3275
+941,62000,8.5893
+945,62400,7.9071
+949,62800,7.8808
+953,63200,7.8324
+957,63600,8.5609
+961,64000,8.4952
+965,64400,7.8793
+971,64800,5.9811
+975,65200,6.3148
+979,65600,8.3047
+985,66000,6.6049
+991,66400,5.5465
+995,66800,6.7472
+1000,67200,6.688
+1004,67600,8.5059
+1008,68000,8.0416
+1012,68400,9.3594
+1017,68800,7.135
+1022,69200,7.7882
+1028,69600,5.1304
+1032,70000,7.9267
+1036,70400,7.9253
+1040,70800,7.522
+1045,71200,6.7979
+1050,71600,7.0769
+1055,72000,7.1814
+1059,72400,7.9156
+1063,72800,6.9876
+1069,73200,5.6939
+1073,73600,7.603
+1078,74000,6.9544
+1085,74400,5.1872
+1089,74800,8.0712
+1094,75200,5.4866
+1099,75600,6.856
+1104,76000,6.6695
+1111,76400,5.2366
+1115,76800,8.231
+1120,77200,6.3017
+1125,77600,6.1984
+1129,78000,8.0981
+1134,78400,5.6874
+1139,78800,6.7436
+1144,79200,6.7781
+1148,79600,8.035
+1153,80000,7.0241
+1157,80400,8.1216
+1161,80800,7.7847
+1165,81200,7.0458
+1172,81600,4.9159
+1176,82000,8.8252
+1180,82400,8.1435
+1184,82800,8.0794
+1188,83200,8.4439
+1194,83600,6.5179
+1198,84000,9.5129
+1202,84400,9.5982
+1206,84800,8.1605
+1213,85200,5.1735
+1218,85600,6.3955
+1222,86000,6.5836
+1227,86400,7.8827
+1231,86800,9.0583
+1236,87200,7.4464
+1242,87600,6.9449
+1247,88000,7.6785
+1251,88400,7.7551
+1255,88800,9.1958
+1260,89200,7.8329
+1265,89600,9.6998
+1269,90000,12.2941
+1274,90400,7.8743
+1281,90800,6.7261
+1285,91200,10.3406
+1289,91600,10.3629
+1294,92000,12.8416
+1298,92400,11.8637
+1302,92800,10.95
+1307,93200,12.6815
+1312,93600,10.2359
+1316,94000,14.5616
+1320,94400,12.2057
+1325,94800,10.3496
+1331,95200,10.0749
+1336,95600,12.5332
+1341,96000,7.9488
+1347,96400,11.6734
+1353,96800,8.6258
+1359,97200,8.6982
+1363,97600,16.8782
+1369,98000,13.076
+1374,98400,12.6862
+1380,98800,13.0069
+1385,99200,13.2064
+1394,99600,7.5732
+1399,100000,13.571
+1406,100400,11.8885
+1411,100800,14.0567
+1416,101200,14.4459
+1422,101600,14.7572
+1430,102000,9.0038
+1436,102400,11.7543
+1443,102800,12.3206
+1448,103200,13.6686
+1454,103600,14.6563
+1461,104000,10.5416
+1467,104400,11.8187
+1473,104800,12.6766
+1478,105200,13.234
+1487,105600,10.4377
+1494,106000,10.4682
+1499,106400,14.2136
+1505,106800,13.3214
+1512,107200,9.9131
+1518,107600,11.3795
+1524,108000,13.6525
+1531,108400,10.5166
+1536,108800,15.6816
+1541,109200,13.5686
+1546,109600,15.8877
+1552,110000,10.798
+1556,110400,15.505
+1564,110800,8.7457
+1571,111200,9.8442
+1576,111600,16.783
+1581,112000,16.6507
+1587,112400,12.5058
+1597,112800,7.0879
+1604,113200,9.4795
+1612,113600,8.8247
+1619,114000,9.8475
+1626,114400,9.1637
+1631,114800,14.0166
+1638,115200,8.0027
+1642,115600,17.5209
+1650,116000,7.7073
+1654,116400,20.1086
+1661,116800,7.9623
+1668,117200,12.2066
+1675,117600,8.7988
+1681,118000,14.56
+1686,118400,16.3382
+1691,118800,12.1992
+1699,119200,10.4266
+1703,119600,19.3562
+1708,120000,11.3536
+1712,120400,20.2356
+1716,120800,19.5323
+1721,121200,17.4551
+1728,121600,10.1801
+1732,122000,16.947
+1739,122400,9.827
+1745,122800,13.3383
+1749,123200,19.4348
+1755,123600,12.4996
+1761,124000,13.8883
+1765,124400,20.16
+1769,124800,19.6822
+1776,125200,10.0647
+1780,125600,18.9827
+1785,126000,10.8816
+1791,126400,12.6382
+1797,126800,11.2938
+1806,127200,8.8997
+1811,127600,15.7514
+1817,128000,9.8611
+1825,128400,10.0803
+1833,128800,9.5121
+1841,129200,7.1428
+1850,129600,8.5931
+1858,130000,9.1178
+1866,130400,10.6061
+1871,130800,15.7645
+1875,131200,19.0111
+1881,131600,10.6172
+1885,132000,20.6998
+1889,132400,20.2484
+1894,132800,16.4684
+1900,133200,13.5329
+1904,133600,20.9186
+1910,134000,15.0977
+1915,134400,14.7358
+1920,134800,15.5285
+1928,135200,8.3647
+1934,135600,11.9479
+1940,136000,14.4666
+1947,136400,9.6302
+1953,136800,13.6634
+1958,137200,14.0396
+1966,137600,9.1948
+1973,138000,9.3788
+1982,138400,9.0964
+1991,138800,7.105
+1998,139200,11.6642
+2005,139600,8.2024
+2014,140000,7.7126
+2021,140400,11.0422
+2027,140800,11.2108
+2032,141200,15.9589
+2039,141600,8.9692
+2046,142000,12.0016
+2052,142400,13.3516
+2056,142800,19.3885
+2061,143200,14.0276
+2066,143600,14.164
+2073,144000,11.3607
+2082,144400,6.5908
+2089,144800,10.36
+2094,145200,14.2824
+2105,145600,4.9849
+2111,146000,11.1523
+2116,146400,12.4602
+2123,146800,10.201
+2129,147200,11.5534
+2138,147600,5.9679
+2142,148000,16.4718
+2147,148400,17.5833
+2152,148800,16.768
+2156,149200,19.1331
+2162,149600,12.3424
+2166,150000,18.0845
+2170,150400,18.9557
+2175,150800,14.0846
+2179,151200,20.1474
+2184,151600,15.3493
+2188,152000,18.1315
+2193,152400,15.8044
+2197,152800,17.7751
+2203,153200,11.6984
+2209,153600,13.0837
+2213,154000,15.855
+2218,154400,16.8814
+2223,154800,16.0311
+2227,155200,15.3192
+2234,155600,11.457
+2239,156000,16.1934
+2246,156400,11.9881
+2252,156800,9.2926
+2257,157200,16.73
+2261,157600,14.6868
+2268,158000,7.9747
+2273,158400,10.7901
+2277,158800,18.3051
+2283,159200,14.0301
+2288,159600,15.5108
+2294,160000,10.0966
+2300,160400,8.0898
+2306,160800,8.0323
+2315,161200,5.836
+2321,161600,10.9128
+2329,162000,5.4973
+2334,162400,11.0091
+2340,162800,8.3576
+2349,163200,6.2928
+2354,163600,13.008
+2358,164000,15.7582
+2364,164400,10.6657
+2372,164800,7.3352
+2378,165200,7.9416
+2385,165600,9.7033
+2390,166000,14.8003
+2395,166400,13.5949
+2399,166800,16.5582
+2405,167200,13.1281
+2410,167600,13.6583
+2415,168000,13.3859
+2422,168400,12.7336
+2427,168800,13.4928
+2432,169200,14.222
+2438,169600,12.2052
+2442,170000,17.538
+2449,170400,13.4975
+2454,170800,13.4316
+2460,171200,14.3904
+2466,171600,14.1914
+2472,172000,10.7212
+2478,172400,13.2043
+2484,172800,12.3505
+2488,173200,18.4707
+2493,173600,13.7594
+2498,174000,15.2434
+2503,174400,17.6234
+2509,174800,11.5482
+2513,175200,18.5919
+2518,175600,18.2289
+2525,176000,10.1992
+2529,176400,19.2187
+2533,176800,20.1763
+2538,177200,15.8706
+2542,177600,20.1437
+2547,178000,14.6606
+2552,178400,16.5006
+2557,178800,17.2947
+2561,179200,20.4127
+2566,179600,16.8026
+2571,180000,16.4537
+2575,180400,19.0096
+2580,180800,12.4961
+2586,181200,13.702
+2590,181600,19.4121
+2595,182000,19.1734
+2599,182400,15.6252
+2604,182800,17.4543
+2608,183200,18.4527
+2613,183600,17.0079
+2617,184000,15.8574
+2623,184400,14.6486
+2627,184800,16.7431
+2632,185200,16.3204
+2636,185600,18.1816
+2640,186000,19.3138
+2646,186400,13.6693
+2652,186800,12.1288
+2656,187200,19.5385
+2662,187600,12.7194
+2666,188000,18.8978
+2671,188400,17.7266
+2675,188800,20.3572
+2680,189200,16.1169
+2685,189600,14.712
+2692,190000,11.1334
+2696,190400,17.4639
+2701,190800,16.8716
+2705,191200,19.8286
+2710,191600,18.7337
+2714,192000,19.8553
+2719,192400,14.2238
+2725,192800,12.8373
+2735,193200,7.0151
+2739,193600,13.3644
+2745,194000,13.3385
+2753,194400,8.739
+2758,194800,13.8917
+2764,195200,10.3893
+2773,195600,4.8033
+2784,196000,5.5534
+2792,196400,7.2659
+2796,196800,13.9818
+2802,197200,13.2211
+2807,197600,16.0152
+2814,198000,9.9562
+2823,198400,6.7525
+2828,198800,14.4238
+2833,199200,10.7252
+2838,199600,13.7153
+2842,200000,19.2825

code/Lake application/logs/frozen_lake_PDPPO/PDPPO_frozen_lake_log_2.csv

@@ -0,0 +1,501 @@
+episode,timestep,reward
+25,400,1.2001
+50,800,1.1974
+67,1200,1.7766
+87,1600,1.4873
+104,2000,1.7077
+114,2400,2.5908
+134,2800,1.8992
+145,3200,2.8642
+154,3600,2.9615
+163,4000,3.7629
+170,4400,4.1044
+180,4800,3.4985
+187,5200,4.3872
+192,5600,5.6852
+198,6000,5.7852
+205,6400,4.5497
+210,6800,5.1203
+215,7200,6.3622
+222,7600,4.9101
+230,8000,3.7216
+237,8400,3.9412
+243,8800,5.1797
+250,9200,4.9764
+255,9600,5.0597
+260,10000,6.6622
+266,10400,5.4034
+274,10800,3.7803
+279,11200,4.8601
+284,11600,6.8704
+289,12000,5.884
+298,12400,3.1085
+306,12800,3.7402
+317,13200,2.4474
+325,13600,4.697
+333,14000,3.7155
+340,14400,4.1133
+345,14800,6.3342
+354,15200,3.4713
+363,15600,2.973
+369,16000,5.3844
+376,16400,3.9942
+384,16800,4.3917
+390,17200,3.9032
+396,17600,5.5522
+402,18000,5.0959
+410,18400,4.0555
+419,18800,3.8592
+424,19200,5.5717
+430,19600,5.4533
+436,20000,4.4112
+442,20400,5.2485
+453,20800,3.6337
+458,21200,5.7439
+464,21600,5.8554
+470,22000,6.1356
+478,22400,4.1016
+483,22800,6.9287
+487,23200,7.9029
+492,23600,7.2121
+496,24000,6.8316
+501,24400,7.3779
+506,24800,7.0025
+511,25200,7.3224
+516,25600,6.0262
+521,26000,7.3493
+525,26400,8.5734
+530,26800,7.505
+536,27200,5.6146
+541,27600,7.32
+547,28000,5.8791
+551,28400,8.3691
+555,28800,8.5242
+560,29200,6.8016
+564,29600,6.4966
+570,30000,6.5884
+574,30400,7.9429
+579,30800,8.2591
+586,31200,5.1781
+590,31600,6.725
+595,32000,7.3339
+602,32400,5.5153
+606,32800,7.294
+611,33200,7.861
+617,33600,5.8614
+622,34000,6.2895
+626,34400,7.641
+631,34800,7.1765
+635,35200,8.3742
+639,35600,8.5644
+644,36000,7.5061
+648,36400,7.137
+654,36800,6.317
+659,37200,7.5171
+663,37600,8.4671
+667,38000,8.8055
+672,38400,7.2286
+677,38800,7.0709
+682,39200,6.0232
+687,39600,7.7556
+691,40000,7.7135
+695,40400,8.3451
+701,40800,6.0775
+705,41200,8.3363
+709,41600,8.6071
+715,42000,5.3694
+720,42400,7.6011
+724,42800,8.7117
+729,43200,7.0684
+735,43600,5.433
+739,44000,8.6411
+743,44400,8.5924
+747,44800,8.9168
+752,45200,7.2905
+756,45600,8.9831
+761,46000,7.4007
+766,46400,6.664
+770,46800,7.3928
+777,47200,5.7024
+783,47600,5.151
+788,48000,7.2108
+793,48400,7.2763
+799,48800,7.0493
+803,49200,8.5485
+808,49600,7.5421
+812,50000,9.5257
+816,50400,9.1144
+822,50800,5.6572
+828,51200,7.8776
+832,51600,10.0686
+838,52000,6.2858
+842,52400,8.7905
+847,52800,6.9064
+852,53200,8.591
+857,53600,7.9326
+861,54000,10.7555
+865,54400,11.0084
+870,54800,8.5343
+874,55200,10.2597
+878,55600,11.0706
+883,56000,9.7567
+887,56400,11.0066
+891,56800,11.0205
+897,57200,8.6028
+902,57600,7.8706
+906,58000,11.6827
+911,58400,11.5678
+916,58800,7.3302
+920,59200,11.8686
+925,59600,9.6659
+929,60000,11.8218
+933,60400,11.753
+938,60800,10.8838
+942,61200,11.9606
+946,61600,11.9891
+950,62000,10.7408
+955,62400,9.9649
+959,62800,10.1192
+963,63200,11.9243
+967,63600,12.0996
+971,64000,12.2641
+975,64400,12.2119
+980,64800,9.6896
+984,65200,11.985
+988,65600,12.0389
+993,66000,9.7261
+998,66400,10.0918
+1002,66800,11.9409
+1007,67200,9.7976
+1011,67600,11.9562
+1015,68000,11.9393
+1019,68400,11.9489
+1023,68800,10.6665
+1029,69200,8.6481
+1035,69600,8.2744
+1040,70000,9.8761
+1044,70400,12.7648
+1048,70800,12.0824
+1053,71200,9.9615
+1059,71600,7.5596
+1063,72000,12.066
+1068,72400,10.3717
+1072,72800,11.9771
+1076,73200,11.9456
+1080,73600,9.1042
+1084,74000,12.076
+1090,74400,9.4681
+1094,74800,11.529
+1098,75200,12.0008
+1103,75600,10.1566
+1108,76000,9.751
+1112,76400,12.3104
+1116,76800,9.2405
+1120,77200,12.0348
+1125,77600,10.0069
+1130,78000,10.0386
+1136,78400,10.4839
+1142,78800,7.6819
+1149,79200,5.9398
+1156,79600,7.5437
+1163,80000,7.4114
+1167,80400,9.315
+1172,80800,11.2425
+1176,81200,10.7758
+1180,81600,12.0153
+1185,82000,10.7721
+1190,82400,9.8986
+1194,82800,11.9053
+1198,83200,11.9558
+1203,83600,9.3269
+1207,84000,12.2099
+1213,84400,8.4628
+1217,84800,10.7038
+1221,85200,12.2837
+1226,85600,9.9915
+1231,86000,9.9481
+1236,86400,10.1545
+1240,86800,12.2327
+1244,87200,12.075
+1248,87600,12.01
+1252,88000,10.5855
+1256,88400,12.134
+1260,88800,12.3757
+1264,89200,11.9803
+1269,89600,9.3804
+1274,90000,9.8415
+1278,90400,11.9713
+1284,90800,8.3152
+1289,91200,8.8216
+1293,91600,12.3776
+1299,92000,8.471
+1303,92400,12.5027
+1307,92800,12.517
+1313,93200,8.4122
+1317,93600,13.1839
+1322,94000,10.4515
+1326,94400,11.1728
+1330,94800,13.0548
+1334,95200,12.0536
+1338,95600,12.4526
+1344,96000,9.0872
+1351,96400,7.9055
+1355,96800,11.2378
+1360,97200,11.1121
+1364,97600,12.7524
+1368,98000,12.6111
+1373,98400,11.2153
+1378,98800,8.2927
+1382,99200,12.5971
+1388,99600,9.538
+1392,100000,12.5941
+1397,100400,10.2156
+1401,100800,12.5003
+1406,101200,9.1204
+1411,101600,11.3426
+1418,102000,6.7031
+1422,102400,13.0065
+1427,102800,10.4644
+1431,103200,12.6849
+1435,103600,12.3948
+1441,104000,9.7853
+1446,104400,9.4532
+1450,104800,10.5596
+1456,105200,8.6362
+1461,105600,11.1625
+1465,106000,9.7383
+1471,106400,9.5542
+1476,106800,10.1016
+1480,107200,11.3806
+1484,107600,12.8012
+1488,108000,12.754
+1493,108400,9.2045
+1497,108800,12.9866
+1504,109200,7.3446
+1509,109600,11.8225
+1514,110000,8.9346
+1518,110400,12.7821
+1523,110800,11.2293
+1527,111200,12.976
+1532,111600,8.6775
+1536,112000,12.7469
+1540,112400,12.7591
+1545,112800,10.3778
+1550,113200,10.3744
+1554,113600,12.7611
+1558,114000,11.8172
+1562,114400,12.8452
+1566,114800,12.6974
+1570,115200,12.7894
+1575,115600,10.9595
+1579,116000,13.0989
+1583,116400,12.9278
+1589,116800,9.0682
+1593,117200,12.9978
+1597,117600,10.7087
+1601,118000,13.5158
+1605,118400,13.435
+1610,118800,11.5973
+1614,119200,13.0533
+1618,119600,13.0017
+1623,120000,10.8166
+1629,120400,9.0293
+1634,120800,9.2539
+1639,121200,10.6134
+1644,121600,11.5216
+1648,122000,13.3206
+1652,122400,12.207
+1657,122800,11.3809
+1661,123200,10.1333
+1665,123600,13.4154
+1669,124000,13.7245
+1673,124400,13.2766
+1677,124800,13.6751
+1681,125200,13.3002
+1686,125600,11.3154
+1690,126000,14.442
+1697,126400,9.1752
+1701,126800,12.6002
+1710,127200,8.7327
+1715,127600,11.4596
+1722,128000,9.0605
+1730,128400,6.6071
+1738,128800,7.8556
+1743,129200,8.6705
+1748,129600,13.2619
+1757,130000,8.0688
+1762,130400,11.255
+1769,130800,10.1434
+1776,131200,7.9637
+1780,131600,16.234
+1784,132000,16.009
+1790,132400,11.5696
+1795,132800,13.03
+1801,133200,11.5952
+1807,133600,10.1351
+1811,134000,14.54
+1823,134400,4.607
+1829,134800,12.6576
+1833,135200,16.8332
+1838,135600,11.8201
+1842,136000,17.1005
+1846,136400,16.8673
+1851,136800,15.3565
+1857,137200,8.8723
+1862,137600,13.1234
+1868,138000,12.9135
+1876,138400,6.6299
+1883,138800,8.6404
+1892,139200,8.0842
+1896,139600,16.6064
+1900,140000,12.9285
+1912,140400,5.249
+1918,140800,9.7863
+1926,141200,7.8766
+1932,141600,7.9322
+1942,142000,5.3181
+1947,142400,12.7024
+1956,142800,8.2081
+1968,143200,2.9574
+1975,143600,6.7944
+1981,144000,11.6649
+1994,144400,5.4107
+2003,144800,6.3419
+2012,145200,6.9728
+2023,145600,5.0183
+2031,146000,7.8319
+2041,146400,4.6116
+2054,146800,4.337
+2063,147200,5.3691
+2069,147600,12.9849
+2078,148000,8.5405
+2091,148400,4.9827
+2108,148800,3.026
+2123,149200,3.3827
+2132,149600,6.8501
+2141,150000,7.5638
+2152,150400,5.2852
+2162,150800,7.5752
+2170,151200,7.9508
+2177,151600,10.2038
+2191,152000,3.0956
+2200,152400,7.7211
+2208,152800,6.9913
+2215,153200,10.9929
+2219,153600,15.1937
+2226,154000,12.1173
+2235,154400,6.3827
+2244,154800,9.904
+2251,155200,10.4535
+2257,155600,11.1928
+2262,156000,11.8628
+2269,156400,11.1129
+2275,156800,10.8419
+2282,157200,10.6974
+2288,157600,12.6491
+2293,158000,17.1227
+2297,158400,15.1925
+2302,158800,15.8225
+2310,159200,10.5351
+2314,159600,20.5124
+2318,160000,20.7472
+2323,160400,16.1619
+2328,160800,17.0157
+2332,161200,19.5865
+2336,161600,20.3359
+2340,162000,15.7826
+2344,162400,20.7786
+2349,162800,19.027
+2353,163200,16.8306
+2358,163600,15.0345
+2364,164000,13.7065
+2370,164400,15.8193
+2375,164800,15.9792
+2379,165200,16.8467
+2384,165600,17.832
+2388,166000,20.4626
+2393,166400,14.8119
+2399,166800,12.4114
+2403,167200,20.9186
+2408,167600,13.3934
+2412,168000,20.5788
+2418,168400,16.2933
+2422,168800,18.8223
+2427,169200,19.3578
+2432,169600,13.3396
+2437,170000,17.3548
+2443,170400,16.2848
+2448,170800,18.1538
+2453,171200,16.7561
+2457,171600,17.7607
+2463,172000,15.1953
+2469,172400,13.134
+2476,172800,12.5457
+2482,173200,14.6165
+2487,173600,18.5189
+2492,174000,19.1314
+2496,174400,17.353
+2500,174800,22.18
+2506,175200,15.5115
+2511,175600,19.8744
+2515,176000,18.1736
+2519,176400,21.4187
+2525,176800,17.5231
+2531,177200,12.1789
+2536,177600,17.784
+2540,178000,20.6193
+2545,178400,17.4618
+2549,178800,21.499
+2553,179200,17.4586
+2557,179600,20.6227
+2561,180000,20.1806
+2567,180400,13.3524
+2573,180800,14.6734
+2577,181200,21.1076
+2582,181600,17.1151
+2587,182000,12.7979
+2592,182400,15.9493
+2599,182800,14.3687
+2604,183200,15.514
+2610,183600,12.1476
+2616,184000,13.7643
+2622,184400,14.6858
+2627,184800,13.8197
+2631,185200,20.2988
+2637,185600,13.8708
+2641,186000,21.6494
+2646,186400,15.6542
+2653,186800,11.9435
+2659,187200,13.8883
+2666,187600,12.5814
+2671,188000,17.3771
+2678,188400,11.9801
+2683,188800,13.1353
+2692,189200,10.337
+2698,189600,11.3343
+2705,190000,12.8742
+2711,190400,14.6771
+2717,190800,11.9189
+2726,191200,8.7059
+2732,191600,13.1508
+2739,192000,9.7473
+2743,192400,19.009
+2748,192800,16.8758
+2754,193200,15.7392
+2758,193600,20.0703
+2764,194000,12.4249
+2768,194400,21.7073
+2773,194800,18.1804
+2779,195200,15.5854
+2785,195600,12.3091
+2790,196000,15.0442
+2794,196400,20.9159
+2802,196800,11.6643
+2811,197200,7.4276
+2819,197600,8.6076
+2826,198000,10.6653
+2831,198400,16.6902
+2837,198800,13.4796
+2843,199200,14.7899
+2848,199600,14.6657
+2855,200000,11.2761

code/Lake application/logs/frozen_lake_PDPPO/PDPPO_frozen_lake_log_3.csv

@@ -0,0 +1,501 @@
+episode,timestep,reward
+20,400,1.4511
+42,800,1.4159
+60,1200,1.7077
+73,1600,2.1135
+84,2000,2.7468
+105,2400,1.7674
+113,2800,3.7814
+122,3200,3.7896
+134,3600,2.7067
+146,4000,2.5524
+152,4400,6.5864
+157,4800,6.5195
+162,5200,6.1349
+168,5600,5.317
+177,6000,3.6724
+183,6400,6.1446
+189,6800,5.5608
+195,7200,6.1726
+199,7600,6.344
+205,8000,5.8716
+213,8400,4.0247
+217,8800,6.9902
+224,9200,6.2118
+229,9600,6.7983
+236,10000,6.4537
+241,10400,9.274
+246,10800,7.4492
+253,11200,6.7536
+257,11600,10.0378
+264,12000,5.3493
+270,12400,6.8639
+274,12800,9.7271
+278,13200,10.6744
+284,13600,7.8632
+290,14000,8.3459
+296,14400,6.5538
+301,14800,9.4489
+307,15200,8.4332
+313,15600,7.6024
+318,16000,9.6646
+323,16400,7.5682
+327,16800,12.2827
+332,17200,11.2367
+336,17600,10.6158
+341,18000,8.7318
+347,18400,8.9077
+352,18800,10.0014
+357,19200,9.9727
+362,19600,8.4138
+368,20000,8.7518
+372,20400,12.3645
+376,20800,10.8224
+381,21200,11.2952
+387,21600,7.5035
+392,22000,10.1362
+398,22400,8.0769
+402,22800,11.4559
+408,23200,10.9315
+412,23600,12.5869
+416,24000,9.7607
+421,24400,10.3972
+427,24800,9.1654
+434,25200,6.5726
+440,25600,7.9518
+445,26000,12.163
+449,26400,12.5389
+453,26800,12.4047
+457,27200,12.8434
+461,27600,12.508
+465,28000,12.7977
+469,28400,12.7722
+475,28800,7.5211
+481,29200,8.8252
+485,29600,11.5555
+489,30000,11.477
+493,30400,12.9982
+497,30800,13.1206
+504,31200,9.1732
+509,31600,9.6877
+515,32000,8.65
+522,32400,6.6807
+529,32800,7.7346
+534,33200,7.8476
+541,33600,7.9549
+548,34000,6.9946
+555,34400,5.7617
+562,34800,7.4759
+567,35200,9.4369
+574,35600,8.9111
+581,36000,6.5248
+586,36400,10.6508
+594,36800,4.8766
+606,37200,5.0255
+611,37600,8.7737
+619,38000,7.297
+625,38400,8.7064
+631,38800,7.6392
+637,39200,8.1329
+643,39600,8.6996
+648,40000,12.1593
+654,40400,8.0097
+658,40800,13.7355
+663,41200,10.8767
+667,41600,13.4623
+671,42000,13.7128
+677,42400,7.9521
+685,42800,7.0389
+693,43200,7.1228
+699,43600,7.1906
+704,44000,10.2499
+711,44400,8.8439
+715,44800,10.723
+722,45200,8.0561
+726,45600,12.8837
+730,46000,11.6005
+735,46400,10.5382
+740,46800,10.9579
+744,47200,12.8439
+749,47600,8.5832
+755,48000,10.2625
+759,48400,11.0394
+764,48800,10.5021
+771,49200,7.5662
+775,49600,11.8596
+779,50000,10.9197
+785,50400,9.6613
+791,50800,7.1842
+797,51200,8.9205
+810,51600,3.7799
+814,52000,11.7011
+818,52400,13.1067
+825,52800,6.8794
+837,53200,4.1435
+844,53600,7.8758
+850,54000,6.8512
+857,54400,7.279
+863,54800,8.8461
+870,55200,5.1664
+876,55600,8.291
+882,56000,7.4186
+886,56400,12.7099
+895,56800,5.8991
+900,57200,10.3267
+905,57600,10.4081
+910,58000,8.2201
+917,58400,7.2555
+923,58800,8.6902
+929,59200,8.4311
+935,59600,9.3899
+940,60000,9.48
+947,60400,8.8701
+952,60800,10.6712
+957,61200,12.1303
+963,61600,10.0041
+967,62000,11.8506
+975,62400,7.9768
+981,62800,10.121
+985,63200,12.7484
+992,63600,8.6506
+996,64000,12.0166
+1003,64400,8.6989
+1007,64800,13.5841
+1012,65200,11.8948
+1020,65600,7.5362
+1024,66000,14.5183
+1031,66400,8.4667
+1039,66800,6.099
+1044,67200,9.9433
+1051,67600,10.2187
+1056,68000,12.4386
+1065,68400,4.846
+1071,68800,12.9973
+1076,69200,11.6068
+1084,69600,9.6249
+1090,70000,8.821
+1098,70400,8.9721
+1104,70800,10.8627
+1110,71200,12.9207
+1117,71600,9.6223
+1124,72000,9.5217
+1129,72400,12.7009
+1137,72800,7.1291
+1142,73200,15.8915
+1147,73600,12.773
+1152,74000,14.0152
+1158,74400,12.8197
+1165,74800,7.6715
+1171,75200,11.6273
+1176,75600,14.7481
+1182,76000,10.9798
+1186,76400,17.6803
+1193,76800,10.2361
+1198,77200,11.135
+1203,77600,13.5392
+1207,78000,16.3684
+1213,78400,12.9063
+1218,78800,11.9918
+1226,79200,8.0133
+1231,79600,13.8717
+1236,80000,13.592
+1240,80400,16.6494
+1247,80800,9.0394
+1253,81200,11.6721
+1257,81600,12.4117
+1263,82000,13.1805
+1268,82400,12.8848
+1273,82800,13.0017
+1279,83200,11.932
+1286,83600,10.3167
+1292,84000,11.0747
+1298,84400,12.7847
+1302,84800,16.4195
+1307,85200,14.5389
+1313,85600,11.3515
+1317,86000,19.0045
+1324,86400,12.204
+1328,86800,18.172
+1334,87200,11.4799
+1339,87600,15.2817
+1346,88000,12.8543
+1351,88400,15.2124
+1355,88800,20.012
+1360,89200,15.6753
+1364,89600,18.9953
+1369,90000,14.7316
+1373,90400,18.3781
+1379,90800,12.0495
+1383,91200,19.1038
+1388,91600,15.5228
+1394,92000,13.1508
+1399,92400,13.739
+1404,92800,15.4669
+1409,93200,12.2052
+1413,93600,18.4534
+1417,94000,18.4736
+1421,94400,19.451
+1426,94800,17.0253
+1431,95200,13.6257
+1436,95600,15.849
+1447,96000,6.1302
+1452,96400,13.5995
+1460,96800,10.4263
+1466,97200,12.8175
+1470,97600,19.8985
+1474,98000,19.4499
+1480,98400,12.2035
+1485,98800,15.5921
+1493,99200,9.945
+1500,99600,8.3822
+1504,100000,20.1108
+1511,100400,13.2678
+1517,100800,13.3653
+1521,101200,20.5694
+1526,101600,12.9576
+1531,102000,18.6283
+1537,102400,13.6185
+1542,102800,13.5109
+1547,103200,14.7459
+1555,103600,11.7803
+1562,104000,8.6873
+1572,104400,8.425
+1577,104800,12.4127
+1583,105200,11.9189
+1590,105600,12.8694
+1597,106000,10.6233
+1604,106400,8.3289
+1613,106800,9.3008
+1622,107200,7.9757
+1627,107600,15.1024
+1633,108000,12.5267
+1641,108400,10.7986
+1646,108800,10.9844
+1650,109200,20.3921
+1657,109600,12.9882
+1664,110000,8.8107
+1671,110400,13.2145
+1677,110800,13.7578
+1683,111200,11.6441
+1691,111600,10.6352
+1698,112000,9.8465
+1704,112400,13.4316
+1708,112800,19.3443
+1715,113200,11.1754
+1722,113600,10.6371
+1728,114000,9.1292
+1734,114400,14.4382
+1741,114800,13.7379
+1745,115200,20.8923
+1749,115600,16.3165
+1755,116000,15.3684
+1762,116400,9.6267
+1766,116800,17.2582
+1772,117200,14.4907
+1777,117600,16.4743
+1783,118000,13.0177
+1792,118400,7.393
+1799,118800,10.4686
+1804,119200,17.1223
+1814,119600,7.6813
+1820,120000,10.9129
+1826,120400,15.8259
+1830,120800,20.2207
+1837,121200,9.1836
+1843,121600,14.5043
+1848,122000,12.8295
+1853,122400,18.3265
+1858,122800,16.1205
+1863,123200,19.6436
+1869,123600,11.276
+1873,124000,20.749
+1882,124400,9.1372
+1888,124800,13.6346
+1893,125200,17.135
+1899,125600,10.655
+1904,126000,20.1275
+1908,126400,19.847
+1913,126800,15.7564
+1918,127200,14.3012
+1923,127600,15.9916
+1928,128000,15.8541
+1932,128400,20.358
+1937,128800,16.4342
+1941,129200,21.0778
+1946,129600,15.8157
+1952,130000,14.1684
+1956,130400,17.5727
+1962,130800,13.636
+1967,131200,16.7871
+1971,131600,19.9122
+1976,132000,17.2468
+1984,132400,10.4792
+1989,132800,15.3806
+1994,133200,15.9361
+1998,133600,20.595
+2003,134000,14.5683
+2010,134400,12.9443
+2017,134800,11.1499
+2021,135200,16.4114
+2028,135600,11.15
+2033,136000,16.1641
+2039,136400,11.8905
+2045,136800,12.094
+2051,137200,13.1055
+2057,137600,12.3794
+2062,138000,16.7795
+2066,138400,15.2186
+2071,138800,17.5386
+2075,139200,19.7387
+2080,139600,17.7345
+2087,140000,9.286
+2091,140400,20.1843
+2095,140800,20.6579
+2099,141200,20.4451
+2105,141600,12.451
+2111,142000,13.8034
+2116,142400,16.789
+2121,142800,17.3337
+2127,143200,12.512
+2132,143600,18.0873
+2137,144000,16.5148
+2143,144400,14.2245
+2150,144800,11.4969
+2158,145200,6.4741
+2167,145600,10.4108
+2173,146000,9.8208
+2180,146400,14.0351
+2186,146800,11.3705
+2191,147200,17.1942
+2196,147600,16.5483
+2201,148000,17.2101
+2206,148400,15.8254
+2211,148800,16.944
+2216,149200,13.2711
+2222,149600,16.0584
+2227,150000,15.4412
+2231,150400,20.0904
+2235,150800,18.9388
+2241,151200,10.2277
+2248,151600,9.7938
+2257,152000,6.8651
+2265,152400,10.0151
+2269,152800,16.5873
+2275,153200,13.0192
+2280,153600,16.4787
+2284,154000,19.9945
+2288,154400,21.1409
+2293,154800,17.4874
+2298,155200,16.9422
+2302,155600,16.5356
+2307,156000,17.7096
+2314,156400,11.4627
+2318,156800,20.033
+2323,157200,14.9603
+2327,157600,19.5649
+2334,158000,10.3648
+2339,158400,17.1039
+2343,158800,21.143
+2348,159200,17.2194
+2352,159600,21.0071
+2356,160000,21.8045
+2361,160400,17.6473
+2367,160800,15.5177
+2371,161200,20.4818
+2376,161600,17.1564
+2381,162000,14.4102
+2386,162400,18.4027
+2390,162800,22.3808
+2395,163200,14.4186
+2400,163600,13.4343
+2404,164000,19.605
+2409,164400,14.7853
+2415,164800,11.0992
+2421,165200,12.6908
+2426,165600,17.289
+2431,166000,17.1976
+2435,166400,21.6014
+2441,166800,12.793
+2445,167200,14.232
+2450,167600,15.6217
+2457,168000,14.2949
+2461,168400,16.1677
+2465,168800,21.0584
+2470,169200,19.4555
+2474,169600,16.2403
+2481,170000,10.8878
+2491,170400,4.966
+2497,170800,11.4854
+2509,171200,4.8386
+2517,171600,9.0965
+2532,172000,4.4732
+2539,172400,10.3322
+2548,172800,6.617
+2559,173200,6.6052
+2566,173600,8.1639
+2571,174000,14.7295
+2575,174400,19.1853
+2581,174800,11.4953
+2586,175200,15.2959
+2591,175600,17.3706
+2597,176000,16.6239
+2602,176400,16.1638
+2608,176800,13.7811
+2614,177200,11.7461
+2624,177600,8.2969
+2629,178000,16.7226
+2636,178400,10.9358
+2642,178800,16.5572
+2649,179200,10.4538
+2655,179600,12.1145
+2660,180000,17.3922
+2669,180400,10.0921
+2675,180800,14.2412
+2680,181200,16.8869
+2684,181600,17.4712
+2689,182000,19.3122
+2693,182400,22.4468
+2699,182800,15.3699
+2704,183200,18.0721
+2710,183600,13.5026
+2717,184000,12.2414
+2721,184400,17.901
+2726,184800,16.2374
+2731,185200,17.4196
+2736,185600,18.1275
+2740,186000,20.9471
+2745,186400,20.6423
+2750,186800,18.2358
+2755,187200,17.1191
+2760,187600,17.9405
+2766,188000,12.8474
+2770,188400,19.6427
+2776,188800,15.7317
+2780,189200,19.7856
+2787,189600,10.9979
+2792,190000,18.7782
+2798,190400,13.124
+2803,190800,15.1497
+2808,191200,15.8551
+2818,191600,6.6157
+2824,192000,15.3035
+2831,192400,11.4718
+2835,192800,22.8164
+2841,193200,16.0213
+2846,193600,17.6302
+2851,194000,14.8716
+2855,194400,20.6271
+2862,194800,14.8134
+2868,195200,10.8108
+2872,195600,22.52
+2879,196000,13.9515
+2885,196400,12.6253
+2893,196800,8.5209
+2900,197200,11.9898
+2906,197600,13.4315
+2911,198000,18.833
+2918,198400,11.5714
+2923,198800,15.1536
+2931,199200,9.8851
+2937,199600,12.3961
+2943,200000,10.9256

code/Lake application/logs/frozen_lake_PDPPO/PDPPO_frozen_lake_log_4.csv

@@ -0,0 +1,501 @@
+episode,timestep,reward
+27,400,1.0785
+45,800,1.6504
+65,1200,1.4004
+75,1600,2.9704
+92,2000,1.7748
+101,2400,3.44
+112,2800,2.7486
+118,3200,4.5942
+124,3600,5.5212
+132,4000,3.7269
+139,4400,4.3077
+146,4800,4.8111
+151,5200,5.5228
+157,5600,6.1189
+163,6000,5.2543
+168,6400,5.2306
+175,6800,5.3845
+182,7200,4.3935
+187,7600,5.7774
+192,8000,6.1537
+197,8400,5.7906
+202,8800,7.0905
+209,9200,4.5122
+213,9600,6.5891
+218,10000,7.7331
+223,10400,6.6175
+227,10800,10.1737
+231,11200,8.0792
+238,11600,5.0695
+243,12000,6.8807
+247,12400,6.8492
+252,12800,7.4324
+256,13200,7.9897
+263,13600,4.5812
+270,14000,4.8277
+275,14400,6.4227
+281,14800,7.0743
+287,15200,5.8331
+292,15600,5.6509
+297,16000,7.8764
+303,16400,4.8805
+308,16800,6.1394
+312,17200,7.3848
+319,17600,5.4064
+326,18000,4.2903
+333,18400,4.4865
+338,18800,6.5078
+344,19200,5.3164
+348,19600,6.5957
+352,20000,8.4949
+357,20400,8.0414
+361,20800,8.259
+366,21200,8.7431
+371,21600,8.1425
+375,22000,7.6031
+379,22400,10.6469
+384,22800,7.4268
+389,23200,7.8948
+393,23600,9.7095
+399,24000,7.9959
+403,24400,12.6126
+407,24800,10.4783
+414,25200,8.8362
+418,25600,12.1951
+422,26000,12.6242
+426,26400,13.1493
+430,26800,11.1338
+434,27200,11.6175
+440,27600,7.7618
+445,28000,10.8874
+451,28400,9.7606
+456,28800,11.5253
+462,29200,6.7856
+467,29600,11.7964
+472,30000,8.8168
+476,30400,13.8556
+481,30800,11.2407
+485,31200,13.1211
+491,31600,10.3695
+496,32000,8.6598
+500,32400,13.3756
+506,32800,9.0459
+511,33200,10.4668
+516,33600,9.0482
+522,34000,9.8797
+526,34400,13.0444
+532,34800,7.3031
+536,35200,12.3843
+541,35600,11.6548
+545,36000,12.1808
+550,36400,8.937
+554,36800,13.32
+559,37200,10.2187
+563,37600,11.1425
+569,38000,10.4837
+573,38400,9.4084
+579,38800,9.5927
+585,39200,8.1512
+591,39600,7.4306
+598,40000,7.4929
+603,40400,12.0014
+609,40800,6.8877
+615,41200,8.7736
+622,41600,6.575
+632,42000,3.9331
+638,42400,9.7754
+644,42800,8.4288
+649,43200,9.7733
+656,43600,6.2225
+666,44000,4.1808
+672,44400,6.0964
+685,44800,3.3983
+694,45200,5.6596
+702,45600,6.6803
+708,46000,7.2517
+716,46400,5.9717
+729,46800,3.6724
+740,47200,4.6841
+744,47600,12.0424
+751,48000,8.4528
+757,48400,7.1028
+761,48800,12.6503
+770,49200,4.9012
+775,49600,9.6465
+782,50000,7.875
+787,50400,12.0186
+793,50800,9.2714
+798,51200,9.0638
+804,51600,9.1859
+808,52000,13.3962
+814,52400,9.1845
+819,52800,8.6704
+823,53200,12.922
+827,53600,13.1999
+832,54000,10.8399
+837,54400,12.6242
+842,54800,11.2661
+847,55200,10.5994
+853,55600,8.1275
+858,56000,12.1206
+866,56400,6.3821
+871,56800,8.825
+875,57200,13.9679
+880,57600,13.0994
+884,58000,14.0452
+888,58400,10.9111
+892,58800,13.7891
+897,59200,12.1729
+902,59600,12.7809
+907,60000,10.9993
+914,60400,6.609
+922,60800,6.1909
+927,61200,10.7844
+934,61600,7.0903
+945,62000,4.2601
+953,62400,5.6534
+958,62800,10.075
+963,63200,10.4977
+967,63600,11.9044
+971,64000,13.6212
+975,64400,13.086
+980,64800,12.0809
+987,65200,8.3579
+993,65600,8.2868
+997,66000,14.3792
+1001,66400,11.228
+1008,66800,8.2525
+1017,67200,6.1501
+1022,67600,9.1965
+1027,68000,11.0591
+1032,68400,11.1621
+1038,68800,10.1783
+1042,69200,10.0313
+1052,69600,5.234
+1060,70000,7.2449
+1064,70400,11.1294
+1070,70800,7.2192
+1075,71200,10.5247
+1082,71600,8.6408
+1087,72000,12.3906
+1091,72400,10.4689
+1095,72800,14.3503
+1100,73200,13.3414
+1104,73600,14.6343
+1108,74000,12.0234
+1113,74400,10.4043
+1118,74800,13.1958
+1122,75200,14.1113
+1128,75600,8.9121
+1133,76000,9.4241
+1138,76400,12.1937
+1142,76800,10.8917
+1148,77200,9.6075
+1152,77600,12.5465
+1157,78000,12.2911
+1161,78400,14.3991
+1166,78800,11.7178
+1170,79200,14.2131
+1176,79600,9.4352
+1182,80000,9.8783
+1187,80400,8.9671
+1192,80800,11.4625
+1197,81200,13.3021
+1201,81600,14.9308
+1205,82000,11.3369
+1212,82400,9.6094
+1217,82800,11.86
+1221,83200,12.7279
+1227,83600,10.7913
+1232,84000,11.0473
+1236,84400,14.2463
+1242,84800,9.8461
+1246,85200,14.7624
+1252,85600,10.4951
+1259,86000,8.9123
+1265,86400,10.1258
+1271,86800,9.0527
+1276,87200,12.5009
+1280,87600,14.9112
+1286,88000,9.7683
+1291,88400,12.2766
+1295,88800,14.6523
+1303,89200,8.0044
+1307,89600,14.4251
+1311,90000,14.7435
+1316,90400,13.5846
+1322,90800,11.1363
+1326,91200,13.1161
+1330,91600,14.8782
+1335,92000,12.9898
+1341,92400,10.5745
+1349,92800,7.7201
+1355,93200,9.7362
+1362,93600,9.0283
+1371,94000,5.595
+1376,94400,9.584
+1384,94800,7.7162
+1389,95200,9.5471
+1395,95600,11.8725
+1400,96000,11.4859
+1406,96400,10.0133
+1411,96800,10.0898
+1419,97200,7.1929
+1425,97600,10.2308
+1431,98000,10.8156
+1436,98400,11.1904
+1441,98800,13.0126
+1448,99200,9.5278
+1456,99600,7.7219
+1460,100000,11.838
+1467,100400,10.3963
+1471,100800,15.1954
+1477,101200,9.7102
+1483,101600,10.8302
+1487,102000,13.6986
+1491,102400,17.6513
+1497,102800,11.7696
+1502,103200,14.6363
+1506,103600,12.3268
+1510,104000,15.2753
+1515,104400,13.8737
+1520,104800,9.0522
+1524,105200,15.7886
+1529,105600,14.429
+1534,106000,12.0745
+1539,106400,13.0085
+1543,106800,16.5044
+1547,107200,12.8213
+1552,107600,10.883
+1557,108000,12.4299
+1561,108400,12.7274
+1565,108800,15.701
+1571,109200,11.4077
+1576,109600,13.4852
+1580,110000,15.1326
+1585,110400,12.6496
+1592,110800,11.1331
+1598,111200,10.001
+1604,111600,9.8463
+1610,112000,9.5269
+1615,112400,12.4523
+1623,112800,8.4184
+1627,113200,14.1469
+1634,113600,7.9029
+1640,114000,11.989
+1646,114400,10.0607
+1654,114800,7.935
+1658,115200,15.8146
+1662,115600,16.4529
+1666,116000,16.031
+1670,116400,16.961
+1674,116800,16.9752
+1680,117200,11.4042
+1687,117600,10.9484
+1693,118000,11.3393
+1697,118400,19.1626
+1702,118800,16.0651
+1707,119200,12.5378
+1713,119600,14.3033
+1720,120000,11.5588
+1724,120400,15.8731
+1729,120800,16.0878
+1734,121200,16.3894
+1739,121600,14.4359
+1745,122000,14.7653
+1752,122400,10.6896
+1756,122800,17.8908
+1762,123200,14.3452
+1768,123600,13.2548
+1772,124000,19.6647
+1776,124400,19.1847
+1780,124800,16.8379
+1786,125200,14.3642
+1790,125600,16.4554
+1794,126000,19.7687
+1801,126400,10.1183
+1806,126800,18.1265
+1812,127200,11.8105
+1818,127600,11.0402
+1823,128000,16.0082
+1830,128400,9.7205
+1836,128800,10.7887
+1841,129200,12.7927
+1845,129600,17.2088
+1852,130000,11.4728
+1857,130400,13.019
+1861,130800,18.7369
+1868,131200,11.284
+1873,131600,13.2389
+1878,132000,15.4627
+1882,132400,19.8252
+1889,132800,10.8438
+1894,133200,12.0874
+1901,133600,11.009
+1906,134000,13.7517
+1915,134400,8.6863
+1921,134800,10.9831
+1928,135200,11.1524
+1934,135600,11.4495
+1941,136000,9.4189
+1945,136400,18.9608
+1952,136800,9.5868
+1958,137200,10.5218
+1964,137600,13.0402
+1972,138000,7.7796
+1980,138400,7.3364
+1986,138800,13.7623
+1991,139200,16.2419
+1996,139600,11.4699
+2003,140000,10.983
+2012,140400,7.5045
+2016,140800,20.0741
+2022,141200,12.6223
+2027,141600,15.3017
+2033,142000,12.7772
+2037,142400,20.7311
+2041,142800,18.8427
+2046,143200,19.43
+2051,143600,15.0906
+2056,144000,13.0377
+2065,144400,7.2051
+2071,144800,13.2424
+2078,145200,11.0955
+2088,145600,5.3895
+2093,146000,12.318
+2100,146400,12.8436
+2105,146800,14.4873
+2109,147200,19.8853
+2117,147600,10.7438
+2123,148000,13.8923
+2128,148400,15.6226
+2132,148800,18.0464
+2137,149200,16.3704
+2144,149600,12.576
+2148,150000,20.3183
+2156,150400,9.807
+2161,150800,12.9068
+2165,151200,21.0426
+2171,151600,15.371
+2175,152000,17.7154
+2180,152400,17.2565
+2185,152800,18.8765
+2190,153200,17.2862
+2194,153600,19.4167
+2200,154000,13.2546
+2205,154400,15.271
+2211,154800,14.3329
+2215,155200,21.1205
+2221,155600,12.1206
+2227,156000,14.4072
+2232,156400,16.5247
+2236,156800,21.231
+2240,157200,20.8302
+2244,157600,19.4892
+2248,158000,20.936
+2252,158400,21.2178
+2256,158800,21.1056
+2261,159200,16.8762
+2267,159600,14.0483
+2272,160000,13.3591
+2279,160400,12.6142
+2285,160800,9.0069
+2291,161200,13.7843
+2300,161600,7.8473
+2307,162000,9.9078
+2312,162400,15.3595
+2317,162800,14.6767
+2324,163200,10.7115
+2331,163600,9.728
+2336,164000,14.8431
+2341,164400,15.2026
+2346,164800,12.322
+2351,165200,11.1342
+2356,165600,14.9371
+2361,166000,13.2367
+2365,166400,20.3463
+2371,166800,10.1723
+2377,167200,10.4504
+2381,167600,18.0134
+2388,168000,10.336
+2395,168400,11.1223
+2403,168800,6.6491
+2410,169200,9.6499
+2418,169600,7.0802
+2423,170000,10.1417
+2431,170400,6.2054
+2440,170800,7.8601
+2449,171200,7.7198
+2454,171600,14.5012
+2459,172000,15.7605
+2467,172400,7.7054
+2473,172800,12.5347
+2478,173200,15.5908
+2486,173600,5.7416
+2494,174000,9.3572
+2501,174400,9.0669
+2505,174800,11.2773
+2512,175200,6.6277
+2518,175600,4.9397
+2524,176000,6.4387
+2529,176400,9.8694
+2534,176800,8.5331
+2538,177200,8.5411
+2545,177600,6.2453
+2549,178000,11.7878
+2554,178400,10.175
+2558,178800,12.4336
+2565,179200,6.5908
+2570,179600,6.219
+2577,180000,6.0009
+2582,180400,5.8638
+2588,180800,6.9441
+2592,181200,6.8218
+2598,181600,9.0203
+2604,182000,7.8072
+2609,182400,6.4259
+2616,182800,6.3331
+2622,183200,9.3925
+2631,183600,6.0891
+2639,184000,5.3467
+2645,184400,5.1621
+2652,184800,7.0998
+2656,185200,8.041
+2663,185600,4.7481
+2670,186000,5.0002
+2677,186400,4.4118
+2682,186800,5.4232
+2689,187200,7.2963
+2693,187600,6.6073
+2699,188000,8.326
+2707,188400,5.5025
+2713,188800,5.5795
+2721,189200,3.796
+2728,189600,5.6303
+2734,190000,6.2013
+2740,190400,6.0202
+2746,190800,4.2373
+2751,191200,8.5135
+2758,191600,3.9532
+2763,192000,7.6742
+2770,192400,4.637
+2776,192800,4.4964
+2781,193200,8.2152
+2789,193600,4.0498
+2793,194000,7.944
+2798,194400,6.7629
+2802,194800,6.5532
+2807,195200,7.4252
+2811,195600,8.0144
+2815,196000,7.9792
+2819,196400,7.9971
+2824,196800,5.5573
+2828,197200,7.3163
+2833,197600,8.0791
+2838,198000,5.7746
+2843,198400,7.1306
+2848,198800,7.198
+2852,199200,8.5707
+2858,199600,5.5788
+2862,200000,8.7377

code/Lake application/logs/frozen_lake_PDPPO/PDPPO_frozen_lake_log_5.csv

@@ -0,0 +1,501 @@
+episode,timestep,reward
+22,400,1.2943
+41,800,1.6015
+55,1200,1.9737
+64,1600,3.5487
+73,2000,3.146
+82,2400,3.2991
+92,2800,2.7673
+100,3200,3.8523
+107,3600,3.7979
+115,4000,4.256
+119,4400,6.4793
+124,4800,5.5414
+128,5200,7.3476
+133,5600,5.8006
+138,6000,6.1344
+144,6400,5.8326
+150,6800,4.8366
+154,7200,6.7648
+159,7600,6.5947
+163,8000,7.3957
+167,8400,7.3241
+172,8800,5.9961
+177,9200,6.0296
+182,9600,7.2685
+187,10000,6.637
+192,10400,5.999
+196,10800,6.4386
+201,11200,6.6646
+207,11600,5.4069
+212,12000,6.6252
+216,12400,6.4882
+223,12800,5.5298
+228,13200,6.2035
+233,13600,5.9536
+237,14000,6.9952
+244,14400,4.9771
+249,14800,6.1157
+253,15200,7.8858
+257,15600,7.8068
+261,16000,7.9837
+266,16400,6.6484
+270,16800,8.0041
+274,17200,6.6656
+278,17600,8.0284
+283,18000,7.3533
+290,18400,4.5723
+295,18800,6.8589
+299,19200,8.1765
+303,19600,8.1
+307,20000,8.1549
+312,20400,5.818
+318,20800,6.0381
+323,21200,5.8448
+327,21600,8.4165
+332,22000,6.3011
+336,22400,8.2937
+343,22800,5.503
+347,23200,8.5968
+353,23600,5.1292
+358,24000,7.4338
+363,24400,6.6991
+368,24800,6.5642
+373,25200,6.8225
+377,25600,8.3731
+382,26000,7.4055
+387,26400,7.1132
+393,26800,5.6948
+399,27200,5.2845
+404,27600,5.9669
+408,28000,8.3363
+415,28400,5.6672
+420,28800,5.4611
+425,29200,7.714
+429,29600,7.5512
+434,30000,7.2347
+439,30400,6.5507
+443,30800,8.4242
+449,31200,6.0604
+453,31600,8.2476
+458,32000,8.0065
+462,32400,8.2966
+467,32800,6.7805
+471,33200,9.484
+475,33600,8.7649
+481,34000,8.2474
+485,34400,9.5795
+490,34800,8.6045
+496,35200,7.4124
+501,35600,7.6157
+505,36000,10.9111
+510,36400,8.8556
+514,36800,9.9951
+519,37200,11.3025
+523,37600,11.5043
+527,38000,11.7302
+532,38400,10.3442
+536,38800,12.0948
+541,39200,10.2846
+545,39600,12.2089
+549,40000,11.8314
+554,40400,9.8942
+559,40800,10.0485
+563,41200,12.4629
+567,41600,12.2332
+571,42000,11.1847
+575,42400,12.4524
+580,42800,10.2353
+587,43200,6.303
+591,43600,12.3469
+596,44000,11.1024
+600,44400,12.4106
+604,44800,11.5728
+608,45200,10.9579
+614,45600,9.6399
+618,46000,11.3217
+622,46400,12.529
+628,46800,9.0714
+633,47200,9.9013
+638,47600,10.0751
+643,48000,10.2499
+647,48400,12.5708
+652,48800,8.7569
+656,49200,12.3604
+660,49600,12.5165
+664,50000,12.3671
+668,50400,12.618
+672,50800,12.6543
+677,51200,10.6911
+681,51600,11.8317
+685,52000,12.6955
+690,52400,10.1157
+695,52800,10.0116
+700,53200,10.2901
+705,53600,8.1824
+710,54000,9.8285
+716,54400,8.8717
+721,54800,9.9854
+725,55200,12.9736
+731,55600,8.6973
+737,56000,8.4719
+742,56400,10.3744
+747,56800,8.9466
+752,57200,10.6086
+758,57600,10.0045
+763,58000,9.8052
+768,58400,11.2697
+774,58800,7.2144
+780,59200,9.6594
+784,59600,12.3888
+789,60000,10.4215
+795,60400,8.8573
+800,60800,10.8871
+807,61200,5.8992
+812,61600,10.12
+817,62000,10.8824
+821,62400,13.1221
+826,62800,10.7535
+830,63200,13.1014
+835,63600,10.7345
+839,64000,13.082
+843,64400,13.2581
+847,64800,13.4413
+851,65200,13.5097
+855,65600,13.3641
+859,66000,13.3583
+863,66400,13.1701
+868,66800,11.1134
+872,67200,13.6245
+880,67600,5.9886
+886,68000,10.2475
+890,68400,14.6202
+894,68800,10.0685
+900,69200,11.0053
+904,69600,13.6849
+909,70000,9.6346
+913,70400,13.5021
+917,70800,13.1414
+922,71200,10.8646
+926,71600,13.4908
+932,72000,9.1396
+936,72400,11.6489
+940,72800,13.2407
+944,73200,13.0342
+948,73600,13.5329
+952,74000,13.4403
+956,74400,13.7446
+960,74800,13.1605
+964,75200,13.3433
+969,75600,10.9952
+974,76000,11.5741
+980,76400,9.3918
+985,76800,11.7496
+992,77200,8.3268
+997,77600,10.0094
+1002,78000,10.6563
+1007,78400,11.5348
+1011,78800,15.119
+1017,79200,12.0912
+1023,79600,8.9712
+1027,80000,11.6409
+1031,80400,15.629
+1036,80800,13.3766
+1040,81200,15.2536
+1045,81600,12.2289
+1049,82000,15.4839
+1055,82400,9.1355
+1060,82800,12.202
+1064,83200,14.748
+1068,83600,14.5001
+1072,84000,15.0432
+1080,84400,7.0194
+1085,84800,10.1806
+1090,85200,11.795
+1095,85600,12.7322
+1100,86000,10.2449
+1109,86400,6.4535
+1117,86800,6.893
+1123,87200,8.8297
+1129,87600,9.9537
+1134,88000,9.4762
+1138,88400,15.7652
+1144,88800,12.4334
+1153,89200,6.025
+1157,89600,14.0408
+1162,90000,12.8277
+1167,90400,12.4492
+1173,90800,8.5448
+1180,91200,9.9864
+1184,91600,15.1804
+1190,92000,8.7061
+1196,92400,11.9629
+1202,92800,9.8801
+1207,93200,10.6953
+1211,93600,16.5854
+1221,94000,6.0867
+1229,94400,9.2746
+1234,94800,12.5134
+1240,95200,8.38
+1245,95600,14.4995
+1249,96000,16.0927
+1255,96400,8.763
+1261,96800,11.1095
+1265,97200,15.5608
+1271,97600,12.1043
+1278,98000,9.2772
+1283,98400,10.4808
+1291,98800,9.4265
+1296,99200,13.4881
+1303,99600,8.7073
+1311,100000,7.7693
+1323,100400,5.2588
+1328,100800,12.0091
+1338,101200,6.3879
+1346,101600,9.2148
+1354,102000,6.5712
+1359,102400,15.4022
+1363,102800,17.8751
+1372,103200,5.6517
+1379,103600,11.7892
+1383,104000,13.7202
+1390,104400,12.0935
+1398,104800,8.3031
+1403,105200,11.4273
+1409,105600,13.6581
+1414,106000,12.8389
+1421,106400,10.5737
+1426,106800,14.0672
+1431,107200,15.1905
+1435,107600,16.499
+1441,108000,14.9027
+1447,108400,12.2743
+1453,108800,12.5351
+1458,109200,13.3324
+1464,109600,10.1987
+1470,110000,14.6681
+1480,110400,6.3981
+1484,110800,17.443
+1489,111200,11.7544
+1494,111600,12.3627
+1500,112000,14.2433
+1507,112400,9.9183
+1514,112800,10.1194
+1520,113200,11.6173
+1526,113600,12.6407
+1530,114000,14.9395
+1537,114400,10.6247
+1546,114800,8.2285
+1555,115200,7.4419
+1565,115600,6.9423
+1571,116000,11.8016
+1576,116400,13.4006
+1582,116800,12.1714
+1589,117200,9.5481
+1594,117600,17.1002
+1599,118000,13.3729
+1605,118400,14.4748
+1610,118800,15.9086
+1616,119200,13.515
+1621,119600,13.6216
+1627,120000,13.3638
+1631,120400,15.8724
+1637,120800,13.1813
+1642,121200,16.7492
+1647,121600,16.2201
+1652,122000,14.2471
+1658,122400,11.9562
+1663,122800,14.3429
+1670,123200,8.2829
+1675,123600,12.7689
+1679,124000,17.9555
+1686,124400,10.3992
+1693,124800,9.3774
+1700,125200,10.5229
+1705,125600,13.095
+1709,126000,16.827
+1716,126400,11.5151
+1720,126800,15.0954
+1725,127200,17.1223
+1729,127600,19.2025
+1734,128000,16.3688
+1740,128400,13.0476
+1744,128800,17.5761
+1751,129200,9.9195
+1758,129600,12.3492
+1763,130000,12.9946
+1768,130400,15.0853
+1774,130800,12.7493
+1780,131200,13.7049
+1784,131600,16.4027
+1791,132000,10.02
+1796,132400,13.2953
+1802,132800,13.2571
+1807,133200,16.9227
+1816,133600,7.5362
+1823,134000,9.0337
+1831,134400,8.2112
+1839,134800,9.5222
+1845,135200,10.5068
+1851,135600,15.2168
+1860,136000,7.9646
+1866,136400,9.9186
+1872,136800,11.7983
+1878,137200,15.5265
+1884,137600,11.4403
+1889,138000,14.1125
+1893,138400,18.7814
+1898,138800,15.9716
+1907,139200,6.6898
+1916,139600,8.6765
+1926,140000,6.0023
+1930,140400,18.1008
+1938,140800,8.4123
+1946,141200,8.4666
+1953,141600,9.8084
+1958,142000,12.6225
+1962,142400,16.6253
+1969,142800,11.879
+1979,143200,5.891
+1983,143600,14.6341
+1991,144000,10.6154
+1995,144400,19.3371
+2000,144800,14.6999
+2005,145200,17.3212
+2010,145600,13.511
+2019,146000,8.9176
+2025,146400,11.9234
+2029,146800,18.7508
+2035,147200,11.3652
+2042,147600,10.7875
+2047,148000,17.2345
+2052,148400,13.469
+2057,148800,15.0187
+2062,149200,18.2483
+2067,149600,15.1589
+2072,150000,13.8506
+2078,150400,13.5154
+2083,150800,17.4931
+2088,151200,13.0434
+2094,151600,12.4448
+2098,152000,16.2641
+2103,152400,18.5631
+2109,152800,13.4721
+2114,153200,17.5024
+2118,153600,17.2629
+2123,154000,16.0443
+2130,154400,12.4068
+2134,154800,19.7433
+2141,155200,12.1725
+2145,155600,17.4482
+2151,156000,14.9725
+2158,156400,8.3286
+2163,156800,15.3569
+2168,157200,16.8249
+2176,157600,10.7159
+2181,158000,13.5756
+2186,158400,16.8722
+2191,158800,16.1566
+2196,159200,16.4402
+2200,159600,19.2901
+2206,160000,13.408
+2216,160400,6.5277
+2221,160800,16.5398
+2225,161200,17.9313
+2229,161600,16.0893
+2234,162000,13.5297
+2239,162400,18.5255
+2244,162800,15.412
+2249,163200,16.7656
+2254,163600,12.2883
+2260,164000,12.8931
+2268,164400,9.994
+2273,164800,14.903
+2278,165200,14.9247
+2284,165600,13.8873
+2288,166000,15.893
+2294,166400,16.0733
+2298,166800,20.3023
+2306,167200,8.823
+2311,167600,14.7959
+2316,168000,15.5506
+2320,168400,18.9412
+2328,168800,9.9269
+2337,169200,7.8662
+2342,169600,13.8098
+2347,170000,16.0699
+2352,170400,14.2326
+2357,170800,14.3457
+2365,171200,7.755
+2372,171600,11.1969
+2377,172000,17.7461
+2382,172400,14.5072
+2387,172800,15.6355
+2392,173200,16.9889
+2397,173600,16.7953
+2402,174000,14.2381
+2407,174400,15.2855
+2414,174800,9.9614
+2422,175200,6.716
+2429,175600,9.1914
+2433,176000,15.9262
+2439,176400,13.2029
+2444,176800,12.0649
+2448,177200,11.2086
+2453,177600,17.596
+2458,178000,10.4265
+2464,178400,12.7507
+2470,178800,10.5018
+2477,179200,9.6174
+2484,179600,12.3463
+2493,180000,7.463
+2502,180400,7.0977
+2511,180800,6.566
+2516,181200,12.4002
+2522,181600,12.8125
+2526,182000,16.8705
+2532,182400,15.6621
+2537,182800,15.5005
+2545,183200,7.1256
+2549,183600,19.091
+2553,184000,20.7269
+2558,184400,17.8736
+2562,184800,16.5145
+2568,185200,12.29
+2574,185600,11.165
+2580,186000,14.862
+2585,186400,12.9498
+2590,186800,16.6379
+2597,187200,12.0999
+2601,187600,19.986
+2605,188000,21.4165
+2610,188400,12.3688
+2614,188800,17.9821
+2618,189200,20.5196
+2623,189600,19.5269
+2629,190000,11.2376
+2633,190400,20.1738
+2637,190800,20.5599
+2642,191200,15.8952
+2649,191600,12.2397
+2654,192000,16.6287
+2659,192400,14.966
+2665,192800,13.1073
+2671,193200,11.1274
+2676,193600,15.6753
+2681,194000,15.8626
+2688,194400,9.1965
+2693,194800,15.0967
+2697,195200,19.8348
+2701,195600,14.855
+2706,196000,13.0618
+2715,196400,9.1275
+2721,196800,7.1574
+2726,197200,16.5246
+2732,197600,10.026
+2739,198000,11.8092
+2745,198400,10.2368
+2751,198800,11.7544
+2757,199200,11.6034
+2764,199600,9.3743
+2769,200000,15.3278

code/Lake application/logs/frozen_lake_PPO/PPO_frozen_lake_0_0.pth

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:8370e4fdc218fd2e709261132b6cb059f988bd8753b44228b5191c345b8cc2ed
+size 205481

code/Lake application/logs/frozen_lake_PPO/PPO_frozen_lake_log_1.csv

@@ -0,0 +1,501 @@
+episode,timestep,reward
+15,400,1.9345
+32,800,1.8784
+42,1200,3.0732
+52,1600,2.9835
+65,2000,2.3838
+69,2400,7.2159
+75,2800,4.758
+80,3200,5.6732
+85,3600,7.076
+89,4000,5.77
+95,4400,5.4889
+101,4800,4.8132
+106,5200,7.4812
+115,5600,2.816
+123,6000,4.0168
+129,6400,5.4723
+137,6800,3.675
+142,7200,5.8091
+147,7600,5.6523
+151,8000,7.3976
+158,8400,4.279
+164,8800,4.7596
+168,9200,7.4893
+173,9600,6.204
+177,10000,7.3342
+183,10400,4.9965
+188,10800,6.2773
+193,11200,6.1197
+197,11600,7.2967
+201,12000,7.4756
+205,12400,7.1791
+209,12800,7.3157
+213,13200,7.0592
+218,13600,5.8954
+222,14000,7.1992
+227,14400,6.5493
+232,14800,5.9315
+237,15200,7.0655
+243,15600,5.2935
+251,16000,3.6833
+258,16400,4.511
+267,16800,4.1031
+271,17200,7.4993
+275,17600,9.0943
+279,18000,7.6642
+285,18400,6.6167
+291,18800,5.4259
+295,19200,7.9226
+299,19600,8.2753
+304,20000,6.8015
+308,20400,7.8775
+315,20800,4.5316
+321,21200,5.4028
+326,21600,6.3314
+330,22000,8.1528
+334,22400,7.4973
+339,22800,6.6922
+343,23200,7.7072
+347,23600,7.2187
+351,24000,7.7859
+356,24400,7.0575
+360,24800,7.2281
+366,25200,5.8797
+372,25600,4.8101
+378,26000,5.0378
+384,26400,5.6277
+389,26800,5.2954
+400,27200,2.9769
+406,27600,4.6418
+416,28000,3.4303
+423,28400,4.0932
+430,28800,4.505
+435,29200,5.5546
+439,29600,10.5906
+445,30000,6.1133
+451,30400,5.9823
+455,30800,8.4017
+462,31200,5.0732
+468,31600,5.7932
+474,32000,6.708
+479,32400,6.8853
+485,32800,9.0958
+490,33200,6.4707
+495,33600,7.4192
+501,34000,6.0651
+506,34400,6.7035
+511,34800,6.8316
+518,35200,5.9218
+524,35600,7.2932
+528,36000,8.8676
+534,36400,8.8016
+538,36800,8.7465
+544,37200,8.41
+549,37600,8.3048
+555,38000,8.0676
+561,38400,6.0523
+566,38800,8.8973
+571,39200,8.7871
+577,39600,6.6108
+582,40000,10.0771
+587,40400,7.6992
+594,40800,5.8859
+602,41200,5.2757
+607,41600,7.4604
+614,42000,6.3297
+624,42400,4.3
+630,42800,7.3462
+636,43200,7.7832
+642,43600,7.2278
+649,44000,7.0903
+656,44400,7.2864
+660,44800,11.4337
+664,45200,9.4219
+669,45600,10.1337
+674,46000,10.8808
+679,46400,9.5832
+686,46800,6.8856
+692,47200,7.9853
+699,47600,5.4185
+705,48000,8.51
+711,48400,8.7795
+717,48800,8.1386
+721,49200,12.1496
+727,49600,9.671
+734,50000,4.7004
+742,50400,5.4595
+751,50800,7.1566
+756,51200,11.0405
+763,51600,7.8217
+769,52000,6.9623
+774,52400,9.7564
+780,52800,8.2737
+786,53200,7.7905
+792,53600,8.2016
+798,54000,11.1113
+803,54400,9.71
+811,54800,6.3473
+816,55200,10.3967
+822,55600,8.5624
+829,56000,7.9157
+836,56400,7.3681
+843,56800,7.9999
+848,57200,12.6498
+855,57600,6.7851
+864,58000,5.926
+871,58400,8.3211
+877,58800,8.4122
+885,59200,5.5055
+892,59600,4.8152
+896,60000,10.2298
+903,60400,8.8526
+908,60800,10.4385
+914,61200,6.8265
+920,61600,7.4231
+926,62000,7.7046
+932,62400,8.744
+939,62800,6.815
+944,63200,8.9794
+950,63600,9.0955
+960,64000,3.9011
+966,64400,9.1169
+970,64800,9.8831
+975,65200,10.7317
+983,65600,5.231
+988,66000,9.8312
+993,66400,8.1545
+999,66800,8.3283
+1008,67200,3.3866
+1014,67600,6.2135
+1018,68000,8.4834
+1025,68400,4.2607
+1031,68800,7.755
+1035,69200,6.7764
+1043,69600,4.8484
+1049,70000,6.8767
+1055,70400,7.2876
+1061,70800,6.5337
+1066,71200,8.7095
+1071,71600,8.5315
+1076,72000,8.1106
+1082,72400,6.5303
+1088,72800,5.1858
+1092,73200,8.2166
+1097,73600,7.7642
+1103,74000,5.7689
+1109,74400,6.5443
+1113,74800,6.0629
+1119,75200,6.9554
+1124,75600,6.5107
+1131,76000,5.8779
+1136,76400,7.0196
+1141,76800,8.221
+1146,77200,7.0804
+1150,77600,7.0824
+1158,78000,4.9257
+1165,78400,4.3861
+1171,78800,5.412
+1178,79200,5.8524
+1186,79600,4.6331
+1193,80000,5.5782
+1199,80400,7.2403
+1204,80800,8.5925
+1208,81200,9.0404
+1212,81600,10.5095
+1220,82000,7.0118
+1225,82400,8.1882
+1229,82800,10.863
+1233,83200,10.7497
+1239,83600,6.4536
+1243,84000,11.2907
+1247,84400,11.783
+1252,84800,11.0893
+1257,85200,9.8707
+1261,85600,13.0076
+1265,86000,12.8858
+1269,86400,13.1408
+1273,86800,11.8421
+1277,87200,9.9549
+1284,87600,6.7063
+1288,88000,12.4549
+1292,88400,13.2602
+1299,88800,7.2956
+1303,89200,15.6022
+1308,89600,12.0928
+1317,90000,3.9969
+1323,90400,6.6326
+1331,90800,6.7157
+1337,91200,6.7272
+1346,91600,4.2204
+1351,92000,10.5966
+1357,92400,6.7511
+1364,92800,6.8195
+1371,93200,6.92
+1381,93600,4.6135
+1387,94000,9.0048
+1395,94400,4.8047
+1401,94800,9.5174
+1407,95200,11.6457
+1414,95600,6.9109
+1421,96000,5.7336
+1430,96400,5.3074
+1434,96800,9.2165
+1439,97200,9.5083
+1447,97600,5.2046
+1456,98000,4.3764
+1462,98400,5.923
+1467,98800,8.1644
+1473,99200,6.7967
+1480,99600,7.7647
+1484,100000,12.2281
+1488,100400,9.1569
+1494,100800,8.54
+1499,101200,9.4563
+1503,101600,9.5965
+1509,102000,8.6554
+1514,102400,11.0492
+1518,102800,10.4916
+1525,103200,5.7317
+1534,103600,5.4161
+1539,104000,9.7729
+1546,104400,6.9356
+1552,104800,7.0423
+1556,105200,12.5092
+1563,105600,8.7948
+1567,106000,12.3567
+1573,106400,7.8177
+1577,106800,12.8039
+1584,107200,6.8714
+1589,107600,8.3869
+1593,108000,12.6683
+1601,108400,6.5902
+1607,108800,8.628
+1615,109200,5.4762
+1621,109600,9.3018
+1627,110000,7.4751
+1640,110400,4.139
+1647,110800,7.7327
+1652,111200,10.3567
+1662,111600,4.9881
+1668,112000,8.8983
+1673,112400,9.1326
+1683,112800,3.9693
+1693,113200,4.1639
+1699,113600,8.137
+1705,114000,8.6208
+1710,114400,9.1387
+1717,114800,6.4969
+1724,115200,5.8328
+1729,115600,9.6428
+1736,116000,7.7885
+1742,116400,6.1858
+1751,116800,5.3759
+1760,117200,4.9088
+1764,117600,12.8121
+1773,118000,5.528
+1781,118400,4.8728
+1786,118800,9.4107
+1794,119200,5.6412
+1800,119600,8.3779
+1804,120000,14.4553
+1814,120400,4.32
+1819,120800,12.7509
+1828,121200,6.4023
+1833,121600,15.743
+1841,122000,6.9661
+1847,122400,11.3894
+1852,122800,12.2086
+1857,123200,16.0304
+1862,123600,16.1933
+1870,124000,8.6231
+1875,124400,13.6078
+1880,124800,14.3245
+1889,125200,5.8726
+1896,125600,9.655
+1904,126000,6.397
+1912,126400,5.4703
+1917,126800,13.7367
+1923,127200,10.1322
+1928,127600,12.9031
+1932,128000,11.6047
+1941,128400,8.1528
+1948,128800,10.0696
+1953,129200,12.8557
+1964,129600,6.1629
+1974,130000,4.7264
+1983,130400,5.0263
+1989,130800,13.0645
+2000,131200,4.9296
+2006,131600,9.139
+2016,132000,6.4189
+2025,132400,4.758
+2033,132800,7.2135
+2038,133200,8.544
+2045,133600,9.7969
+2051,134000,7.3074
+2063,134400,4.5675
+2072,134800,4.937
+2080,135200,6.392
+2088,135600,4.6451
+2093,136000,9.6252
+2100,136400,5.984
+2106,136800,7.3098
+2114,137200,6.8205
+2122,137600,5.1513
+2130,138000,6.8115
+2140,138400,4.8287
+2149,138800,3.6011
+2157,139200,5.5075
+2163,139600,9.1481
+2171,140000,6.788
+2178,140400,7.9736
+2186,140800,8.5882
+2191,141200,9.2546
+2196,141600,15.4158
+2206,142000,7.1935
+2211,142400,13.2408
+2223,142800,4.9066
+2235,143200,4.5763
+2241,143600,10.3808
+2246,144000,10.8564
+2254,144400,7.7899
+2261,144800,8.8404
+2268,145200,8.8255
+2276,145600,7.4552
+2282,146000,11.216
+2289,146400,10.299
+2295,146800,9.2089
+2302,147200,8.5813
+2311,147600,5.5125
+2318,148000,7.7001
+2323,148400,11.1793
+2333,148800,3.7442
+2340,149200,10.413
+2345,149600,11.7132
+2350,150000,9.5357
+2357,150400,8.7914
+2363,150800,8.3319
+2370,151200,7.1484
+2376,151600,5.6768
+2385,152000,4.1424
+2390,152400,6.528
+2394,152800,7.7268
+2399,153200,8.4871
+2406,153600,6.8809
+2411,154000,8.3506
+2415,154400,11.0307
+2419,154800,10.397
+2427,155200,5.1522
+2434,155600,5.8036
+2439,156000,10.0986
+2445,156400,7.7494
+2452,156800,8.7844
+2458,157200,9.1953
+2464,157600,9.7125
+2469,158000,14.3923
+2475,158400,10.5696
+2479,158800,19.5742
+2488,159200,6.9904
+2494,159600,12.7756
+2499,160000,12.9402
+2506,160400,11.8509
+2510,160800,14.1441
+2522,161200,5.8916
+2530,161600,8.0032
+2536,162000,11.8227
+2540,162400,14.6518
+2548,162800,9.0002
+2555,163200,5.4754
+2560,163600,15.7747
+2568,164000,9.1886
+2572,164400,18.4005
+2580,164800,6.8589
+2586,165200,13.1998
+2591,165600,12.0732
+2599,166000,7.623
+2608,166400,6.2975
+2614,166800,10.325
+2622,167200,7.3189
+2628,167600,11.0537
+2636,168000,8.6657
+2642,168400,13.2533
+2650,168800,5.9973
+2660,169200,7.9659
+2665,169600,14.8961
+2673,170000,9.1592
+2680,170400,12.4329
+2687,170800,12.6719
+2694,171200,11.0606
+2702,171600,10.5086
+2709,172000,10.1179
+2716,172400,9.1879
+2723,172800,11.2291
+2728,173200,12.3325
+2734,173600,12.1718
+2741,174000,10.1937
+2748,174400,9.3497
+2753,174800,10.5969
+2761,175200,10.1798
+2767,175600,10.9461
+2772,176000,13.9387
+2777,176400,14.2562
+2783,176800,8.5326
+2789,177200,9.8383
+2794,177600,12.2766
+2800,178000,14.2094
+2804,178400,15.156
+2811,178800,6.4663
+2816,179200,15.0295
+2823,179600,10.1727
+2828,180000,11.4667
+2834,180400,11.0373
+2840,180800,11.0221
+2844,181200,18.4976
+2850,181600,9.7416
+2857,182000,8.77
+2863,182400,7.9617
+2867,182800,21.1088
+2872,183200,15.0862
+2880,183600,9.2334
+2885,184000,15.0575
+2891,184400,8.7998
+2898,184800,9.4749
+2903,185200,15.2583
+2908,185600,15.9073
+2913,186000,18.1103
+2920,186400,10.0962
+2925,186800,14.1606
+2931,187200,13.2483
+2936,187600,9.6115
+2942,188000,10.5395
+2948,188400,13.0603
+2952,188800,15.9296
+2958,189200,11.0247
+2964,189600,13.2235
+2973,190000,6.3575
+2978,190400,11.5839
+2983,190800,13.9557
+2989,191200,9.2667
+2995,191600,10.0113
+3000,192000,10.4833
+3006,192400,11.3706
+3012,192800,10.4763
+3018,193200,10.3391
+3022,193600,16.8413
+3030,194000,9.8189
+3042,194400,4.0255
+3049,194800,8.3469
+3058,195200,6.7786
+3066,195600,9.455
+3072,196000,9.361
+3078,196400,11.942
+3086,196800,8.1976
+3093,197200,8.8319
+3098,197600,10.2572
+3105,198000,10.4746
+3111,198400,9.7314
+3117,198800,13.439
+3121,199200,15.2623
+3127,199600,15.3384
+3137,200000,8.1123

code/Lake application/logs/frozen_lake_PPO/PPO_frozen_lake_log_2.csv

@@ -0,0 +1,501 @@
+episode,timestep,reward
+19,400,1.6097
+38,800,1.6416
+61,1200,1.4003
+73,1600,2.5901
+81,2000,3.7322
+90,2400,3.1764
+95,2800,6.9871
+102,3200,4.2802
+109,3600,4.7436
+116,4000,4.2818
+120,4400,7.5307
+124,4800,6.3049
+132,5200,4.3085
+137,5600,6.2239
+141,6000,7.0954
+145,6400,6.7039
+150,6800,6.2469
+155,7200,6.36
+160,7600,6.1549
+164,8000,6.9545
+169,8400,7.0148
+175,8800,4.7251
+179,9200,7.9361
+184,9600,6.5351
+189,10000,6.2774
+195,10400,5.7017
+199,10800,7.825
+203,11200,6.8724
+207,11600,7.7336
+211,12000,7.5364
+215,12400,7.8172
+219,12800,7.9367
+223,13200,7.7311
+228,13600,7.2277
+232,14000,7.0049
+238,14400,5.1928
+242,14800,7.6765
+246,15200,7.4842
+250,15600,7.5239
+254,16000,7.1122
+260,16400,5.5361
+265,16800,7.1787
+272,17200,4.4245
+276,17600,6.2406
+280,18000,7.7256
+286,18400,5.6842
+291,18800,6.646
+297,19200,5.5573
+302,19600,4.9828
+308,20000,5.9812
+313,20400,6.4346
+320,20800,4.4117
+324,21200,6.6082
+330,21600,6.2485
+334,22000,7.8137
+339,22400,5.5175
+343,22800,7.8811
+347,23200,7.1971
+353,23600,5.196
+358,24000,6.0231
+364,24400,6.6306
+369,24800,5.7511
+374,25200,6.1309
+378,25600,7.6626
+382,26000,6.8129
+387,26400,6.4101
+391,26800,8.1621
+395,27200,7.6024
+399,27600,7.6139
+404,28000,7.434
+409,28400,7.5108
+413,28800,10.0048
+417,29200,8.9736
+423,29600,5.5721
+428,30000,5.6607
+433,30400,7.3198
+437,30800,7.5887
+441,31200,7.1714
+446,31600,5.5443
+452,32000,6.0095
+456,32400,6.5919
+461,32800,6.9332
+465,33200,5.9742
+472,33600,4.6549
+476,34000,6.7065
+481,34400,6.6351
+485,34800,7.3781
+491,35200,5.3901
+495,35600,7.481
+499,36000,7.3904
+504,36400,5.5203
+508,36800,6.6256
+513,37200,6.2715
+519,37600,5.4467
+523,38000,7.5914
+528,38400,5.8271
+535,38800,4.7497
+539,39200,6.705
+543,39600,7.2193
+547,40000,7.6591
+553,40400,5.4713
+559,40800,5.6278
+563,41200,7.0377
+567,41600,6.5975
+572,42000,6.4589
+577,42400,6.4962
+581,42800,7.5891
+585,43200,7.5598
+593,43600,3.9839
+597,44000,6.7188
+601,44400,6.9495
+606,44800,7.4883
+613,45200,3.7567
+617,45600,7.7218
+622,46000,6.69
+626,46400,7.9262
+630,46800,6.3034
+635,47200,6.0545
+639,47600,7.4195
+644,48000,7.13
+648,48400,7.0398
+652,48800,7.6392
+660,49200,3.7038
+667,49600,4.3578
+672,50000,5.9194
+678,50400,4.8665
+683,50800,6.3008
+691,51200,4.5975
+697,51600,5.1888
+702,52000,5.7734
+707,52400,7.4679
+715,52800,3.3284
+721,53200,6.0641
+727,53600,6.0441
+731,54000,7.6702
+736,54400,7.4219
+740,54800,6.222
+746,55200,6.4839
+751,55600,5.6146
+755,56000,7.5972
+761,56400,5.612
+765,56800,7.8148
+770,57200,5.7253
+774,57600,7.9334
+778,58000,8.3276
+783,58400,6.8913
+787,58800,7.2828
+793,59200,6.9596
+797,59600,8.2329
+803,60000,6.4227
+808,60400,6.7123
+812,60800,7.9976
+819,61200,5.5859
+826,61600,4.4553
+832,62000,6.7451
+837,62400,6.241
+843,62800,6.4673
+848,63200,6.9543
+855,63600,5.0913
+861,64000,7.2159
+865,64400,7.9432
+870,64800,7.8044
+875,65200,7.2243
+880,65600,7.7488
+886,66000,6.3843
+890,66400,9.9507
+894,66800,9.063
+899,67200,7.5112
+903,67600,7.6428
+908,68000,8.2787
+912,68400,8.5889
+918,68800,5.723
+922,69200,8.2975
+926,69600,7.3824
+931,70000,8.2068
+937,70400,7.7218
+941,70800,8.7229
+945,71200,10.2961
+950,71600,7.6121
+955,72000,7.8974
+960,72400,7.8978
+967,72800,5.6801
+979,73200,2.1174
+990,73600,3.7935
+995,74000,10.1194
+1000,74400,6.8977
+1006,74800,7.6098
+1011,75200,7.2811
+1017,75600,7.2474
+1023,76000,8.5673
+1029,76400,6.6197
+1035,76800,8.3262
+1041,77200,7.9771
+1047,77600,7.1542
+1051,78000,11.2335
+1056,78400,8.8322
+1062,78800,7.5779
+1066,79200,13.4042
+1071,79600,10.4175
+1076,80000,13.3333
+1080,80400,12.734
+1086,80800,11.1026
+1090,81200,14.7759
+1096,81600,8.6412
+1104,82000,6.3892
+1110,82400,9.8357
+1118,82800,7.1741
+1125,83200,7.8912
+1131,83600,8.455
+1135,84000,14.1645
+1140,84400,9.4645
+1146,84800,10.1664
+1153,85200,5.8022
+1160,85600,6.215
+1168,86000,4.1062
+1175,86400,6.1067
+1180,86800,6.3874
+1187,87200,6.3452
+1192,87600,6.9666
+1198,88000,5.1382
+1203,88400,6.1001
+1212,88800,4.8099
+1218,89200,7.4769
+1226,89600,7.5724
+1231,90000,10.0412
+1238,90400,8.6996
+1242,90800,13.2195
+1247,91200,11.4526
+1252,91600,12.0908
+1260,92000,6.7688
+1265,92400,10.449
+1270,92800,10.2558
+1277,93200,5.5992
+1284,93600,9.3672
+1292,94000,7.979
+1302,94400,6.4251
+1308,94800,8.6478
+1314,95200,12.1168
+1322,95600,6.8864
+1333,96000,5.5147
+1339,96400,9.0327
+1347,96800,6.0256
+1356,97200,4.7894
+1365,97600,5.5282
+1371,98000,7.7578
+1380,98400,8.3785
+1389,98800,5.239
+1395,99200,8.3681
+1400,99600,9.7949
+1406,100000,9.5216
+1414,100400,5.3299
+1420,100800,7.8871
+1428,101200,6.0441
+1434,101600,6.6909
+1444,102000,5.0341
+1450,102400,7.7649
+1461,102800,4.1021
+1472,103200,2.9832
+1482,103600,4.4576
+1489,104000,7.3442
+1497,104400,5.6498
+1505,104800,6.4131
+1511,105200,10.5803
+1516,105600,11.3073
+1520,106000,13.2675
+1527,106400,10.6688
+1535,106800,5.8265
+1540,107200,10.7056
+1545,107600,7.6742
+1553,108000,5.7264
+1560,108400,6.6821
+1567,108800,5.4216
+1573,109200,8.4766
+1578,109600,8.6739
+1582,110000,11.4666
+1586,110400,9.0832
+1594,110800,6.2276
+1599,111200,8.6866
+1606,111600,6.2615
+1614,112000,5.6982
+1621,112400,5.8051
+1626,112800,7.93
+1631,113200,10.595
+1636,113600,7.6407
+1640,114000,11.1847
+1647,114400,5.6078
+1651,114800,9.1446
+1658,115200,6.8432
+1662,115600,12.9911
+1671,116000,4.613
+1676,116400,11.0305
+1681,116800,8.694
+1688,117200,6.11
+1698,117600,6.4954
+1703,118000,9.7062
+1712,118400,5.8668
+1717,118800,7.5547
+1724,119200,8.3224
+1729,119600,8.329
+1737,120000,7.1094
+1742,120400,9.2663
+1747,120800,9.6127
+1757,121200,5.6769
+1762,121600,8.5658
+1770,122000,7.4468
+1776,122400,8.0238
+1782,122800,8.0927
+1790,123200,5.6296
+1798,123600,6.9246
+1806,124000,6.5561
+1813,124400,7.4058
+1821,124800,6.9491
+1833,125200,4.4584
+1839,125600,8.2732
+1848,126000,6.3892
+1853,126400,10.4589
+1858,126800,12.1972
+1867,127200,5.6719
+1874,127600,7.9428
+1880,128000,7.5333
+1885,128400,12.1847
+1889,128800,11.591
+1895,129200,9.8666
+1902,129600,9.6165
+1910,130000,8.469
+1917,130400,5.6391
+1924,130800,8.5008
+1931,131200,9.7077
+1935,131600,13.5229
+1939,132000,16.4664
+1944,132400,13.0046
+1949,132800,8.4371
+1955,133200,8.8647
+1959,133600,14.3521
+1963,134000,11.7871
+1968,134400,10.4688
+1974,134800,9.3431
+1979,135200,8.226
+1984,135600,10.8513
+1988,136000,10.6682
+1998,136400,4.4273
+2003,136800,12.778
+2011,137200,8.0067
+2017,137600,9.4886
+2027,138000,5.6532
+2033,138400,7.9827
+2039,138800,5.9282
+2045,139200,9.6567
+2053,139600,7.7935
+2059,140000,5.9489
+2064,140400,6.9939
+2070,140800,7.3466
+2075,141200,6.8939
+2079,141600,9.1796
+2085,142000,9.0719
+2089,142400,13.9325
+2094,142800,11.0623
+2099,143200,6.7284
+2107,143600,5.6728
+2115,144000,6.321
+2119,144400,10.3033
+2124,144800,8.3466
+2130,145200,6.2657
+2136,145600,5.7428
+2142,146000,7.4074
+2147,146400,7.8209
+2153,146800,7.3415
+2161,147200,5.2917
+2166,147600,12.5109
+2174,148000,5.0395
+2179,148400,10.3555
+2187,148800,5.349
+2192,149200,10.0253
+2198,149600,7.0379
+2205,150000,7.5125
+2211,150400,7.8038
+2217,150800,7.8167
+2221,151200,11.7039
+2229,151600,5.489
+2236,152000,6.0868
+2243,152400,6.2941
+2249,152800,6.6384
+2254,153200,8.3705
+2259,153600,8.5642
+2265,154000,6.8584
+2272,154400,7.3834
+2278,154800,8.2766
+2286,155200,5.1656
+2290,155600,12.9118
+2294,156000,11.8071
+2299,156400,8.9169
+2303,156800,8.9791
+2308,157200,9.3741
+2312,157600,11.1361
+2316,158000,11.7926
+2320,158400,11.7151
+2326,158800,8.0207
+2333,159200,6.0691
+2338,159600,9.577
+2344,160000,6.4232
+2350,160400,8.8049
+2354,160800,9.1235
+2359,161200,8.3861
+2365,161600,7.0742
+2369,162000,7.6221
+2373,162400,7.9897
+2378,162800,5.8857
+2384,163200,7.0723
+2389,163600,6.654
+2395,164000,7.1041
+2399,164400,8.4728
+2407,164800,4.6772
+2411,165200,7.5167
+2417,165600,7.8597
+2422,166000,9.3692
+2427,166400,8.0704
+2431,166800,10.7773
+2435,167200,9.905
+2440,167600,8.5513
+2444,168000,8.9629
+2450,168400,7.6352
+2457,168800,6.9678
+2463,169200,7.899
+2469,169600,7.6206
+2474,170000,8.0358
+2481,170400,7.7953
+2486,170800,10.2047
+2492,171200,12.0048
+2500,171600,5.3765
+2509,172000,5.1863
+2513,172400,12.2984
+2517,172800,11.7797
+2524,173200,7.875
+2534,173600,3.9569
+2539,174000,11.8781
+2544,174400,10.3608
+2550,174800,8.7275
+2556,175200,8.4987
+2561,175600,10.7181
+2566,176000,8.2619
+2571,176400,8.8454
+2575,176800,8.7559
+2580,177200,8.4039
+2584,177600,11.9876
+2592,178000,6.4634
+2597,178400,7.8467
+2602,178800,6.8881
+2607,179200,7.1522
+2612,179600,7.6053
+2620,180000,5.2739
+2628,180400,4.7562
+2636,180800,3.7301
+2641,181200,10.6831
+2645,181600,8.9943
+2652,182000,5.277
+2661,182400,4.275
+2671,182800,3.453
+2676,183200,8.5581
+2681,183600,7.2929
+2686,184000,12.0382
+2690,184400,12.133
+2694,184800,12.2167
+2701,185200,5.3062
+2706,185600,10.3605
+2714,186000,5.9351
+2718,186400,9.3307
+2723,186800,9.5625
+2730,187200,7.1449
+2735,187600,7.7182
+2741,188000,6.4756
+2745,188400,8.736
+2751,188800,8.123
+2757,189200,8.8964
+2761,189600,10.7888
+2765,190000,12.0779
+2769,190400,10.5991
+2775,190800,8.0868
+2780,191200,8.917
+2785,191600,9.9389
+2789,192000,11.5622
+2795,192400,8.386
+2800,192800,8.3721
+2805,193200,8.8166
+2811,193600,8.1763
+2817,194000,8.3255
+2823,194400,8.8563
+2827,194800,12.6773
+2833,195200,7.9132
+2837,195600,10.4681
+2844,196000,8.2131
+2849,196400,9.2795
+2855,196800,7.094
+2860,197200,10.5194
+2864,197600,12.9269
+2870,198000,7.9327
+2876,198400,7.223
+2881,198800,10.8269
+2885,199200,12.3889
+2892,199600,6.5769
+2897,200000,11.076

code/Lake application/logs/frozen_lake_PPO/PPO_frozen_lake_log_3.csv

@@ -0,0 +1,501 @@
+episode,timestep,reward
+15,400,1.8261
+32,800,1.7685
+45,1200,2.4142
+51,1600,4.6704
+61,2000,3.1899
+67,2400,5.3735
+75,2800,4.1154
+82,3200,3.7725
+92,3600,3.7704
+103,4000,2.6905
+110,4400,4.0276
+117,4800,5.8064
+124,5200,4.4713
+132,5600,4.0093
+137,6000,5.9495
+142,6400,5.8014
+148,6800,5.8551
+155,7200,4.6169
+162,7600,4.2494
+170,8000,3.9722
+177,8400,3.9192
+186,8800,3.8894
+192,9200,5.2858
+196,9600,6.1068
+201,10000,7.4978
+206,10400,6.5813
+210,10800,7.0333
+216,11200,5.7853
+220,11600,6.9548
+226,12000,6.9295
+231,12400,8.6606
+235,12800,8.1602
+241,13200,5.5479
+248,13600,5.2991
+253,14000,5.5222
+259,14400,6.6097
+266,14800,5.7681
+272,15200,5.1424
+281,15600,3.9703
+289,16000,5.446
+296,16400,5.6469
+301,16800,8.1023
+309,17200,5.4118
+313,17600,10.0884
+318,18000,7.85
+324,18400,7.1867
+334,18800,4.1973
+341,19200,6.5775
+346,19600,8.8721
+353,20000,8.3356
+358,20400,8.4092
+364,20800,7.0598
+369,21200,9.7722
+375,21600,8.7407
+381,22000,7.1701
+387,22400,6.8198
+391,22800,11.2731
+399,23200,6.9213
+404,23600,7.6675
+408,24000,8.5927
+412,24400,9.5454
+416,24800,11.0606
+421,25200,11.0094
+426,25600,9.2321
+431,26000,7.2778
+436,26400,10.7602
+440,26800,10.2041
+444,27200,10.7586
+448,27600,8.5868
+452,28000,9.4227
+458,28400,8.901
+462,28800,9.059
+467,29200,9.2612
+472,29600,10.802
+478,30000,7.4041
+483,30400,8.8905
+489,30800,6.3011
+493,31200,12.0364
+497,31600,11.5981
+501,32000,11.5024
+505,32400,11.9104
+509,32800,11.9212
+513,33200,12.0185
+518,33600,9.4035
+522,34000,11.5176
+526,34400,11.0239
+534,34800,6.1402
+538,35200,9.7348
+546,35600,6.35
+551,36000,7.8883
+558,36400,7.7455
+563,36800,7.1019
+571,37200,6.6505
+576,37600,7.5348
+581,38000,12.1861
+586,38400,9.1601
+591,38800,8.1292
+596,39200,7.3226
+602,39600,9.3
+608,40000,10.1455
+614,40400,7.404
+620,40800,9.5543
+627,41200,8.0328
+636,41600,5.051
+648,42000,4.3144
+654,42400,8.7103
+661,42800,8.5619
+666,43200,9.0912
+671,43600,12.1562
+679,44000,6.8929
+683,44400,12.4673
+690,44800,7.4547
+700,45200,6.1627
+708,45600,5.2344
+712,46000,14.522
+718,46400,9.7264
+724,46800,9.4083
+731,47200,7.3673
+735,47600,10.918
+741,48000,9.7135
+746,48400,11.6226
+753,48800,6.5335
+760,49200,6.1922
+765,49600,11.59
+772,50000,7.6406
+779,50400,7.3931
+785,50800,8.8649
+790,51200,13.0236
+796,51600,9.1355
+802,52000,9.2798
+812,52400,4.6073
+818,52800,8.5625
+823,53200,8.0732
+829,53600,8.2494
+837,54000,5.0721
+849,54400,3.926
+857,54800,5.9843
+866,55200,5.4496
+872,55600,9.6436
+877,56000,9.8259
+882,56400,12.0831
+886,56800,11.8707
+892,57200,9.3723
+897,57600,8.75
+902,58000,9.1673
+908,58400,8.2213
+919,58800,3.6353
+929,59200,3.9628
+935,59600,6.8984
+942,60000,6.928
+948,60400,8.007
+954,60800,7.1696
+962,61200,6.8068
+970,61600,5.7813
+979,62000,6.7075
+990,62400,4.5979
+995,62800,11.0131
+1001,63200,9.9881
+1007,63600,8.771
+1013,64000,6.4708
+1020,64400,8.4602
+1024,64800,12.5658
+1029,65200,12.6734
+1033,65600,13.9195
+1037,66000,10.7454
+1043,66400,10.9443
+1048,66800,10.9429
+1053,67200,9.4126
+1059,67600,8.414
+1066,68000,6.8977
+1071,68400,8.4342
+1077,68800,6.9781
+1081,69200,9.3134
+1087,69600,8.1705
+1091,70000,8.8618
+1096,70400,10.7669
+1100,70800,10.662
+1104,71200,9.404
+1108,71600,10.9212
+1114,72000,7.7906
+1120,72400,6.9977
+1124,72800,10.0251
+1130,73200,8.4128
+1134,73600,9.9691
+1138,74000,11.4181
+1143,74400,8.6228
+1152,74800,5.3614
+1157,75200,8.1655
+1164,75600,7.1774
+1171,76000,5.9159
+1180,76400,4.0023
+1189,76800,4.7476
+1197,77200,5.5766
+1202,77600,8.0878
+1209,78000,6.6897
+1213,78400,13.7633
+1221,78800,6.482
+1230,79200,6.0141
+1234,79600,12.2026
+1242,80000,5.6537
+1251,80400,4.3695
+1259,80800,7.4921
+1264,81200,9.8077
+1269,81600,10.9606
+1275,82000,9.6273
+1280,82400,12.195
+1287,82800,7.4125
+1292,83200,12.1273
+1296,83600,13.0822
+1303,84000,7.0237
+1308,84400,11.6651
+1313,84800,9.4606
+1318,85200,12.5532
+1324,85600,9.9701
+1329,86000,11.7337
+1333,86400,14.7119
+1339,86800,9.0294
+1345,87200,7.5184
+1353,87600,7.3597
+1359,88000,10.1495
+1365,88400,8.7107
+1375,88800,4.0411
+1383,89200,5.6476
+1389,89600,6.5996
+1397,90000,8.0321
+1404,90400,8.9973
+1410,90800,8.9065
+1417,91200,5.712
+1427,91600,4.6027
+1436,92000,5.296
+1441,92400,7.6209
+1449,92800,7.4688
+1456,93200,5.0893
+1464,93600,6.9208
+1472,94000,7.4646
+1479,94400,8.2095
+1484,94800,9.7461
+1490,95200,8.781
+1494,95600,8.3202
+1499,96000,9.0412
+1506,96400,6.9789
+1511,96800,7.9199
+1517,97200,10.6694
+1524,97600,6.481
+1532,98000,6.7254
+1541,98400,6.2726
+1547,98800,7.5851
+1555,99200,5.3696
+1560,99600,10.4877
+1567,100000,8.8012
+1577,100400,5.5168
+1584,100800,10.155
+1588,101200,8.5114
+1595,101600,6.3359
+1602,102000,6.6452
+1608,102400,6.8052
+1612,102800,8.0109
+1617,103200,5.8693
+1621,103600,8.5857
+1626,104000,9.6799
+1634,104400,5.6426
+1639,104800,6.5151
+1644,105200,7.8849
+1648,105600,8.9338
+1654,106000,5.6678
+1659,106400,6.3244
+1665,106800,5.6218
+1669,107200,9.5578
+1676,107600,5.3338
+1686,108000,4.6969
+1693,108400,5.4701
+1698,108800,10.4455
+1704,109200,8.6891
+1710,109600,10.0138
+1717,110000,7.5448
+1726,110400,7.8171
+1731,110800,10.4446
+1739,111200,8.1385
+1748,111600,6.0488
+1756,112000,7.1346
+1761,112400,9.6157
+1767,112800,9.1395
+1774,113200,8.8882
+1782,113600,8.0369
+1789,114000,9.7018
+1795,114400,8.5519
+1802,114800,12.4735
+1810,115200,6.0752
+1815,115600,11.7469
+1824,116000,5.4505
+1829,116400,9.3351
+1835,116800,10.7987
+1840,117200,15.1342
+1846,117600,14.5398
+1853,118000,10.6334
+1861,118400,11.3101
+1866,118800,14.8907
+1873,119200,9.6076
+1882,119600,7.7126
+1893,120000,4.7907
+1901,120400,6.0066
+1906,120800,15.4955
+1911,121200,13.3978
+1919,121600,9.6642
+1924,122000,15.7393
+1928,122400,18.9361
+1932,122800,19.4331
+1940,123200,8.6073
+1947,123600,10.161
+1954,124000,8.8061
+1959,124400,13.5384
+1966,124800,9.9289
+1981,125200,4.1506
+1991,125600,8.5539
+1995,126000,20.6911
+2002,126400,11.9305
+2007,126800,16.2045
+2016,127200,6.5353
+2021,127600,17.3839
+2027,128000,13.1571
+2035,128400,11.3393
+2039,128800,22.3028
+2047,129200,12.8433
+2055,129600,12.7583
+2066,130000,6.9837
+2071,130400,18.0303
+2076,130800,18.2896
+2084,131200,8.7988
+2093,131600,8.7713
+2098,132000,11.0877
+2109,132400,7.8183
+2115,132800,12.4679
+2124,133200,10.5047
+2137,133600,6.17
+2142,134000,20.3565
+2148,134400,10.4538
+2154,134800,12.7331
+2161,135200,12.5367
+2171,135600,5.7754
+2178,136000,13.3435
+2182,136400,13.3376
+2195,136800,5.0278
+2203,137200,8.039
+2215,137600,5.0622
+2225,138000,7.6281
+2232,138400,12.4199
+2243,138800,5.7324
+2249,139200,14.5818
+2255,139600,14.0929
+2262,140000,13.6329
+2267,140400,18.3515
+2272,140800,18.0695
+2280,141200,12.0349
+2287,141600,13.6652
+2296,142000,9.2929
+2305,142400,10.1985
+2312,142800,12.7522
+2323,143200,7.2459
+2331,143600,8.9751
+2338,144000,11.4881
+2344,144400,15.2227
+2351,144800,12.8927
+2358,145200,10.6543
+2362,145600,22.496
+2368,146000,13.9616
+2373,146400,18.1932
+2378,146800,16.1787
+2382,147200,21.2142
+2386,147600,22.1002
+2396,148000,8.9528
+2401,148400,15.8869
+2408,148800,13.7149
+2413,149200,13.7033
+2419,149600,17.2193
+2425,150000,11.3894
+2432,150400,13.8544
+2437,150800,17.5939
+2444,151200,12.8075
+2449,151600,12.1515
+2457,152000,10.4033
+2465,152400,11.4859
+2470,152800,14.4762
+2477,153200,12.3627
+2483,153600,14.8347
+2488,154000,18.2382
+2497,154400,9.2311
+2501,154800,19.7235
+2509,155200,13.3697
+2515,155600,11.9598
+2525,156000,7.2526
+2534,156400,9.3025
+2545,156800,8.9835
+2551,157200,12.1765
+2558,157600,14.0303
+2564,158000,13.4739
+2573,158400,9.8322
+2578,158800,19.6338
+2584,159200,15.7125
+2588,159600,17.0086
+2594,160000,14.7127
+2598,160400,23.1588
+2607,160800,11.0373
+2615,161200,9.348
+2619,161600,21.6514
+2624,162000,12.9316
+2631,162400,12.1088
+2636,162800,20.0918
+2640,163200,18.6887
+2644,163600,19.3577
+2653,164000,8.5057
+2662,164400,8.0083
+2668,164800,15.0007
+2676,165200,8.8861
+2682,165600,15.3621
+2689,166000,13.6995
+2696,166400,10.5381
+2701,166800,19.4263
+2708,167200,12.0695
+2713,167600,11.9025
+2719,168000,10.0897
+2725,168400,15.0383
+2731,168800,14.8992
+2735,169200,14.9242
+2739,169600,20.4302
+2745,170000,15.2987
+2750,170400,16.7812
+2754,170800,16.0345
+2759,171200,16.4285
+2768,171600,9.1208
+2780,172000,5.583
+2787,172400,8.2014
+2793,172800,10.1961
+2798,173200,17.1725
+2806,173600,9.115
+2814,174000,9.2754
+2821,174400,10.8946
+2827,174800,11.5879
+2834,175200,10.3869
+2840,175600,13.8918
+2845,176000,12.5769
+2854,176400,10.322
+2863,176800,6.8967
+2869,177200,17.4846
+2874,177600,19.6151
+2881,178000,12.9361
+2886,178400,18.2368
+2892,178800,12.8876
+2898,179200,12.3181
+2903,179600,17.6907
+2908,180000,15.7174
+2915,180400,11.7662
+2920,180800,17.438
+2925,181200,14.2649
+2931,181600,12.5882
+2936,182000,17.2888
+2942,182400,15.7864
+2946,182800,19.7236
+2952,183200,14.7757
+2957,183600,13.2554
+2962,184000,16.9161
+2966,184400,19.4477
+2971,184800,14.806
+2976,185200,14.7174
+2981,185600,14.6584
+2985,186000,13.1555
+2993,186400,8.2998
+2999,186800,10.4079
+3004,187200,14.5865
+3011,187600,9.0036
+3015,188000,13.7298
+3022,188400,8.8899
+3026,188800,15.7034
+3032,189200,11.7676
+3036,189600,17.0897
+3044,190000,10.1182
+3049,190400,13.9028
+3054,190800,16.7113
+3059,191200,16.4022
+3064,191600,18.3592
+3069,192000,17.6439
+3074,192400,15.5535
+3079,192800,15.6137
+3085,193200,14.7975
+3091,193600,11.9988
+3099,194000,10.8644
+3106,194400,14.6474
+3113,194800,12.833
+3117,195200,22.6677
+3121,195600,21.1913
+3126,196000,17.2308
+3131,196400,14.7737
+3140,196800,6.5658
+3147,197200,8.1251
+3155,197600,8.3248
+3162,198000,10.0096
+3168,198400,9.34
+3175,198800,9.7054
+3181,199200,9.9748
+3186,199600,11.6185
+3191,200000,10.6864

code/Lake application/logs/frozen_lake_PPO/PPO_frozen_lake_log_4.csv

@@ -0,0 +1,501 @@
+episode,timestep,reward
+30,400,0.8897
+47,800,1.6985
+70,1200,1.5344
+81,1600,2.8837
+93,2000,2.7853
+99,2400,4.9228
+105,2800,4.8879
+109,3200,6.7205
+116,3600,5.1037
+121,4000,6.8188
+126,4400,5.7358
+130,4800,7.5711
+136,5200,5.2419
+141,5600,5.8543
+147,6000,5.2331
+152,6400,4.9293
+161,6800,4.0985
+166,7200,5.2082
+172,7600,5.3103
+177,8000,6.0557
+181,8400,6.9039
+188,8800,4.5342
+192,9200,9.1478
+199,9600,5.3605
+206,10000,5.1273
+211,10400,7.8887
+216,10800,6.2693
+220,11200,7.3964
+224,11600,7.1054
+230,12000,5.4118
+234,12400,7.4287
+238,12800,7.4478
+245,13200,4.4117
+249,13600,7.2105
+253,14000,6.7015
+257,14400,7.4679
+262,14800,6.5292
+267,15200,6.025
+271,15600,7.6356
+275,16000,7.6501
+280,16400,7.0316
+284,16800,8.6843
+289,17200,6.7495
+294,17600,6.023
+299,18000,6.415
+305,18400,5.4163
+311,18800,5.4882
+316,19200,6.2097
+320,19600,8.0329
+324,20000,7.5882
+329,20400,7.3095
+333,20800,7.5687
+339,21200,6.6201
+343,21600,9.5695
+349,22000,6.358
+354,22400,7.729
+362,22800,3.8919
+367,23200,6.2263
+372,23600,8.9296
+377,24000,8.9107
+383,24400,6.8109
+391,24800,4.3733
+397,25200,7.8942
+402,25600,8.1473
+408,26000,6.342
+414,26400,6.6643
+419,26800,8.5145
+423,27200,9.8487
+427,27600,8.3884
+432,28000,8.1417
+438,28400,6.5363
+442,28800,8.3084
+447,29200,7.5203
+451,29600,8.0109
+455,30000,7.1375
+459,30400,6.972
+464,30800,7.2792
+468,31200,8.3772
+472,31600,8.6912
+476,32000,7.6424
+480,32400,8.1047
+484,32800,6.6454
+489,33200,7.4736
+493,33600,8.1904
+497,34000,7.1456
+503,34400,6.2541
+507,34800,7.7885
+511,35200,7.3507
+515,35600,8.0471
+520,36000,8.5436
+524,36400,6.8725
+529,36800,8.4028
+535,37200,5.2433
+542,37600,4.7139
+546,38000,7.3213
+555,38400,3.8831
+561,38800,5.6601
+568,39200,4.7948
+576,39600,4.6981
+584,40000,4.3181
+589,40400,7.5472
+593,40800,9.3392
+602,41200,3.9924
+609,41600,7.1339
+615,42000,6.7132
+620,42400,6.7015
+628,42800,5.4925
+636,43200,3.7468
+644,43600,4.3569
+651,44000,5.8671
+655,44400,8.3115
+660,44800,9.1009
+665,45200,7.2625
+672,45600,5.378
+678,46000,5.686
+684,46400,5.8378
+688,46800,7.674
+693,47200,7.7574
+697,47600,9.4904
+706,48000,4.0155
+712,48400,9.193
+718,48800,6.0672
+723,49200,8.703
+729,49600,6.4219
+737,50000,5.2146
+742,50400,7.6968
+747,50800,9.559
+753,51200,6.7111
+758,51600,9.7201
+764,52000,7.5954
+770,52400,8.0675
+775,52800,7.1163
+782,53200,5.3886
+786,53600,10.9581
+791,54000,9.5825
+800,54400,4.9313
+808,54800,3.2748
+813,55200,9.4975
+819,55600,8.5919
+828,56000,4.0659
+834,56400,6.4677
+839,56800,8.6157
+847,57200,7.6231
+854,57600,6.1867
+864,58000,5.138
+875,58400,4.1107
+884,58800,4.6541
+890,59200,8.6775
+898,59600,4.5193
+903,60000,10.8015
+909,60400,7.2792
+916,60800,6.9898
+920,61200,9.2429
+926,61600,7.8279
+930,62000,9.559
+938,62400,6.2201
+942,62800,12.4695
+949,63200,6.0011
+955,63600,7.5678
+960,64000,8.5841
+965,64400,8.8059
+969,64800,9.559
+974,65200,8.137
+979,65600,6.2258
+985,66000,6.0418
+990,66400,7.6972
+994,66800,10.6031
+999,67200,6.4527
+1004,67600,7.6003
+1009,68000,8.036
+1014,68400,11.559
+1018,68800,9.9028
+1024,69200,8.8209
+1030,69600,6.8682
+1034,70000,9.1513
+1039,70400,8.8808
+1045,70800,6.2892
+1052,71200,6.6137
+1056,71600,9.1258
+1061,72000,8.2712
+1069,72400,5.1346
+1073,72800,9.3301
+1079,73200,8.6006
+1083,73600,9.7199
+1088,74000,8.5393
+1093,74400,9.4136
+1098,74800,9.2309
+1104,75200,9.9483
+1111,75600,6.868
+1117,76000,7.3642
+1123,76400,8.7512
+1128,76800,7.7363
+1133,77200,11.2048
+1138,77600,8.7672
+1142,78000,11.2022
+1148,78400,8.6267
+1156,78800,5.7085
+1165,79200,5.2502
+1171,79600,9.1847
+1181,80000,3.8257
+1187,80400,9.0944
+1193,80800,7.8396
+1204,81200,4.6099
+1214,81600,4.2423
+1223,82000,4.2463
+1231,82400,6.0866
+1236,82800,9.3631
+1244,83200,6.0139
+1250,83600,9.3664
+1258,84000,5.5445
+1263,84400,9.2419
+1270,84800,6.5126
+1277,85200,5.3284
+1284,85600,4.7442
+1289,86000,6.6238
+1295,86400,5.9787
+1300,86800,6.273
+1304,87200,7.7552
+1310,87600,6.0625
+1315,88000,6.2442
+1320,88400,7.2193
+1326,88800,5.8607
+1333,89200,5.8177
+1338,89600,5.6992
+1344,90000,6.309
+1350,90400,7.4904
+1357,90800,5.3341
+1362,91200,9.1276
+1370,91600,5.6335
+1377,92000,5.5404
+1382,92400,10.4014
+1387,92800,8.972
+1393,93200,7.6199
+1400,93600,7.0028
+1408,94000,6.7953
+1417,94400,4.607
+1425,94800,6.7686
+1431,95200,6.3672
+1437,95600,7.3133
+1442,96000,5.5286
+1449,96400,8.1326
+1454,96800,6.6459
+1459,97200,9.6138
+1465,97600,8.3167
+1470,98000,11.7781
+1474,98400,17.2643
+1481,98800,9.2478
+1490,99200,8.6222
+1497,99600,10.0905
+1504,100000,7.0319
+1510,100400,11.7434
+1515,100800,12.7016
+1519,101200,14.5775
+1525,101600,7.4347
+1531,102000,17.0148
+1535,102400,21.5497
+1541,102800,13.3595
+1546,103200,20.2214
+1553,103600,12.2627
+1562,104000,10.1528
+1572,104400,7.0324
+1577,104800,13.988
+1587,105200,9.7205
+1591,105600,22.2021
+1599,106000,12.8697
+1611,106400,6.7423
+1618,106800,9.1311
+1624,107200,13.5325
+1630,107600,15.3574
+1637,108000,11.4605
+1647,108400,8.1593
+1657,108800,7.3692
+1664,109200,12.4204
+1671,109600,16.0635
+1677,110000,11.0747
+1687,110400,10.0776
+1695,110800,9.713
+1704,111200,6.6402
+1709,111600,16.4947
+1714,112000,12.0573
+1720,112400,12.4928
+1726,112800,16.9818
+1731,113200,16.4082
+1735,113600,19.4684
+1741,114000,17.2942
+1746,114400,17.3803
+1752,114800,14.3429
+1759,115200,15.4686
+1764,115600,18.3797
+1771,116000,10.6607
+1778,116400,12.9278
+1783,116800,21.2477
+1789,117200,12.1737
+1795,117600,15.1248
+1801,118000,11.4594
+1808,118400,11.8572
+1816,118800,8.6953
+1822,119200,12.9991
+1830,119600,8.346
+1835,120000,16.5775
+1840,120400,18.7012
+1845,120800,18.5211
+1852,121200,12.3151
+1860,121600,11.191
+1868,122000,10.6305
+1875,122400,12.1361
+1883,122800,12.0561
+1887,123200,21.1206
+1895,123600,10.3102
+1901,124000,15.5468
+1905,124400,21.1214
+1913,124800,12.598
+1919,125200,8.6702
+1923,125600,19.5976
+1928,126000,17.347
+1936,126400,12.0519
+1944,126800,6.2953
+1949,127200,13.6435
+1956,127600,9.3424
+1960,128000,22.6692
+1966,128400,12.2863
+1973,128800,15.4013
+1978,129200,17.9858
+1988,129600,7.2154
+1996,130000,10.964
+2004,130400,10.9658
+2009,130800,16.1921
+2015,131200,19.8994
+2020,131600,12.5598
+2026,132000,18.5603
+2034,132400,8.9442
+2039,132800,15.7247
+2044,133200,19.6043
+2048,133600,22.708
+2055,134000,12.1769
+2059,134400,30.2886
+2064,134800,19.3976
+2069,135200,24.011
+2075,135600,22.3232
+2079,136000,22.4054
+2087,136400,14.8207
+2095,136800,14.1154
+2102,137200,13.3378
+2106,137600,22.9892
+2112,138000,19.1975
+2119,138400,16.2562
+2125,138800,16.5325
+2134,139200,9.7804
+2143,139600,12.9261
+2149,140000,15.1729
+2157,140400,11.4505
+2163,140800,16.225
+2168,141200,15.0464
+2175,141600,12.2286
+2181,142000,14.5324
+2187,142400,17.9193
+2192,142800,21.9792
+2202,143200,7.5693
+2214,143600,7.0395
+2219,144000,20.2988
+2230,144400,8.1503
+2237,144800,12.8959
+2246,145200,11.8272
+2254,145600,15.8534
+2259,146000,20.5079
+2266,146400,10.7379
+2271,146800,21.599
+2279,147200,12.679
+2284,147600,14.8514
+2291,148000,8.6118
+2297,148400,7.5502
+2306,148800,5.1645
+2313,149200,10.6152
+2319,149600,15.1497
+2324,150000,13.6594
+2331,150400,10.1251
+2337,150800,10.9294
+2345,151200,5.0712
+2350,151600,13.3293
+2359,152000,6.394
+2370,152400,4.9969
+2379,152800,7.9595
+2391,153200,4.0272
+2398,153600,7.3762
+2407,154000,7.3333
+2413,154400,10.8586
+2423,154800,9.7345
+2432,155200,7.9822
+2439,155600,10.3486
+2450,156000,6.3284
+2458,156400,9.4372
+2473,156800,3.132
+2481,157200,10.3754
+2487,157600,10.447
+2493,158000,15.9101
+2503,158400,8.3842
+2510,158800,15.4866
+2518,159200,11.682
+2526,159600,11.0361
+2537,160000,8.7871
+2545,160400,11.1971
+2555,160800,4.4022
+2562,161200,13.1779
+2568,161600,12.9045
+2579,162000,8.0949
+2587,162400,11.4998
+2597,162800,7.2315
+2604,163200,14.4484
+2609,163600,15.2563
+2622,164000,7.1052
+2631,164400,8.0708
+2636,164800,15.3412
+2646,165200,7.0698
+2654,165600,10.9479
+2661,166000,12.6783
+2666,166400,16.1794
+2671,166800,18.6718
+2676,167200,13.5239
+2681,167600,12.3507
+2691,168000,7.2502
+2700,168400,6.2099
+2707,168800,13.9091
+2712,169200,16.7988
+2717,169600,22.2866
+2728,170000,8.0224
+2738,170400,11.7132
+2743,170800,22.9338
+2750,171200,15.1354
+2762,171600,8.9432
+2768,172000,14.02
+2773,172400,19.0923
+2780,172800,14.6205
+2787,173200,12.9528
+2796,173600,11.7497
+2803,174000,14.0874
+2808,174400,21.8773
+2813,174800,16.507
+2819,175200,19.8032
+2827,175600,13.8242
+2833,176000,19.0166
+2838,176400,27.172
+2846,176800,11.3364
+2851,177200,19.2692
+2857,177600,21.4003
+2865,178000,13.1762
+2871,178400,20.7159
+2878,178800,12.615
+2882,179200,17.292
+2886,179600,22.37
+2893,180000,16.9824
+2903,180400,7.4276
+2912,180800,12.7024
+2918,181200,14.4444
+2926,181600,14.2302
+2932,182000,18.124
+2938,182400,13.5542
+2943,182800,33.0073
+2950,183200,12.0173
+2954,183600,22.766
+2959,184000,16.2093
+2965,184400,15.9019
+2972,184800,13.7782
+2976,185200,26.2554
+2985,185600,10.2687
+2991,186000,14.767
+2998,186400,12.8508
+3002,186800,19.9929
+3008,187200,16.2693
+3016,187600,14.6283
+3026,188000,9.5896
+3034,188400,11.7475
+3041,188800,12.2039
+3045,189200,23.8742
+3056,189600,8.3229
+3063,190000,11.436
+3070,190400,10.1774
+3077,190800,10.9202
+3082,191200,16.139
+3087,191600,17.0197
+3096,192000,6.3408
+3103,192400,10.6383
+3108,192800,14.7467
+3113,193200,16.4579
+3121,193600,9.5428
+3125,194000,22.9954
+3129,194400,11.4853
+3136,194800,8.377
+3141,195200,13.0133
+3145,195600,14.8357
+3150,196000,20.832
+3159,196400,8.4116
+3166,196800,16.5597
+3178,197200,6.8153
+3186,197600,11.2551
+3197,198000,9.8681
+3202,198400,16.0912
+3210,198800,13.4439
+3216,199200,11.4534
+3222,199600,16.3593
+3229,200000,11.4777

code/Lake application/logs/frozen_lake_PPO/PPO_frozen_lake_log_5.csv

@@ -0,0 +1,501 @@
+episode,timestep,reward
+25,400,1.1394
+48,800,1.3002
+55,1200,4.2174
+65,1600,3.1185
+72,2000,4.6995
+77,2400,5.4912
+83,2800,5.3572
+87,3200,6.4659
+93,3600,5.7893
+97,4000,7.5009
+101,4400,7.0671
+105,4800,6.9336
+109,5200,7.0397
+114,5600,6.2202
+120,6000,4.966
+126,6400,5.4569
+131,6800,5.7776
+135,7200,7.3573
+139,7600,6.3645
+144,8000,6.1568
+148,8400,7.3496
+152,8800,6.4834
+157,9200,6.0306
+162,9600,6.5141
+167,10000,6.2576
+172,10400,5.8399
+176,10800,5.7246
+182,11200,5.6336
+186,11600,6.3594
+192,12000,5.9243
+196,12400,7.3959
+200,12800,7.3087
+204,13200,7.4288
+208,13600,6.0092
+212,14000,7.4051
+219,14400,5.0751
+223,14800,6.1337
+228,15200,6.0328
+233,15600,6.2323
+237,16000,7.2947
+242,16400,6.21
+247,16800,5.2467
+251,17200,7.301
+256,17600,6.7575
+260,18000,7.3051
+264,18400,7.3659
+270,18800,4.9364
+275,19200,4.7764
+280,19600,6.6525
+286,20000,5.1666
+290,20400,7.305
+295,20800,5.3332
+301,21200,5.8009
+306,21600,6.3444
+311,22000,5.38
+316,22400,5.8886
+321,22800,5.9808
+326,23200,6.1518
+331,23600,6.2596
+335,24000,7.4699
+340,24400,5.0459
+344,24800,7.1979
+348,25200,7.1906
+354,25600,5.5289
+358,26000,7.4249
+362,26400,7.3209
+366,26800,6.8339
+371,27200,6.2896
+375,27600,5.8839
+382,28000,4.5678
+387,28400,6.4948
+391,28800,7.3334
+395,29200,6.8561
+400,29600,5.7921
+405,30000,5.775
+411,30400,5.2429
+416,30800,6.3733
+420,31200,6.9394
+424,31600,7.4063
+430,32000,5.6989
+434,32400,6.6094
+439,32800,6.4591
+443,33200,7.5158
+448,33600,5.1683
+457,34000,3.2012
+462,34400,6.4385
+466,34800,7.6264
+471,35200,6.8552
+475,35600,6.6808
+479,36000,7.5093
+485,36400,6.0369
+491,36800,5.9012
+497,37200,4.7016
+507,37600,3.1521
+513,38000,5.874
+517,38400,7.713
+522,38800,6.4352
+526,39200,7.6985
+530,39600,7.8557
+540,40000,3.9398
+546,40400,5.5794
+551,40800,6.7952
+555,41200,7.4607
+561,41600,6.2385
+565,42000,6.9421
+571,42400,5.5255
+577,42800,7.0349
+583,43200,7.038
+587,43600,8.1308
+592,44000,6.715
+598,44400,6.6102
+603,44800,5.7423
+609,45200,5.7222
+616,45600,4.6904
+621,46000,6.6802
+627,46400,5.1794
+631,46800,7.8428
+636,47200,6.6569
+640,47600,6.0637
+645,48000,6.6306
+649,48400,7.8827
+654,48800,7.2829
+660,49200,5.3484
+666,49600,4.965
+671,50000,6.2847
+675,50400,7.2723
+681,50800,5.6506
+689,51200,4.2316
+693,51600,6.588
+699,52000,5.8609
+704,52400,6.7229
+710,52800,5.2104
+716,53200,5.5105
+720,53600,6.7428
+728,54000,4.2067
+733,54400,6.1901
+737,54800,6.1446
+743,55200,6.4328
+747,55600,7.5352
+753,56000,4.6297
+758,56400,6.7488
+765,56800,4.4836
+769,57200,7.332
+773,57600,7.3174
+778,58000,7.5865
+784,58400,5.1354
+788,58800,7.5499
+792,59200,7.6182
+802,59600,2.9436
+807,60000,6.7174
+812,60400,6.5712
+820,60800,3.8267
+827,61200,5.0311
+831,61600,7.6775
+835,62000,6.2429
+842,62400,5.1519
+847,62800,5.1346
+858,63200,3.2845
+862,63600,6.7612
+868,64000,5.4555
+874,64400,5.6836
+879,64800,5.4058
+883,65200,7.5245
+887,65600,7.5208
+892,66000,7.063
+897,66400,6.6028
+903,66800,6.9216
+908,67200,7.874
+912,67600,6.5846
+918,68000,6.0015
+923,68400,6.5042
+927,68800,6.4518
+932,69200,7.5216
+937,69600,7.0083
+942,70000,6.8853
+948,70400,5.5392
+954,70800,8.848
+960,71200,11.4058
+967,71600,7.5975
+972,72000,8.9093
+979,72400,9.266
+985,72800,9.023
+991,73200,11.5379
+1000,73600,7.4839
+1006,74000,10.8982
+1011,74400,9.6794
+1016,74800,11.5398
+1022,75200,12.7577
+1032,75600,7.5257
+1040,76000,10.313
+1052,76400,4.9592
+1057,76800,9.929
+1065,77200,9.0269
+1074,77600,10.0283
+1080,78000,9.1994
+1084,78400,17.318
+1090,78800,8.6919
+1099,79200,8.3069
+1104,79600,14.1304
+1110,80000,13.6171
+1114,80400,16.2281
+1119,80800,13.7672
+1127,81200,9.2344
+1133,81600,12.9287
+1139,82000,9.0991
+1146,82400,10.4085
+1154,82800,6.4297
+1161,83200,11.2657
+1170,83600,8.6181
+1176,84000,10.1173
+1182,84400,10.5116
+1189,84800,7.5418
+1197,85200,7.8979
+1204,85600,10.4355
+1214,86000,5.9039
+1228,86400,4.1987
+1238,86800,6.374
+1246,87200,5.9424
+1251,87600,15.9749
+1257,88000,14.0111
+1261,88400,19.8135
+1270,88800,7.7016
+1276,89200,10.2966
+1281,89600,12.6069
+1288,90000,10.6588
+1293,90400,17.1633
+1300,90800,9.8388
+1308,91200,9.1061
+1314,91600,10.2858
+1319,92000,15.5991
+1323,92400,19.9744
+1329,92800,13.7349
+1333,93200,16.2973
+1340,93600,12.1433
+1346,94000,15.6216
+1350,94400,16.604
+1356,94800,17.9473
+1360,95200,22.7261
+1367,95600,9.7798
+1372,96000,18.8177
+1376,96400,23.3835
+1380,96800,22.7676
+1385,97200,17.9196
+1391,97600,11.0045
+1398,98000,12.9032
+1403,98400,19.9428
+1411,98800,10.0649
+1415,99200,23.4105
+1421,99600,14.4659
+1427,100000,12.6086
+1434,100400,11.2016
+1440,100800,16.4195
+1445,101200,14.8528
+1451,101600,13.3423
+1457,102000,15.6651
+1462,102400,19.3589
+1467,102800,19.2814
+1472,103200,17.4048
+1479,103600,10.9156
+1484,104000,19.1347
+1489,104400,16.7974
+1500,104800,7.7077
+1508,105200,8.3873
+1515,105600,11.8502
+1522,106000,12.4617
+1529,106400,10.997
+1534,106800,18.1086
+1538,107200,21.5753
+1542,107600,18.1229
+1548,108000,19.0807
+1553,108400,19.9151
+1557,108800,24.3347
+1565,109200,11.5838
+1571,109600,10.4892
+1576,110000,18.4124
+1583,110400,9.6659
+1589,110800,15.3845
+1594,111200,19.4332
+1603,111600,9.1848
+1608,112000,19.8579
+1614,112400,14.6327
+1620,112800,15.4716
+1628,113200,7.6968
+1633,113600,14.4689
+1637,114000,19.6793
+1642,114400,20.0721
+1647,114800,15.0668
+1652,115200,17.4454
+1657,115600,19.6026
+1663,116000,14.572
+1669,116400,12.3857
+1675,116800,15.0434
+1679,117200,23.0521
+1685,117600,15.9115
+1691,118000,16.4641
+1695,118400,18.5005
+1701,118800,13.3055
+1705,119200,20.5855
+1711,119600,15.2568
+1716,120000,17.1653
+1721,120400,16.2964
+1726,120800,17.3911
+1731,121200,18.9176
+1735,121600,20.2643
+1741,122000,21.9711
+1748,122400,15.1474
+1752,122800,21.0002
+1756,123200,27.017
+1761,123600,21.0847
+1768,124000,16.9835
+1774,124400,16.988
+1780,124800,17.102
+1784,125200,27.0598
+1790,125600,18.8929
+1795,126000,18.4346
+1799,126400,27.4704
+1803,126800,20.451
+1812,127200,12.5548
+1816,127600,24.9355
+1821,128000,14.002
+1826,128400,16.9177
+1834,128800,9.9734
+1847,129200,4.5403
+1853,129600,10.6147
+1865,130000,6.26
+1872,130400,12.1423
+1877,130800,16.8818
+1882,131200,14.1034
+1887,131600,19.5902
+1894,132000,12.8515
+1899,132400,16.2843
+1904,132800,15.5745
+1914,133200,8.3905
+1922,133600,13.0687
+1929,134000,13.9548
+1935,134400,11.7435
+1944,134800,12.2644
+1949,135200,18.9015
+1957,135600,10.5449
+1968,136000,6.614
+1979,136400,7.9006
+1988,136800,8.6919
+1993,137200,19.6558
+1999,137600,13.7705
+2004,138000,19.7431
+2010,138400,16.1015
+2018,138800,7.969
+2024,139200,10.7627
+2033,139600,9.8075
+2038,140000,15.1353
+2044,140400,14.33
+2051,140800,13.0915
+2059,141200,11.0496
+2067,141600,8.5425
+2074,142000,12.6574
+2079,142400,18.6865
+2083,142800,18.9614
+2091,143200,7.6956
+2097,143600,19.3319
+2106,144000,6.1586
+2112,144400,11.7879
+2117,144800,14.6574
+2124,145200,9.78
+2131,145600,8.172
+2138,146000,9.3161
+2145,146400,10.1464
+2151,146800,13.3546
+2158,147200,10.2643
+2162,147600,17.7297
+2167,148000,12.2066
+2174,148400,11.723
+2181,148800,12.61
+2185,149200,20.9512
+2192,149600,9.257
+2200,150000,13.0471
+2206,150400,10.6689
+2212,150800,16.0447
+2219,151200,13.6559
+2225,151600,13.2487
+2235,152000,7.2764
+2242,152400,11.6686
+2248,152800,12.3615
+2255,153200,13.5621
+2263,153600,9.6251
+2269,154000,9.0672
+2276,154400,13.0372
+2281,154800,16.5969
+2286,155200,18.0225
+2292,155600,14.2052
+2298,156000,11.6988
+2304,156400,9.5336
+2312,156800,8.0191
+2324,157200,5.3825
+2330,157600,8.1571
+2337,158000,10.3493
+2344,158400,10.4621
+2350,158800,10.5959
+2356,159200,7.2691
+2364,159600,6.3992
+2372,160000,7.9295
+2377,160400,7.4555
+2384,160800,7.9996
+2389,161200,13.726
+2395,161600,7.7046
+2399,162000,16.8889
+2410,162400,5.528
+2422,162800,4.9575
+2429,163200,8.7608
+2438,163600,7.2575
+2446,164000,7.0835
+2452,164400,10.8246
+2459,164800,7.831
+2467,165200,6.133
+2476,165600,7.8923
+2483,166000,8.0733
+2489,166400,11.0754
+2493,166800,20.1624
+2500,167200,12.4293
+2504,167600,15.0355
+2510,168000,13.0286
+2518,168400,7.4877
+2523,168800,12.2261
+2531,169200,7.3993
+2536,169600,8.9622
+2542,170000,10.8549
+2547,170400,11.6566
+2555,170800,8.8997
+2562,171200,8.861
+2568,171600,13.2091
+2574,172000,10.3659
+2580,172400,11.7853
+2585,172800,14.792
+2592,173200,10.6782
+2602,173600,6.9546
+2609,174000,9.9301
+2614,174400,17.7772
+2623,174800,6.2142
+2630,175200,12.9292
+2637,175600,10.1204
+2645,176000,9.0597
+2651,176400,15.3755
+2657,176800,13.128
+2662,177200,19.3868
+2670,177600,10.7437
+2679,178000,6.904
+2686,178400,9.2907
+2696,178800,6.4837
+2705,179200,8.2248
+2711,179600,12.1069
+2720,180000,8.3973
+2724,180400,22.0167
+2729,180800,15.3768
+2734,181200,15.6707
+2738,181600,21.4503
+2744,182000,14.5199
+2750,182400,16.9138
+2756,182800,12.2078
+2762,183200,15.9948
+2769,183600,12.4933
+2775,184000,14.7625
+2781,184400,16.4597
+2786,184800,12.6036
+2793,185200,11.1748
+2799,185600,13.5976
+2805,186000,13.175
+2811,186400,14.25
+2816,186800,22.0337
+2822,187200,17.4297
+2827,187600,17.1395
+2832,188000,18.1786
+2837,188400,16.0257
+2844,188800,11.8928
+2850,189200,16.6968
+2855,189600,19.1383
+2860,190000,21.8792
+2864,190400,27.2875
+2868,190800,25.2937
+2873,191200,20.9754
+2882,191600,12.5236
+2886,192000,26.9158
+2896,192400,9.8619
+2912,192800,5.0885
+2923,193200,4.6341
+2930,193600,13.8767
+2937,194000,11.1766
+2944,194400,14.2145
+2952,194800,7.6092
+2961,195200,9.0705
+2968,195600,10.5332
+2973,196000,13.6747
+2979,196400,17.6262
+2986,196800,12.0028
+2997,197200,5.1463
+3008,197600,9.2429
+3015,198000,11.3805
+3022,198400,7.9289
+3029,198800,12.3534
+3033,199200,26.1072
+3041,199600,13.3515
+3048,200000,11.5205

code/Lake application/logs/results_1/PDPPO_frozen_lake_log_1.csv

@@ -0,0 +1,501 @@
+episode,timestep,reward
+23,400,1.2374
+44,800,1.475
+62,1200,1.6597
+75,1600,2.2131
+86,2000,2.8599
+98,2400,2.246
+109,2800,3.0708
+120,3200,3.0907
+134,3600,2.2625
+148,4000,2.1622
+162,4400,2.2727
+177,4800,1.635
+192,5200,2.4163
+203,5600,2.6421
+211,6000,4.1822
+219,6400,3.2539
+229,6800,3.5347
+238,7200,3.3297
+246,7600,3.7071
+254,8000,3.3388
+263,8400,3.4999
+272,8800,3.3479
+279,9200,3.8583
+284,9600,6.0999
+292,10000,3.6754
+301,10400,3.3741
+307,10800,5.4199
+315,11200,3.3395
+320,11600,6.3554
+329,12000,3.8476
+333,12400,7.4853
+340,12800,4.5367
+345,13200,5.1413
+350,13600,5.9491
+355,14000,6.9454
+362,14400,4.5899
+366,14800,7.3912
+371,15200,5.3159
+378,15600,4.6648
+382,16000,6.0774
+387,16400,6.5447
+393,16800,4.8197
+401,17200,4.653
+405,17600,7.3019
+410,18000,6.4426
+415,18400,6.2152
+420,18800,5.3748
+424,19200,7.3437
+428,19600,6.2297
+432,20000,7.2185
+438,20400,5.2351
+442,20800,6.6825
+448,21200,4.8845
+455,21600,4.6814
+459,22000,7.2773
+463,22400,7.2303
+467,22800,7.3327
+474,23200,4.6069
+479,23600,5.9596
+483,24000,7.4753
+487,24400,5.694
+492,24800,6.0146
+497,25200,6.2868
+501,25600,7.4234
+505,26000,7.4502
+509,26400,7.501
+514,26800,6.3456
+518,27200,6.6853
+523,27600,6.0334
+527,28000,7.6037
+531,28400,7.5199
+535,28800,7.0901
+540,29200,6.0599
+545,29600,7.4143
+549,30000,7.4906
+553,30400,7.4618
+558,30800,5.9038
+562,31200,7.5903
+567,31600,5.8108
+572,32000,6.3685
+577,32400,6.0121
+581,32800,7.5516
+585,33200,7.5793
+589,33600,6.2426
+593,34000,7.6518
+597,34400,7.5643
+601,34800,7.5213
+606,35200,7.1744
+610,35600,7.5771
+614,36000,7.4704
+618,36400,8.3538
+622,36800,6.0225
+626,37200,7.6223
+631,37600,6.5412
+636,38000,6.5792
+641,38400,6.4666
+645,38800,6.1159
+649,39200,7.8856
+653,39600,7.7125
+658,40000,7.68
+662,40400,6.3203
+666,40800,7.7558
+672,41200,5.3449
+678,41600,5.3048
+683,42000,6.0098
+687,42400,7.7948
+693,42800,5.2185
+698,43200,6.8124
+703,43600,7.1864
+707,44000,6.9499
+712,44400,7.3481
+716,44800,6.9955
+722,45200,5.2644
+727,45600,7.1592
+731,46000,7.8775
+736,46400,6.8166
+740,46800,7.7517
+745,47200,6.382
+749,47600,9.0147
+755,48000,5.9239
+760,48400,6.0407
+766,48800,5.7661
+770,49200,7.6221
+777,49600,4.4481
+782,50000,6.4733
+786,50400,7.9791
+790,50800,6.8551
+797,51200,5.3038
+801,51600,6.3203
+808,52000,4.4792
+813,52400,7.2173
+817,52800,7.9572
+821,53200,7.8356
+826,53600,6.6151
+830,54000,6.719
+835,54400,6.5328
+839,54800,7.9386
+843,55200,7.8856
+847,55600,7.8485
+852,56000,6.087
+858,56400,5.8267
+862,56800,6.6001
+869,57200,5.4358
+874,57600,5.1707
+879,58000,6.6192
+883,58400,7.9342
+887,58800,7.1165
+891,59200,7.814
+896,59600,6.5746
+900,60000,7.7367
+905,60400,7.0371
+911,60800,6.1385
+915,61200,6.8106
+920,61600,6.758
+924,62000,8.4806
+928,62400,7.9427
+933,62800,7.618
+937,63200,7.3321
+941,63600,7.6414
+945,64000,7.5861
+952,64400,5.5493
+956,64800,7.9559
+960,65200,7.2965
+966,65600,6.0452
+971,66000,6.785
+975,66400,7.7964
+979,66800,7.9213
+983,67200,6.4723
+987,67600,8.0026
+992,68000,7.5901
+996,68400,7.9942
+1001,68800,5.7542
+1007,69200,5.245
+1011,69600,8.1308
+1016,70000,6.861
+1026,70400,4.9051
+1033,70800,4.1145
+1039,71200,8.9367
+1045,71600,6.8587
+1051,72000,6.208
+1056,72400,9.627
+1062,72800,6.7777
+1068,73200,8.1144
+1072,73600,10.0016
+1078,74000,8.3931
+1083,74400,8.5617
+1090,74800,6.4565
+1096,75200,7.7086
+1102,75600,8.0011
+1108,76000,7.8719
+1117,76400,5.4102
+1125,76800,5.3871
+1132,77200,6.1708
+1138,77600,9.6164
+1143,78000,9.3242
+1151,78400,6.2785
+1156,78800,9.268
+1163,79200,7.2284
+1169,79600,7.7116
+1174,80000,10.1517
+1179,80400,10.7016
+1184,80800,9.2447
+1189,81200,9.5792
+1194,81600,10.5578
+1202,82000,6.7523
+1206,82400,10.3011
+1214,82800,8.1824
+1219,83200,9.1268
+1224,83600,9.169
+1229,84000,11.3677
+1234,84400,10.2567
+1238,84800,12.7622
+1242,85200,13.0512
+1247,85600,11.1427
+1253,86000,7.929
+1258,86400,10.1415
+1262,86800,13.3642
+1267,87200,10.9775
+1271,87600,12.6409
+1276,88000,10.5992
+1282,88400,9.3828
+1289,88800,7.4124
+1293,89200,11.4734
+1298,89600,12.2623
+1302,90000,10.1244
+1307,90400,9.9046
+1313,90800,9.779
+1319,91200,8.0214
+1325,91600,8.5971
+1331,92000,9.6304
+1336,92400,8.3368
+1341,92800,9.7023
+1345,93200,13.0409
+1350,93600,11.4912
+1356,94000,8.8157
+1360,94400,11.3592
+1366,94800,8.2193
+1372,95200,10.369
+1377,95600,9.2107
+1381,96000,11.1205
+1387,96400,9.8026
+1392,96800,10.0977
+1397,97200,11.0417
+1401,97600,10.8291
+1406,98000,10.6965
+1413,98400,8.3182
+1417,98800,12.0391
+1422,99200,10.573
+1427,99600,11.3273
+1431,100000,13.3514
+1436,100400,8.3171
+1442,100800,9.6702
+1446,101200,12.2739
+1450,101600,13.6371
+1454,102000,13.3332
+1461,102400,8.6744
+1468,102800,7.1006
+1473,103200,9.9262
+1479,103600,9.9459
+1483,104000,12.8872
+1487,104400,13.5411
+1493,104800,8.3614
+1501,105200,6.8721
+1505,105600,12.9734
+1510,106000,11.0086
+1514,106400,13.0415
+1519,106800,10.641
+1525,107200,9.6682
+1529,107600,11.3172
+1537,108000,7.6408
+1545,108400,7.5169
+1554,108800,5.0547
+1562,109200,8.1673
+1566,109600,11.351
+1572,110000,11.8269
+1576,110400,14.5999
+1581,110800,11.9343
+1587,111200,9.0708
+1599,111600,4.4076
+1604,112000,9.6173
+1610,112400,10.2097
+1615,112800,11.4213
+1619,113200,15.4919
+1627,113600,8.0107
+1633,114000,8.2533
+1637,114400,15.4628
+1643,114800,11.739
+1648,115200,10.4027
+1652,115600,14.405
+1658,116000,10.397
+1662,116400,15.0393
+1667,116800,12.0286
+1673,117200,10.1889
+1678,117600,13.0175
+1682,118000,13.202
+1689,118400,8.8392
+1694,118800,12.0206
+1699,119200,15.2346
+1706,119600,7.4044
+1710,120000,15.8376
+1716,120400,9.405
+1721,120800,12.1716
+1727,121200,10.639
+1731,121600,12.4195
+1735,122000,15.801
+1740,122400,14.7642
+1744,122800,14.9512
+1749,123200,12.8902
+1754,123600,12.6179
+1758,124000,15.3085
+1763,124400,15.2636
+1767,124800,15.9938
+1771,125200,13.2417
+1776,125600,12.6163
+1782,126000,10.864
+1787,126400,12.165
+1792,126800,12.2211
+1796,127200,16.1173
+1802,127600,9.7973
+1811,128000,6.8105
+1818,128400,7.4772
+1824,128800,11.3689
+1830,129200,9.7769
+1837,129600,7.5446
+1842,130000,12.6235
+1848,130400,11.2806
+1852,130800,13.906
+1858,131200,12.9947
+1865,131600,7.8085
+1870,132000,12.0638
+1876,132400,12.8332
+1881,132800,12.0972
+1888,133200,10.0171
+1894,133600,8.1607
+1906,134000,5.2001
+1912,134400,10.769
+1918,134800,10.4093
+1922,135200,14.9802
+1928,135600,12.3873
+1932,136000,16.5533
+1938,136400,10.1507
+1944,136800,11.7987
+1948,137200,16.9859
+1953,137600,12.8739
+1957,138000,15.2543
+1962,138400,13.5113
+1970,138800,9.3558
+1974,139200,17.044
+1980,139600,10.9737
+1985,140000,12.4654
+1991,140400,12.8888
+1996,140800,14.0378
+2003,141200,9.7178
+2008,141600,14.7386
+2013,142000,14.8715
+2020,142400,9.5865
+2025,142800,12.7104
+2029,143200,16.0303
+2034,143600,15.8244
+2038,144000,16.9077
+2042,144400,16.3177
+2046,144800,17.1757
+2050,145200,15.4536
+2054,145600,16.8005
+2063,146000,7.8445
+2067,146400,13.3113
+2074,146800,10.177
+2085,147200,5.7266
+2089,147600,13.1785
+2097,148000,9.0101
+2103,148400,11.5439
+2110,148800,8.3925
+2120,149200,7.0782
+2127,149600,6.752
+2136,150000,8.609
+2141,150400,12.6775
+2147,150800,11.4195
+2152,151200,14.956
+2159,151600,8.3099
+2163,152000,13.0747
+2169,152400,10.8855
+2173,152800,16.1138
+2179,153200,11.1642
+2183,153600,17.5901
+2189,154000,12.9443
+2193,154400,15.0154
+2199,154800,11.8207
+2205,155200,12.6347
+2209,155600,18.7259
+2215,156000,12.7894
+2224,156400,7.0928
+2229,156800,13.0815
+2237,157200,9.53
+2244,157600,8.183
+2250,158000,11.3156
+2254,158400,17.8963
+2264,158800,6.7863
+2271,159200,9.1487
+2278,159600,10.2853
+2285,160000,11.1034
+2291,160400,12.3193
+2296,160800,12.8277
+2304,161200,10.186
+2311,161600,10.0795
+2317,162000,13.2198
+2322,162400,13.6536
+2327,162800,16.2005
+2333,163200,11.8695
+2337,163600,18.9552
+2341,164000,19.3696
+2346,164400,15.9116
+2351,164800,13.0325
+2356,165200,14.5295
+2362,165600,12.7752
+2366,166000,20.2381
+2370,166400,18.5701
+2374,166800,19.7957
+2378,167200,20.6729
+2383,167600,17.1297
+2388,168000,16.415
+2392,168400,16.3995
+2398,168800,14.5399
+2403,169200,14.3127
+2407,169600,20.0631
+2412,170000,14.8381
+2417,170400,16.6159
+2422,170800,15.6721
+2429,171200,11.7749
+2435,171600,12.1638
+2439,172000,20.5949
+2443,172400,18.4239
+2447,172800,20.2025
+2453,173200,15.569
+2459,173600,12.0643
+2466,174000,11.6623
+2471,174400,12.4459
+2479,174800,8.9067
+2483,175200,19.4989
+2487,175600,19.945
+2491,176000,18.5005
+2497,176400,15.8793
+2503,176800,11.8832
+2509,177200,11.3501
+2514,177600,11.7629
+2518,178000,19.6937
+2522,178400,19.9908
+2526,178800,20.0871
+2534,179200,9.8611
+2539,179600,16.1641
+2544,180000,16.6044
+2551,180400,11.9178
+2556,180800,15.7903
+2565,181200,8.1818
+2573,181600,6.7254
+2582,182000,6.4099
+2593,182400,6.8256
+2608,182800,4.9001
+2620,183200,5.3354
+2625,183600,13.1512
+2645,184000,3.785
+2660,184400,3.8907
+2667,184800,8.5334
+2678,185200,6.3167
+2688,185600,6.4812
+2698,186000,5.3833
+2706,186400,9.5081
+2716,186800,6.8215
+2722,187200,10.4982
+2730,187600,9.5086
+2738,188000,9.3461
+2744,188400,14.2225
+2757,188800,5.7347
+2766,189200,7.2451
+2772,189600,13.8376
+2776,190000,19.2079
+2785,190400,8.2667
+2794,190800,7.8716
+2800,191200,10.8995
+2808,191600,11.1967
+2813,192000,13.3437
+2818,192400,15.5833
+2825,192800,12.1659
+2829,193200,18.1718
+2841,193600,6.6037
+2847,194000,13.7069
+2853,194400,12.6538
+2860,194800,12.0022
+2868,195200,8.8985
+2875,195600,12.2925
+2881,196000,11.201
+2889,196400,11.2774
+2897,196800,9.0422
+2903,197200,12.1459
+2908,197600,17.9243
+2914,198000,12.5648
+2923,198400,8.3362
+2931,198800,9.3883
+2935,199200,19.7271
+2941,199600,13.2621
+2945,200000,18.3102

code/Lake application/logs/results_1/PDPPO_frozen_lake_log_2.csv

@@ -0,0 +1,501 @@
+episode,timestep,reward
+13,400,2.4027
+32,800,1.617
+46,1200,2.0829
+55,1600,3.3664
+70,2000,1.9602
+77,2400,4.2734
+87,2800,3.3484
+96,3200,2.8441
+108,3600,2.8373
+113,4000,6.0449
+122,4400,3.834
+128,4800,4.6497
+134,5200,5.7626
+139,5600,5.8112
+148,6000,3.9162
+153,6400,6.4239
+161,6800,3.2877
+167,7200,6.0931
+173,7600,5.6529
+177,8000,8.1896
+183,8400,5.4595
+188,8800,5.5099
+194,9200,5.9555
+200,9600,5.4228
+207,10000,5.1593
+213,10400,5.8843
+221,10800,3.7587
+226,11200,6.9919
+233,11600,5.0004
+239,12000,5.6778
+245,12400,6.7453
+251,12800,5.6725
+257,13200,5.7192
+263,13600,6.5788
+267,14000,7.3344
+272,14400,7.4462
+278,14800,6.0194
+283,15200,7.349
+289,15600,6.475
+294,16000,8.7377
+302,16400,5.5482
+310,16800,4.4978
+314,17200,12.6707
+322,17600,6.9202
+327,18000,8.4112
+332,18400,7.8845
+337,18800,8.8895
+344,19200,7.0861
+350,19600,8.066
+357,20000,6.4104
+364,20400,6.3238
+369,20800,9.4534
+373,21200,10.6902
+378,21600,10.5201
+383,22000,9.18
+387,22400,11.8679
+393,22800,7.6084
+397,23200,11.8894
+402,23600,8.3823
+406,24000,11.8962
+410,24400,11.9462
+414,24800,12.0936
+418,25200,12.2516
+423,25600,11.6307
+427,26000,12.2494
+431,26400,10.3211
+435,26800,12.193
+441,27200,8.2828
+447,27600,8.323
+451,28000,12.0851
+455,28400,11.8691
+460,28800,10.3406
+464,29200,11.2745
+469,29600,9.7434
+475,30000,6.0876
+481,30400,9.8042
+485,30800,9.8166
+491,31200,8.5955
+496,31600,8.8879
+502,32000,9.6651
+506,32400,12.4816
+513,32800,5.3655
+518,33200,10.5211
+523,33600,8.8889
+529,34000,8.49
+533,34400,11.531
+539,34800,8.149
+543,35200,10.8391
+548,35600,10.0555
+552,36000,12.2689
+557,36400,10.1142
+562,36800,11.1685
+568,37200,7.5849
+574,37600,7.9313
+578,38000,12.7468
+582,38400,12.8371
+587,38800,10.0817
+593,39200,7.806
+599,39600,8.1719
+606,40000,7.7715
+610,40400,12.7892
+615,40800,8.8938
+622,41200,6.7361
+630,41600,6.5825
+638,42000,6.4833
+643,42400,8.2059
+649,42800,10.0644
+658,43200,5.4777
+663,43600,8.1624
+668,44000,10.306
+673,44400,13.0668
+681,44800,5.2228
+686,45200,12.7062
+691,45600,8.3513
+698,46000,7.9629
+702,46400,13.0319
+709,46800,6.41
+713,47200,13.8998
+718,47600,10.1406
+723,48000,11.4989
+732,48400,5.7759
+736,48800,13.8137
+745,49200,3.7937
+753,49600,8.4375
+757,50000,13.3919
+764,50400,7.7574
+770,50800,7.3266
+774,51200,14.4457
+782,51600,7.6632
+786,52000,14.5052
+791,52400,12.7646
+796,52800,12.3906
+801,53200,12.5327
+806,53600,9.7097
+812,54000,9.998
+821,54400,6.5391
+826,54800,10.0426
+830,55200,13.2772
+837,55600,9.5483
+841,56000,11.9928
+848,56400,9.3228
+852,56800,13.7227
+858,57200,8.4757
+862,57600,14.4407
+868,58000,9.607
+873,58400,12.6694
+883,58800,4.7655
+888,59200,11.5319
+894,59600,10.6887
+899,60000,11.8105
+904,60400,10.2226
+910,60800,8.0744
+914,61200,14.8116
+919,61600,12.2418
+928,62000,6.0928
+935,62400,8.7412
+942,62800,8.2501
+951,63200,7.3312
+958,63600,6.9865
+963,64000,13.1523
+968,64400,11.9024
+975,64800,10.2803
+979,65200,15.9866
+983,65600,13.8684
+988,66000,13.0459
+993,66400,13.1794
+997,66800,16.858
+1005,67200,9.2472
+1011,67600,10.5944
+1015,68000,18.199
+1019,68400,16.954
+1023,68800,16.9229
+1028,69200,12.5304
+1032,69600,17.6321
+1039,70000,10.8713
+1048,70400,8.1558
+1053,70800,12.2909
+1060,71200,10.5877
+1066,71600,10.6469
+1073,72000,10.9766
+1077,72400,15.5587
+1082,72800,14.5922
+1088,73200,11.1597
+1093,73600,16.6017
+1098,74000,13.7014
+1103,74400,10.6047
+1108,74800,14.9205
+1112,75200,18.965
+1119,75600,12.518
+1126,76000,7.3995
+1131,76400,15.3457
+1136,76800,13.6769
+1140,77200,17.8605
+1144,77600,18.1357
+1151,78000,11.2644
+1156,78400,11.4578
+1161,78800,16.01
+1167,79200,13.5079
+1171,79600,15.4602
+1176,80000,18.2113
+1181,80400,17.2741
+1185,80800,19.4672
+1193,81200,8.1171
+1198,81600,16.5683
+1204,82000,12.9389
+1209,82400,16.3976
+1217,82800,9.9467
+1221,83200,20.1043
+1229,83600,8.9266
+1235,84000,12.0714
+1241,84400,14.5858
+1245,84800,19.3948
+1252,85200,9.0477
+1256,85600,19.9998
+1263,86000,11.7696
+1271,86400,9.3787
+1278,86800,8.5559
+1284,87200,14.7973
+1290,87600,12.839
+1295,88000,14.3447
+1299,88400,20.0166
+1305,88800,13.8517
+1310,89200,15.5374
+1315,89600,11.2921
+1322,90000,12.7502
+1331,90400,8.0843
+1335,90800,15.8949
+1342,91200,11.7088
+1347,91600,17.2511
+1351,92000,18.5015
+1357,92400,15.5406
+1363,92800,11.4947
+1367,93200,18.5403
+1375,93600,10.876
+1380,94000,16.1795
+1388,94400,9.5337
+1392,94800,18.751
+1401,95200,9.2096
+1406,95600,13.6803
+1413,96000,11.6685
+1424,96400,4.7407
+1436,96800,6.0117
+1448,97200,6.2754
+1453,97600,14.8301
+1457,98000,19.4724
+1462,98400,12.7641
+1470,98800,10.515
+1479,99200,7.7847
+1487,99600,8.115
+1495,100000,9.6315
+1503,100400,9.6751
+1508,100800,14.0532
+1514,101200,13.3345
+1520,101600,14.4949
+1528,102000,7.3491
+1533,102400,16.749
+1537,102800,16.6947
+1544,103200,13.2561
+1549,103600,16.2186
+1555,104000,10.76
+1561,104400,13.0782
+1569,104800,9.7765
+1576,105200,9.9622
+1581,105600,16.9109
+1587,106000,12.2365
+1596,106400,8.4596
+1602,106800,13.4264
+1607,107200,13.8518
+1614,107600,11.4566
+1620,108000,14.6299
+1628,108400,9.2927
+1634,108800,12.3812
+1641,109200,9.7538
+1655,109600,4.8454
+1660,110000,14.5315
+1673,110400,3.9869
+1680,110800,8.9558
+1691,111200,5.7395
+1695,111600,19.8293
+1701,112000,13.327
+1706,112400,13.7603
+1711,112800,14.9665
+1719,113200,7.5555
+1726,113600,10.7098
+1731,114000,14.8993
+1736,114400,14.7975
+1743,114800,7.8648
+1751,115200,7.9691
+1760,115600,9.1848
+1767,116000,8.9283
+1774,116400,11.6902
+1778,116800,18.8354
+1785,117200,12.2704
+1789,117600,15.4445
+1794,118000,16.3556
+1806,118400,5.2073
+1814,118800,9.6861
+1818,119200,15.142
+1825,119600,13.3942
+1830,120000,14.1063
+1836,120400,12.9716
+1844,120800,8.7663
+1853,121200,7.4106
+1857,121600,20.3496
+1863,122000,11.6921
+1868,122400,14.598
+1873,122800,16.1342
+1878,123200,17.5161
+1886,123600,8.8583
+1894,124000,7.4233
+1903,124400,9.8375
+1908,124800,12.7669
+1912,125200,18.0466
+1919,125600,12.422
+1924,126000,13.4632
+1930,126400,14.4262
+1935,126800,16.3081
+1940,127200,17.1442
+1945,127600,13.8432
+1950,128000,17.8724
+1954,128400,15.4276
+1961,128800,11.0026
+1967,129200,13.7937
+1972,129600,13.3407
+1980,130000,11.0252
+1985,130400,13.7636
+1989,130800,20.3436
+1996,131200,9.9471
+2001,131600,14.7954
+2005,132000,19.5569
+2010,132400,15.0541
+2015,132800,16.94
+2020,133200,16.584
+2026,133600,12.1681
+2030,134000,16.941
+2035,134400,18.5808
+2040,134800,16.5108
+2048,135200,9.7247
+2054,135600,13.6772
+2060,136000,12.384
+2066,136400,14.664
+2071,136800,16.1609
+2077,137200,11.9693
+2085,137600,12.2337
+2092,138000,8.7871
+2096,138400,20.9815
+2105,138800,9.5597
+2111,139200,12.4106
+2123,139600,5.885
+2129,140000,12.46
+2137,140400,8.6104
+2141,140800,19.3967
+2146,141200,18.9291
+2153,141600,11.1995
+2157,142000,21.0472
+2166,142400,9.0968
+2173,142800,11.4345
+2177,143200,17.4054
+2182,143600,16.7436
+2187,144000,15.979
+2191,144400,20.6625
+2196,144800,17.967
+2201,145200,14.4191
+2206,145600,17.7912
+2212,146000,12.9955
+2216,146400,20.7862
+2220,146800,18.8328
+2225,147200,17.0398
+2233,147600,9.1588
+2238,148000,17.0543
+2245,148400,11.4016
+2250,148800,13.5677
+2259,149200,9.3432
+2264,149600,16.0615
+2272,150000,7.0293
+2278,150400,15.0398
+2284,150800,13.835
+2288,151200,15.2233
+2294,151600,13.696
+2302,152000,9.5545
+2307,152400,15.6187
+2313,152800,10.671
+2319,153200,13.9607
+2325,153600,10.3803
+2332,154000,11.999
+2339,154400,9.1082
+2347,154800,9.2813
+2355,155200,8.6008
+2359,155600,19.8647
+2369,156000,6.1099
+2376,156400,13.2466
+2381,156800,13.0148
+2390,157200,7.6919
+2395,157600,14.2158
+2400,158000,17.9611
+2408,158400,8.9897
+2415,158800,10.4802
+2420,159200,13.5744
+2425,159600,16.1444
+2430,160000,13.9878
+2437,160400,12.0302
+2445,160800,10.5921
+2449,161200,15.9317
+2453,161600,19.0136
+2462,162000,9.6758
+2467,162400,15.8064
+2474,162800,9.9397
+2478,163200,20.8661
+2488,163600,6.9612
+2496,164000,9.1259
+2502,164400,12.425
+2512,164800,5.2967
+2521,165200,9.5709
+2528,165600,8.9144
+2537,166000,7.9899
+2549,166400,4.6902
+2558,166800,8.4622
+2572,167200,4.6769
+2579,167600,10.6164
+2589,168000,5.4096
+2597,168400,7.1444
+2605,168800,11.1901
+2610,169200,13.3372
+2619,169600,8.1873
+2626,170000,9.9578
+2634,170400,8.5779
+2639,170800,16.6372
+2644,171200,17.5453
+2649,171600,14.0101
+2656,172000,11.4441
+2661,172400,12.8794
+2668,172800,11.5128
+2674,173200,12.0959
+2683,173600,5.4352
+2687,174000,18.7393
+2694,174400,11.3662
+2700,174800,12.8577
+2704,175200,16.8251
+2708,175600,21.1851
+2712,176000,19.6086
+2717,176400,18.0934
+2722,176800,15.998
+2726,177200,20.8385
+2731,177600,13.3021
+2737,178000,15.9943
+2743,178400,11.5759
+2749,178800,13.4046
+2756,179200,9.6437
+2761,179600,16.2845
+2766,180000,15.4722
+2770,180400,16.5676
+2777,180800,10.0904
+2783,181200,13.448
+2788,181600,14.414
+2792,182000,20.08
+2797,182400,16.0141
+2801,182800,20.2584
+2807,183200,14.7895
+2812,183600,14.5707
+2818,184000,13.1659
+2825,184400,11.4905
+2830,184800,15.3281
+2835,185200,15.8941
+2841,185600,11.6085
+2846,186000,12.8389
+2853,186400,12.1964
+2858,186800,15.4969
+2863,187200,16.2343
+2868,187600,13.6736
+2874,188000,13.7629
+2879,188400,18.2208
+2884,188800,14.7769
+2888,189200,20.1099
+2892,189600,20.665
+2897,190000,14.9009
+2902,190400,15.9363
+2908,190800,13.7088
+2914,191200,12.0374
+2919,191600,11.6974
+2923,192000,19.4542
+2929,192400,12.6135
+2935,192800,13.6599
+2941,193200,14.4815
+2945,193600,18.2334
+2950,194000,13.9492
+2956,194400,12.6527
+2960,194800,19.9709
+2964,195200,20.1638
+2970,195600,12.4923
+2976,196000,13.7684
+2983,196400,11.1614
+2988,196800,12.4877
+2996,197200,6.8995
+3005,197600,7.2801
+3013,198000,10.8011
+3018,198400,13.5368
+3023,198800,15.2503
+3032,199200,8.255
+3037,199600,15.5807
+3044,200000,9.8756

code/Lake application/logs/results_1/PDPPO_frozen_lake_log_3.csv

@@ -0,0 +1,501 @@
+episode,timestep,reward
+24,400,1.1789
+42,800,2.1233
+61,1200,1.5503
+75,1600,2.1531
+94,2000,1.8369
+103,2400,3.5188
+113,2800,3.0467
+126,3200,2.5075
+137,3600,2.784
+142,4000,5.3221
+149,4400,4.356
+158,4800,3.5811
+166,5200,4.0608
+174,5600,3.8915
+178,6000,7.7872
+184,6400,5.5597
+189,6800,6.3654
+193,7200,6.812
+198,7600,6.1092
+202,8000,7.5749
+207,8400,6.2928
+213,8800,5.3826
+217,9200,6.452
+221,9600,7.7105
+225,10000,7.7963
+231,10400,5.4617
+235,10800,7.5387
+239,11200,7.7388
+243,11600,7.2572
+247,12000,7.8198
+251,12400,7.7371
+255,12800,7.8481
+260,13200,7.7476
+264,13600,7.8618
+268,14000,6.7995
+272,14400,7.8333
+277,14800,7.2725
+282,15200,5.7045
+286,15600,7.8592
+290,16000,7.9573
+294,16400,7.996
+298,16800,8.0243
+302,17200,7.1519
+307,17600,7.6693
+312,18000,6.182
+317,18400,6.3662
+321,18800,6.6984
+326,19200,7.1547
+332,19600,5.4313
+337,20000,6.58
+341,20400,6.4143
+348,20800,4.6998
+354,21200,6.0579
+359,21600,6.3869
+363,22000,8.4537
+367,22400,7.6145
+371,22800,8.7217
+375,23200,8.0302
+379,23600,7.855
+383,24000,7.9192
+388,24400,7.1253
+392,24800,7.2569
+396,25200,8.0458
+400,25600,8.0878
+404,26000,7.6454
+411,26400,5.4292
+417,26800,6.2442
+421,27200,7.1395
+425,27600,8.138
+429,28000,7.4518
+433,28400,8.2299
+437,28800,8.0705
+443,29200,6.4598
+448,29600,6.5685
+453,30000,7.0016
+457,30400,8.3606
+461,30800,8.0059
+466,31200,6.8728
+470,31600,8.0419
+475,32000,6.5841
+479,32400,8.576
+484,32800,7.5318
+489,33200,7.0436
+493,33600,8.4315
+500,34000,5.9633
+506,34400,6.3842
+511,34800,8.348
+515,35200,11.5177
+520,35600,8.375
+525,36000,8.8358
+530,36400,8.3674
+536,36800,8.5203
+541,37200,8.0434
+548,37600,5.7622
+552,38000,12.1796
+556,38400,11.9279
+562,38800,9.0523
+566,39200,10.6577
+571,39600,9.9867
+575,40000,12.9051
+580,40400,9.6013
+584,40800,9.0039
+588,41200,11.6103
+593,41600,11.3596
+599,42000,8.0795
+604,42400,9.4068
+608,42800,10.1782
+613,43200,10.2511
+618,43600,8.7243
+627,44000,6.9675
+633,44400,6.1848
+638,44800,12.508
+643,45200,9.435
+649,45600,10.0267
+654,46000,12.8219
+658,46400,14.4656
+663,46800,12.4416
+672,47200,7.6444
+679,47600,6.7948
+686,48000,10.5707
+692,48400,11.3461
+696,48800,15.5451
+703,49200,7.6138
+709,49600,14.2849
+713,50000,17.6588
+717,50400,16.8401
+724,50800,9.6536
+732,51200,7.0814
+737,51600,15.1825
+745,52000,9.6236
+749,52400,16.5895
+757,52800,8.9825
+763,53200,13.4571
+769,53600,12.3041
+774,54000,13.3599
+779,54400,16.2315
+783,54800,18.7432
+788,55200,14.2708
+792,55600,19.0263
+796,56000,19.6853
+804,56400,8.6649
+810,56800,13.5299
+815,57200,15.4356
+822,57600,8.4714
+827,58000,14.8482
+832,58400,13.1313
+836,58800,17.7723
+841,59200,15.1604
+846,59600,16.5815
+850,60000,18.4684
+856,60400,11.723
+860,60800,15.37
+866,61200,14.931
+871,61600,11.3223
+876,62000,11.9169
+883,62400,12.8174
+888,62800,13.5601
+893,63200,15.0672
+898,63600,11.3614
+903,64000,15.3905
+908,64400,13.6269
+912,64800,17.5681
+917,65200,13.3706
+922,65600,13.8827
+927,66000,13.7525
+932,66400,15.7249
+937,66800,12.4379
+943,67200,12.0458
+947,67600,18.2474
+953,68000,11.2208
+961,68400,5.6857
+966,68800,11.2325
+977,69200,6.065
+981,69600,12.7678
+986,70000,15.0279
+993,70400,11.0013
+1001,70800,6.6314
+1008,71200,8.9485
+1015,71600,11.0937
+1020,72000,13.6421
+1027,72400,9.3296
+1033,72800,12.498
+1037,73200,13.5609
+1042,73600,15.2572
+1049,74000,9.7781
+1059,74400,6.6146
+1067,74800,8.4938
+1078,75200,5.6932
+1085,75600,9.7875
+1089,76000,14.85
+1095,76400,11.959
+1099,76800,18.4955
+1106,77200,11.1234
+1110,77600,16.6448
+1116,78000,14.7683
+1121,78400,15.4418
+1126,78800,15.6231
+1133,79200,8.6739
+1137,79600,19.0853
+1141,80000,19.3869
+1146,80400,15.0416
+1151,80800,16.4105
+1155,81200,18.5403
+1162,81600,10.4387
+1168,82000,12.8525
+1173,82400,15.0793
+1179,82800,11.1054
+1184,83200,15.8964
+1189,83600,15.9822
+1193,84000,20.0662
+1197,84400,19.8951
+1203,84800,12.0668
+1207,85200,19.6185
+1216,85600,9.3892
+1221,86000,13.0625
+1228,86400,10.6035
+1236,86800,8.5601
+1243,87200,11.3572
+1248,87600,12.3989
+1255,88000,10.0261
+1261,88400,12.0088
+1265,88800,18.8742
+1272,89200,13.0981
+1281,89600,7.8559
+1288,90000,11.3191
+1293,90400,16.281
+1299,90800,12.2251
+1306,91200,10.0396
+1318,91600,7.6743
+1322,92000,17.0425
+1328,92400,13.7841
+1335,92800,11.843
+1339,93200,18.7906
+1348,93600,7.6509
+1355,94000,11.5199
+1360,94400,15.8789
+1365,94800,16.2332
+1372,95200,8.0954
+1378,95600,12.8351
+1386,96000,10.1123
+1391,96400,13.7096
+1398,96800,9.9747
+1405,97200,9.7309
+1413,97600,10.6216
+1417,98000,19.8107
+1425,98400,8.7529
+1430,98800,12.5949
+1437,99200,10.2796
+1442,99600,14.1966
+1448,100000,12.1246
+1453,100400,14.1483
+1459,100800,14.6867
+1466,101200,11.5653
+1470,101600,19.7992
+1475,102000,16.2707
+1480,102400,12.707
+1490,102800,7.9692
+1496,103200,14.6578
+1504,103600,8.9977
+1512,104000,8.5685
+1516,104400,18.8262
+1525,104800,8.1862
+1532,105200,11.2229
+1539,105600,10.9358
+1543,106000,15.9632
+1551,106400,11.0493
+1555,106800,15.4412
+1561,107200,12.9739
+1568,107600,11.9314
+1573,108000,15.9377
+1580,108400,12.2091
+1586,108800,13.3599
+1592,109200,10.2974
+1598,109600,12.8394
+1605,110000,11.8069
+1611,110400,15.0917
+1619,110800,8.6425
+1626,111200,11.7447
+1632,111600,11.9684
+1637,112000,17.0765
+1643,112400,10.8535
+1647,112800,20.3134
+1654,113200,14.2449
+1658,113600,15.6927
+1663,114000,16.4525
+1667,114400,20.7007
+1675,114800,10.9193
+1681,115200,13.8018
+1686,115600,12.168
+1691,116000,16.1065
+1696,116400,16.9738
+1703,116800,12.1336
+1707,117200,17.4688
+1711,117600,20.7223
+1717,118000,12.8775
+1724,118400,10.7785
+1729,118800,16.0037
+1737,119200,9.3322
+1741,119600,15.5575
+1746,120000,16.7479
+1759,120400,4.1965
+1774,120800,5.0799
+1784,121200,7.0285
+1797,121600,6.1574
+1813,122000,4.1487
+1824,122400,6.7635
+1832,122800,10.5592
+1838,123200,11.4983
+1850,123600,6.0096
+1857,124000,13.2751
+1865,124400,10.6374
+1872,124800,10.4461
+1879,125200,10.8001
+1887,125600,10.5679
+1893,126000,13.6904
+1899,126400,13.3654
+1906,126800,11.1289
+1911,127200,14.4808
+1917,127600,14.7797
+1922,128000,16.3632
+1927,128400,14.8871
+1931,128800,18.4437
+1936,129200,13.8601
+1941,129600,19.323
+1948,130000,11.3219
+1957,130400,6.0778
+1963,130800,13.8097
+1968,131200,17.4356
+1972,131600,15.6235
+1978,132000,13.9724
+1983,132400,16.7818
+1990,132800,11.7503
+1995,133200,16.7384
+2000,133600,11.0031
+2007,134000,13.208
+2013,134400,12.1374
+2018,134800,10.6665
+2022,135200,20.4056
+2027,135600,13.8923
+2033,136000,14.9732
+2039,136400,11.7272
+2049,136800,5.9728
+2055,137200,13.4777
+2061,137600,12.7418
+2070,138000,7.9501
+2076,138400,9.8375
+2085,138800,8.8678
+2096,139200,6.0538
+2104,139600,7.5676
+2112,140000,10.4544
+2126,140400,3.3254
+2135,140800,8.5089
+2147,141200,5.0491
+2155,141600,8.8535
+2165,142000,4.7412
+2174,142400,6.3371
+2180,142800,13.6134
+2189,143200,7.7075
+2202,143600,4.1581
+2206,144000,15.2914
+2217,144400,7.21
+2222,144800,14.1197
+2229,145200,12.3017
+2238,145600,7.5941
+2245,146000,8.9586
+2252,146400,11.0808
+2257,146800,14.2912
+2263,147200,13.1251
+2269,147600,12.861
+2274,148000,12.1163
+2279,148400,16.2446
+2284,148800,15.9297
+2288,149200,14.7428
+2295,149600,10.4804
+2302,150000,12.5935
+2308,150400,10.2608
+2315,150800,11.9565
+2321,151200,10.1254
+2325,151600,17.2792
+2332,152000,10.5685
+2343,152400,7.1165
+2355,152800,5.5324
+2362,153200,10.2419
+2367,153600,17.1911
+2372,154000,16.5306
+2377,154400,13.0943
+2383,154800,17.1094
+2390,155200,11.8845
+2394,155600,21.3396
+2400,156000,12.3499
+2409,156400,7.6367
+2414,156800,19.1307
+2418,157200,21.5015
+2424,157600,14.4416
+2431,158000,11.1522
+2438,158400,11.1949
+2450,158800,7.2224
+2457,159200,11.6111
+2468,159600,5.0266
+2476,160000,10.68
+2487,160400,5.6073
+2493,160800,13.4424
+2503,161200,6.8739
+2512,161600,7.4788
+2520,162000,10.2704
+2531,162400,6.3896
+2537,162800,10.6474
+2542,163200,14.1442
+2555,163600,5.9226
+2566,164000,6.0377
+2575,164400,7.4593
+2588,164800,4.9726
+2594,165200,10.6298
+2599,165600,12.0844
+2616,166000,3.8811
+2626,166400,4.2739
+2634,166800,9.1184
+2641,167200,8.623
+2647,167600,12.8333
+2657,168000,5.8736
+2663,168400,7.3345
+2671,168800,6.602
+2680,169200,7.7097
+2692,169600,3.3408
+2698,170000,12.0658
+2710,170400,4.496
+2719,170800,7.4669
+2729,171200,6.8099
+2738,171600,7.104
+2744,172000,11.9163
+2751,172400,11.3711
+2757,172800,10.847
+2765,173200,7.2486
+2774,173600,6.5974
+2783,174000,9.2593
+2790,174400,10.1808
+2797,174800,7.8738
+2805,175200,8.1315
+2811,175600,12.4611
+2817,176000,11.2274
+2824,176400,9.6997
+2829,176800,16.383
+2835,177200,9.7906
+2841,177600,9.8226
+2850,178000,8.9189
+2856,178400,8.719
+2862,178800,14.3174
+2868,179200,9.976
+2873,179600,16.595
+2879,180000,11.7942
+2886,180400,8.4306
+2890,180800,19.6647
+2895,181200,16.7307
+2899,181600,16.9061
+2907,182000,8.0865
+2912,182400,16.0249
+2917,182800,12.6253
+2924,183200,10.9561
+2930,183600,9.5365
+2935,184000,11.5344
+2943,184400,7.9225
+2949,184800,11.1286
+2959,185200,7.7913
+2964,185600,14.7933
+2972,186000,7.3943
+2979,186400,10.7152
+2986,186800,8.1646
+2992,187200,11.7603
+2997,187600,13.334
+3003,188000,12.5786
+3009,188400,12.2962
+3015,188800,14.3177
+3019,189200,15.0194
+3024,189600,19.2136
+3029,190000,15.3857
+3035,190400,9.9101
+3043,190800,10.0892
+3047,191200,17.6767
+3051,191600,19.3897
+3056,192000,15.6982
+3060,192400,19.0069
+3065,192800,17.3691
+3070,193200,13.5637
+3075,193600,14.7953
+3080,194000,16.2313
+3086,194400,13.6929
+3091,194800,17.0558
+3095,195200,17.121
+3100,195600,14.9038
+3108,196000,9.5739
+3112,196400,16.1362
+3118,196800,9.8704
+3127,197200,8.0936
+3132,197600,17.4588
+3137,198000,12.1721
+3143,198400,9.6148
+3147,198800,18.6538
+3151,199200,19.5497
+3156,199600,15.3876
+3163,200000,10.7067

code/Lake application/logs/results_1/PDPPO_frozen_lake_log_4.csv

@@ -0,0 +1,501 @@
+episode,timestep,reward
+20,400,1.5088
+42,800,1.3079
+65,1200,1.3319
+88,1600,1.3245
+101,2000,2.2609
+116,2400,2.0004
+122,2800,4.3646
+130,3200,4.0763
+138,3600,3.5934
+147,4000,3.15
+158,4400,3.0842
+164,4800,4.0592
+171,5200,4.5173
+177,5600,4.8523
+183,6000,4.215
+190,6400,5.0604
+197,6800,4.5393
+201,7200,7.2405
+207,7600,4.3459
+213,8000,4.605
+220,8400,3.9493
+227,8800,4.0154
+233,9200,5.1817
+237,9600,7.2706
+241,10000,6.1275
+246,10400,6.517
+251,10800,6.4718
+256,11200,5.3332
+263,11600,4.8526
+267,12000,7.5372
+273,12400,4.4563
+278,12800,6.2474
+283,13200,5.1259
+288,13600,6.6405
+293,14000,5.4922
+297,14400,7.3697
+301,14800,7.46
+306,15200,6.8119
+310,15600,7.5148
+314,16000,7.566
+318,16400,5.6766
+324,16800,5.7109
+328,17200,6.9979
+333,17600,6.6446
+337,18000,7.1488
+342,18400,6.612
+348,18800,5.6986
+352,19200,7.4129
+357,19600,6.3231
+364,20000,4.8872
+369,20400,6.2171
+373,20800,7.6549
+379,21200,5.0764
+386,21600,3.8937
+392,22000,4.8971
+396,22400,7.7155
+401,22800,6.522
+405,23200,6.6372
+410,23600,7.358
+417,24000,4.7262
+423,24400,4.3083
+428,24800,6.6989
+432,25200,7.4748
+437,25600,6.8524
+441,26000,7.4783
+447,26400,5.58
+451,26800,7.5209
+457,27200,5.4526
+461,27600,7.2869
+466,28000,6.9935
+470,28400,8.5876
+475,28800,6.8761
+480,29200,5.9352
+486,29600,6.2676
+490,30000,7.6652
+494,30400,8.3381
+499,30800,9.2616
+504,31200,6.5226
+509,31600,6.8202
+513,32000,8.7099
+519,32400,7.2073
+523,32800,9.4631
+530,33200,5.9642
+534,33600,9.2221
+539,34000,7.8802
+545,34400,7.4494
+549,34800,10.2656
+554,35200,8.5635
+559,35600,8.5277
+564,36000,9.2301
+570,36400,7.3293
+577,36800,5.9939
+583,37200,8.0029
+590,37600,6.5089
+595,38000,9.9265
+600,38400,7.7368
+605,38800,10.0703
+610,39200,9.405
+615,39600,9.2747
+620,40000,10.6746
+625,40400,10.8504
+629,40800,12.3268
+634,41200,7.9002
+638,41600,12.2213
+643,42000,10.5923
+649,42400,8.3824
+653,42800,12.3659
+659,43200,7.8569
+664,43600,7.7947
+668,44000,11.9043
+673,44400,11.3176
+677,44800,9.5839
+683,45200,8.7337
+687,45600,11.8954
+691,46000,10.9464
+697,46400,9.0987
+701,46800,11.5694
+706,47200,10.0594
+711,47600,8.6043
+715,48000,11.3733
+721,48400,7.4597
+727,48800,8.6735
+733,49200,9.2683
+737,49600,12.2719
+743,50000,8.5308
+748,50400,7.8866
+754,50800,9.0977
+758,51200,10.5799
+763,51600,10.2186
+769,52000,9.4147
+774,52400,9.8676
+779,52800,9.8646
+783,53200,11.9396
+788,53600,8.3464
+792,54000,10.8994
+800,54400,7.1043
+804,54800,11.0494
+810,55200,7.2492
+816,55600,9.008
+820,56000,11.969
+824,56400,12.3475
+830,56800,8.1093
+836,57200,8.6649
+841,57600,10.3044
+848,58000,6.3358
+853,58400,8.6059
+858,58800,10.5647
+862,59200,11.937
+867,59600,10.6032
+871,60000,12.1926
+879,60400,6.4127
+887,60800,6.1397
+892,61200,9.5413
+897,61600,9.9852
+905,62000,6.0649
+909,62400,12.114
+913,62800,10.7127
+919,63200,7.5118
+928,63600,5.1292
+933,64000,9.7503
+938,64400,11.1308
+943,64800,9.8459
+949,65200,9.6871
+955,65600,9.5914
+962,66000,6.487
+970,66400,5.1927
+980,66800,5.9668
+984,67200,13.1207
+988,67600,13.5101
+995,68000,7.9993
+1001,68400,7.3224
+1006,68800,11.6941
+1011,69200,8.4031
+1015,69600,14.3545
+1024,70000,6.3113
+1029,70400,12.9317
+1034,70800,12.4254
+1040,71200,8.7256
+1045,71600,12.1083
+1049,72000,14.6145
+1056,72400,8.6096
+1062,72800,10.0664
+1066,73200,14.5826
+1071,73600,12.4316
+1076,74000,11.2348
+1081,74400,11.6494
+1086,74800,8.9371
+1092,75200,11.5723
+1096,75600,14.9071
+1100,76000,13.8022
+1104,76400,12.762
+1109,76800,12.5704
+1114,77200,13.2544
+1121,77600,7.8204
+1127,78000,11.0577
+1132,78400,10.2288
+1137,78800,12.0947
+1143,79200,10.4587
+1150,79600,8.3463
+1154,80000,15.744
+1159,80400,13.1713
+1166,80800,7.4983
+1173,81200,9.3079
+1177,81600,14.3302
+1183,82000,10.1948
+1187,82400,15.5844
+1193,82800,10.8247
+1198,83200,9.861
+1203,83600,13.0967
+1208,84000,14.2775
+1212,84400,15.9856
+1220,84800,7.0939
+1224,85200,13.172
+1229,85600,14.9623
+1234,86000,10.7564
+1240,86400,12.0071
+1245,86800,12.1013
+1250,87200,11.8996
+1254,87600,14.3552
+1258,88000,15.6836
+1263,88400,14.0425
+1267,88800,16.2951
+1274,89200,8.0649
+1279,89600,12.0771
+1287,90000,8.9504
+1293,90400,9.2019
+1298,90800,13.5403
+1302,91200,16.2009
+1306,91600,15.8977
+1311,92000,12.6036
+1318,92400,9.4293
+1324,92800,11.8471
+1329,93200,9.6477
+1334,93600,13.0578
+1339,94000,12.0729
+1346,94400,9.9596
+1350,94800,16.223
+1355,95200,13.1369
+1364,95600,7.2675
+1369,96000,11.9901
+1377,96400,8.69
+1382,96800,12.3589
+1389,97200,8.9372
+1400,97600,5.9625
+1405,98000,9.6908
+1412,98400,8.4351
+1418,98800,12.2907
+1426,99200,7.9709
+1432,99600,8.709
+1437,100000,12.9995
+1442,100400,12.5061
+1448,100800,11.813
+1453,101200,10.3663
+1459,101600,11.3195
+1471,102000,4.8996
+1482,102400,5.2443
+1492,102800,5.7095
+1498,103200,10.861
+1505,103600,8.0669
+1511,104000,11.5714
+1516,104400,11.8947
+1522,104800,13.1965
+1527,105200,11.7688
+1532,105600,14.6079
+1537,106000,12.5179
+1545,106400,6.0034
+1550,106800,13.8795
+1555,107200,12.5438
+1560,107600,12.3526
+1565,108000,14.0173
+1572,108400,6.5151
+1578,108800,11.821
+1584,109200,10.7337
+1589,109600,12.987
+1593,110000,15.0823
+1598,110400,13.6488
+1603,110800,15.9419
+1607,111200,17.6469
+1613,111600,11.8659
+1620,112000,8.6666
+1626,112400,8.0076
+1634,112800,8.2332
+1639,113200,11.4569
+1645,113600,7.8428
+1651,114000,9.6165
+1659,114400,7.2423
+1665,114800,10.0212
+1670,115200,15.7796
+1675,115600,13.3568
+1681,116000,11.9107
+1686,116400,16.4987
+1690,116800,17.6908
+1695,117200,13.7199
+1700,117600,13.2496
+1704,118000,18.4471
+1709,118400,12.1934
+1714,118800,16.5036
+1720,119200,13.0416
+1726,119600,9.5647
+1734,120000,9.7985
+1742,120400,7.6858
+1747,120800,11.444
+1751,121200,17.1469
+1758,121600,11.2027
+1764,122000,11.57
+1771,122400,11.0437
+1776,122800,12.1007
+1788,123200,7.5098
+1793,123600,15.3339
+1798,124000,11.8126
+1805,124400,11.7518
+1810,124800,15.431
+1817,125200,9.2215
+1822,125600,16.9219
+1827,126000,13.1771
+1832,126400,19.0296
+1838,126800,12.1479
+1842,127200,20.0777
+1847,127600,12.5869
+1857,128000,7.5145
+1862,128400,17.6462
+1871,128800,5.5968
+1880,129200,7.7816
+1890,129600,7.5515
+1897,130000,9.2213
+1904,130400,12.0633
+1913,130800,9.4187
+1923,131200,6.6914
+1933,131600,6.9577
+1938,132000,15.6559
+1943,132400,15.711
+1948,132800,15.8596
+1955,133200,10.39
+1961,133600,13.9018
+1968,134000,10.9876
+1975,134400,8.5861
+1983,134800,9.3426
+1988,135200,16.0291
+1996,135600,10.6449
+2000,136000,15.9081
+2008,136400,9.7453
+2014,136800,14.5408
+2019,137200,13.6499
+2023,137600,17.35
+2030,138000,11.9688
+2038,138400,9.7757
+2043,138800,15.0422
+2049,139200,14.1278
+2054,139600,11.3104
+2060,140000,10.5947
+2065,140400,14.9903
+2071,140800,12.8763
+2076,141200,12.2992
+2085,141600,8.4617
+2091,142000,12.2624
+2099,142400,10.5275
+2114,142800,3.2717
+2125,143200,4.7307
+2136,143600,8.0355
+2147,144000,6.7636
+2154,144400,8.4627
+2164,144800,8.6816
+2173,145200,7.4853
+2179,145600,9.5195
+2186,146000,10.8791
+2193,146400,12.8798
+2198,146800,14.4826
+2204,147200,13.2357
+2210,147600,14.1333
+2216,148000,12.9477
+2221,148400,14.1298
+2229,148800,10.4178
+2234,149200,15.747
+2239,149600,13.7127
+2245,150000,13.5562
+2252,150400,10.7229
+2258,150800,13.8351
+2263,151200,16.2056
+2268,151600,15.9527
+2275,152000,9.0862
+2279,152400,19.7776
+2285,152800,14.6447
+2293,153200,10.0589
+2298,153600,15.9818
+2308,154000,7.2806
+2315,154400,11.1871
+2320,154800,16.5332
+2326,155200,11.4502
+2332,155600,11.2749
+2337,156000,18.7918
+2345,156400,9.0297
+2351,156800,13.5346
+2355,157200,20.3685
+2359,157600,17.5759
+2363,158000,20.9134
+2372,158400,8.7563
+2377,158800,16.3223
+2383,159200,12.3571
+2388,159600,14.8996
+2393,160000,16.1612
+2400,160400,10.695
+2404,160800,15.8143
+2409,161200,14.1756
+2417,161600,10.3059
+2421,162000,15.7669
+2426,162400,16.7237
+2432,162800,13.186
+2436,163200,15.9777
+2442,163600,15.1247
+2451,164000,6.9288
+2455,164400,19.8019
+2459,164800,16.5844
+2463,165200,19.4641
+2469,165600,12.5474
+2475,166000,12.4718
+2482,166400,9.2596
+2492,166800,7.4804
+2499,167200,9.2925
+2506,167600,9.6121
+2512,168000,11.8263
+2519,168400,11.1036
+2524,168800,11.8451
+2530,169200,12.3773
+2535,169600,16.2471
+2541,170000,11.3649
+2545,170400,15.6326
+2550,170800,16.8937
+2554,171200,19.5632
+2558,171600,17.4009
+2564,172000,15.0096
+2569,172400,12.1716
+2574,172800,16.2656
+2579,173200,12.1121
+2584,173600,15.5555
+2590,174000,14.2854
+2597,174400,11.301
+2601,174800,17.2093
+2606,175200,15.6553
+2611,175600,13.2231
+2618,176000,12.4886
+2623,176400,11.6754
+2627,176800,19.8814
+2633,177200,15.667
+2637,177600,17.0859
+2642,178000,14.0034
+2650,178400,9.8961
+2654,178800,19.2615
+2663,179200,8.5405
+2670,179600,9.0917
+2684,180000,3.9164
+2690,180400,12.0612
+2697,180800,10.1366
+2707,181200,7.161
+2713,181600,10.5804
+2719,182000,11.7124
+2725,182400,12.7422
+2735,182800,5.839
+2743,183200,9.3809
+2748,183600,14.0109
+2754,184000,12.5838
+2772,184400,2.7956
+2782,184800,5.9967
+2790,185200,6.7712
+2799,185600,8.4177
+2805,186000,11.2619
+2818,186400,4.5222
+2827,186800,7.9052
+2834,187200,8.1854
+2845,187600,5.2657
+2852,188000,9.6327
+2862,188400,5.4335
+2870,188800,9.4833
+2876,189200,10.3303
+2881,189600,14.843
+2891,190000,6.7218
+2896,190400,13.4077
+2902,190800,12.8647
+2908,191200,9.9087
+2913,191600,16.5216
+2920,192000,9.6137
+2924,192400,15.5697
+2932,192800,8.3219
+2942,193200,4.7098
+2949,193600,11.8933
+2954,194000,14.2076
+2961,194400,10.3591
+2966,194800,12.7551
+2972,195200,12.0142
+2978,195600,11.9555
+2982,196000,17.2151
+2988,196400,11.6288
+2994,196800,11.5739
+2998,197200,15.9745
+3005,197600,10.5003
+3010,198000,12.1237
+3017,198400,11.9958
+3022,198800,13.6748
+3031,199200,8.644
+3036,199600,15.9198
+3043,200000,11.7405

code/Lake application/logs/results_1/PDPPO_frozen_lake_log_5.csv

@@ -0,0 +1,501 @@
+episode,timestep,reward
+30,400,1.0363
+55,800,1.1656
+74,1200,1.2812
+87,1600,2.8849
+101,2000,2.0512
+109,2400,4.4421
+120,2800,2.7307
+130,3200,2.9113
+136,3600,4.8513
+143,4000,4.6759
+147,4400,6.8206
+156,4800,3.5122
+160,5200,7.4682
+167,5600,4.2985
+173,6000,4.6158
+178,6400,6.3235
+182,6800,5.9121
+187,7200,6.0524
+192,7600,6.2336
+196,8000,6.7958
+201,8400,5.8009
+205,8800,7.3525
+209,9200,7.4518
+214,9600,6.8613
+220,10000,4.8876
+224,10400,7.4277
+228,10800,6.8252
+232,11200,7.0368
+236,11600,7.4591
+241,12000,6.5145
+245,12400,7.3888
+250,12800,5.1931
+255,13200,6.1031
+259,13600,7.5133
+264,14000,6.6363
+268,14400,7.6229
+273,14800,6.0136
+277,15200,8.2029
+282,15600,6.2847
+287,16000,6.6472
+291,16400,7.3617
+295,16800,8.1746
+299,17200,6.8719
+304,17600,6.1261
+309,18000,7.0801
+313,18400,7.6909
+317,18800,7.4924
+321,19200,7.4629
+327,19600,5.5724
+333,20000,4.8738
+337,20400,7.1132
+342,20800,6.5451
+347,21200,5.6767
+351,21600,7.6065
+355,22000,7.5147
+360,22400,7.0767
+364,22800,7.5414
+368,23200,7.7461
+373,23600,5.5045
+377,24000,6.8692
+382,24400,6.7735
+386,24800,7.1926
+391,25200,6.3948
+395,25600,6.506
+400,26000,6.8513
+404,26400,6.7321
+410,26800,5.3182
+415,27200,6.3436
+424,27600,3.9625
+433,28000,4.0086
+438,28400,7.455
+448,28800,4.1049
+455,29200,4.8118
+460,29600,8.8025
+468,30000,5.2449
+474,30400,6.9961
+479,30800,9.5919
+486,31200,4.9522
+494,31600,6.2399
+501,32000,6.2058
+507,32400,7.817
+512,32800,10.4552
+519,33200,6.8586
+528,33600,5.2227
+532,34000,11.0335
+537,34400,9.8253
+548,34800,4.943
+552,35200,12.3006
+557,35600,8.6328
+565,36000,6.8881
+573,36400,6.9154
+578,36800,9.7087
+583,37200,11.4344
+588,37600,8.4428
+594,38000,9.4412
+598,38400,13.0931
+602,38800,13.7591
+608,39200,9.2528
+616,39600,5.39
+622,40000,11.192
+628,40400,9.3021
+635,40800,6.3386
+640,41200,13.3335
+645,41600,10.7252
+650,42000,9.8878
+657,42400,7.6309
+662,42800,13.1219
+667,43200,11.3305
+674,43600,8.0326
+682,44000,7.1185
+686,44400,11.0479
+691,44800,12.8963
+696,45200,12.0313
+700,45600,11.8897
+709,46000,7.0573
+714,46400,10.6651
+720,46800,10.3476
+727,47200,10.3755
+731,47600,13.4045
+738,48000,9.1209
+742,48400,13.8504
+749,48800,9.6356
+755,49200,9.325
+761,49600,10.3474
+765,50000,14.3104
+770,50400,12.0887
+774,50800,15.2546
+779,51200,12.14
+784,51600,12.2329
+788,52000,15.0146
+793,52400,10.5181
+799,52800,10.1783
+805,53200,8.4057
+812,53600,8.7315
+818,54000,11.7951
+822,54400,15.2027
+827,54800,11.7442
+831,55200,13.5781
+836,55600,12.2977
+842,56000,10.4294
+847,56400,10.9495
+853,56800,10.9578
+859,57200,11.0167
+865,57600,9.7521
+869,58000,15.5817
+875,58400,9.9763
+880,58800,13.0464
+884,59200,13.8061
+889,59600,12.9356
+896,60000,10.3463
+901,60400,10.6919
+908,60800,8.6734
+914,61200,12.4219
+919,61600,10.0202
+924,62000,15.9255
+929,62400,11.2685
+935,62800,10.8452
+940,63200,11.7721
+946,63600,11.407
+952,64000,12.8059
+958,64400,8.4392
+964,64800,11.9307
+969,65200,15.1005
+974,65600,13.4469
+984,66000,7.6713
+988,66400,14.245
+995,66800,9.1666
+1000,67200,15.6424
+1005,67600,16.3647
+1011,68000,15.1433
+1017,68400,12.4149
+1022,68800,13.4503
+1030,69200,10.6481
+1039,69600,8.0946
+1046,70000,7.7302
+1051,70400,16.2667
+1056,70800,17.1896
+1062,71200,12.1892
+1067,71600,15.6327
+1074,72000,10.8082
+1078,72400,20.0766
+1086,72800,9.649
+1092,73200,11.0206
+1098,73600,11.8751
+1107,74000,8.5445
+1111,74400,16.1052
+1118,74800,11.2793
+1122,75200,17.9738
+1129,75600,11.9216
+1135,76000,15.1108
+1141,76400,10.5198
+1147,76800,16.3762
+1151,77200,18.503
+1160,77600,8.741
+1167,78000,9.7747
+1173,78400,13.8605
+1180,78800,9.6619
+1185,79200,15.3712
+1190,79600,16.7442
+1196,80000,11.6345
+1202,80400,14.0943
+1207,80800,14.2296
+1212,81200,16.3014
+1220,81600,9.8765
+1228,82000,8.7464
+1235,82400,10.1393
+1240,82800,16.1224
+1246,83200,15.2376
+1252,83600,10.7445
+1259,84000,10.0961
+1264,84400,15.7153
+1271,84800,13.0846
+1280,85200,7.0547
+1285,85600,15.492
+1292,86000,9.3012
+1297,86400,12.5845
+1303,86800,12.9457
+1308,87200,15.8783
+1314,87600,12.6196
+1319,88000,13.5926
+1325,88400,9.8846
+1329,88800,19.8844
+1336,89200,11.9627
+1340,89600,19.6276
+1345,90000,11.4204
+1355,90400,9.3768
+1360,90800,13.957
+1365,91200,15.7235
+1372,91600,10.9691
+1376,92000,17.8047
+1380,92400,19.6958
+1385,92800,19.2327
+1391,93200,9.4353
+1400,93600,8.9945
+1407,94000,9.5239
+1416,94400,9.7381
+1422,94800,12.1216
+1429,95200,11.5348
+1437,95600,6.6356
+1447,96000,8.2667
+1456,96400,5.5346
+1463,96800,13.0559
+1471,97200,8.4067
+1483,97600,5.107
+1494,98000,6.8124
+1501,98400,8.9277
+1515,98800,5.11
+1524,99200,8.4644
+1538,99600,4.2993
+1547,100000,6.5226
+1552,100400,16.8617
+1559,100800,10.7992
+1569,101200,7.4671
+1577,101600,8.0795
+1586,102000,8.0661
+1595,102400,7.4557
+1602,102800,9.2675
+1612,103200,7.525
+1621,103600,8.0409
+1628,104000,10.8619
+1638,104400,7.0936
+1648,104800,6.5034
+1656,105200,7.5528
+1666,105600,7.0664
+1674,106000,9.2449
+1684,106400,6.3384
+1695,106800,4.2875
+1701,107200,15.9626
+1710,107600,8.014
+1720,108000,7.4489
+1729,108400,8.3794
+1737,108800,8.7056
+1745,109200,7.9263
+1751,109600,11.078
+1758,110000,12.3928
+1763,110400,12.4602
+1769,110800,16.5946
+1775,111200,12.9342
+1780,111600,15.1111
+1785,112000,13.7242
+1791,112400,14.9992
+1797,112800,11.5528
+1802,113200,16.7522
+1806,113600,14.0396
+1813,114000,12.6857
+1817,114400,16.7741
+1822,114800,12.9523
+1827,115200,16.5158
+1832,115600,17.1408
+1839,116000,9.8454
+1843,116400,17.8182
+1849,116800,11.2671
+1855,117200,14.7182
+1860,117600,14.9587
+1867,118000,8.7827
+1874,118400,11.6531
+1879,118800,12.538
+1887,119200,9.8758
+1892,119600,14.5453
+1897,120000,14.467
+1904,120400,12.7871
+1909,120800,14.9706
+1914,121200,13.51
+1921,121600,12.2767
+1926,122000,16.5016
+1930,122400,20.9919
+1934,122800,18.5356
+1939,123200,11.7085
+1945,123600,14.2539
+1955,124000,7.2888
+1965,124400,6.2359
+1974,124800,5.9543
+1989,125200,3.2663
+2001,125600,4.2557
+2015,126000,5.6426
+2032,126400,2.605
+2046,126800,4.8766
+2059,127200,4.8677
+2070,127600,5.7594
+2081,128000,5.4336
+2092,128400,5.9113
+2103,128800,4.5887
+2114,129200,4.1911
+2122,129600,5.8893
+2134,130000,3.7628
+2144,130400,4.9745
+2152,130800,7.0797
+2158,131200,9.2503
+2168,131600,6.9398
+2176,132000,7.7078
+2185,132400,6.6676
+2192,132800,7.4282
+2201,133200,7.9222
+2208,133600,8.5393
+2214,134000,11.7517
+2219,134400,13.6645
+2225,134800,11.5794
+2232,135200,9.2883
+2237,135600,15.8834
+2242,136000,13.0821
+2249,136400,11.0747
+2255,136800,12.4104
+2263,137200,6.2022
+2268,137600,15.7658
+2272,138000,18.5306
+2283,138400,6.9062
+2289,138800,10.1887
+2296,139200,12.4334
+2300,139600,13.9879
+2304,140000,19.1339
+2312,140400,10.1769
+2317,140800,14.9144
+2322,141200,15.6578
+2327,141600,12.3308
+2333,142000,13.2801
+2340,142400,8.8488
+2345,142800,15.9321
+2351,143200,14.0744
+2357,143600,13.7016
+2361,144000,17.1141
+2366,144400,16.093
+2371,144800,15.3966
+2377,145200,11.9135
+2383,145600,12.9568
+2388,146000,16.5526
+2394,146400,13.9278
+2400,146800,13.1973
+2407,147200,10.658
+2411,147600,17.6656
+2418,148000,9.9268
+2424,148400,12.6209
+2429,148800,15.6302
+2434,149200,15.4722
+2438,149600,15.8488
+2443,150000,15.5069
+2452,150400,8.9388
+2458,150800,11.1183
+2465,151200,12.9985
+2472,151600,9.298
+2482,152000,6.9406
+2488,152400,10.9889
+2494,152800,13.8809
+2500,153200,9.4922
+2508,153600,9.8473
+2517,154000,5.4676
+2521,154400,17.306
+2529,154800,10.5859
+2535,155200,8.9943
+2539,155600,19.018
+2545,156000,10.541
+2553,156400,7.199
+2559,156800,13.1617
+2564,157200,13.507
+2568,157600,19.7962
+2574,158000,15.0976
+2581,158400,9.5981
+2587,158800,11.6193
+2593,159200,11.8875
+2599,159600,12.6915
+2608,160000,7.097
+2616,160400,10.2592
+2622,160800,11.4696
+2627,161200,14.5994
+2632,161600,14.607
+2637,162000,17.4054
+2643,162400,13.1822
+2649,162800,13.4214
+2654,163200,15.3761
+2658,163600,20.4652
+2664,164000,11.7819
+2668,164400,21.1348
+2672,164800,19.7497
+2676,165200,19.9861
+2680,165600,20.6264
+2686,166000,15.0114
+2690,166400,16.8975
+2696,166800,16.6914
+2703,167200,9.2727
+2711,167600,10.1419
+2717,168000,8.6377
+2722,168400,15.6912
+2729,168800,11.2312
+2737,169200,6.9751
+2743,169600,14.1282
+2748,170000,14.9263
+2753,170400,16.8727
+2757,170800,20.6722
+2761,171200,21.0708
+2767,171600,10.5146
+2772,172000,18.3501
+2777,172400,16.4924
+2782,172800,15.8319
+2786,173200,18.9688
+2792,173600,14.616
+2796,174000,18.8745
+2802,174400,13.8155
+2806,174800,20.1499
+2810,175200,16.8318
+2817,175600,10.7324
+2824,176000,11.1007
+2830,176400,12.4775
+2841,176800,5.9412
+2847,177200,13.0488
+2855,177600,9.6263
+2861,178000,9.2555
+2868,178400,13.6159
+2874,178800,10.7991
+2880,179200,13.2894
+2885,179600,12.8225
+2894,180000,10.0183
+2898,180400,18.6829
+2904,180800,11.2808
+2909,181200,18.8581
+2914,181600,15.8593
+2919,182000,15.171
+2923,182400,15.3617
+2929,182800,12.993
+2934,183200,19.4859
+2938,183600,16.5612
+2943,184000,18.0036
+2948,184400,14.3423
+2954,184800,14.2031
+2958,185200,17.8896
+2964,185600,14.1217
+2969,186000,15.7024
+2976,186400,10.0496
+2982,186800,15.1974
+2990,187200,8.1909
+2996,187600,11.4299
+3002,188000,9.0973
+3007,188400,18.8057
+3019,188800,5.6121
+3029,189200,6.4934
+3035,189600,10.5633
+3040,190000,14.1835
+3049,190400,6.6539
+3055,190800,11.4925
+3061,191200,12.6726
+3066,191600,13.2938
+3072,192000,12.4183
+3079,192400,9.0459
+3084,192800,15.8784
+3089,193200,14.8581
+3095,193600,11.5812
+3099,194000,19.8301
+3107,194400,4.9365
+3115,194800,8.6926
+3122,195200,10.0389
+3131,195600,8.3146
+3136,196000,12.6695
+3141,196400,16.2173
+3146,196800,15.5667
+3152,197200,13.6761
+3158,197600,9.9176
+3164,198000,12.4659
+3170,198400,10.7513
+3175,198800,10.0257
+3179,199200,17.8125
+3184,199600,15.2902
+3188,200000,13.6833

code/Lake application/logs/results_1/PPO_frozen_lake_log_1.csv

@@ -0,0 +1,501 @@
+episode,timestep,reward
+27,400,1.0685
+52,800,1.1781
+66,1200,2.0866
+74,1600,3.6635
+83,2000,3.21
+89,2400,4.7845
+96,2800,4.0053
+104,3200,3.8626
+110,3600,4.2643
+117,4000,4.8568
+122,4400,5.5491
+126,4800,7.4627
+131,5200,6.4184
+136,5600,4.5238
+141,6000,6.3553
+145,6400,7.4742
+150,6800,6.7629
+154,7200,6.507
+159,7600,6.5995
+164,8000,5.5815
+168,8400,7.363
+172,8800,7.377
+176,9200,7.3108
+180,9600,7.5117
+184,10000,7.5528
+189,10400,6.4767
+194,10800,6.4779
+199,11200,6.0249
+204,11600,5.7716
+209,12000,6.2201
+213,12400,6.6677
+217,12800,7.5942
+222,13200,6.4747
+228,13600,5.2064
+234,14000,4.7457
+240,14400,4.9933
+245,14800,7.1465
+250,15200,5.3496
+254,15600,6.5622
+259,16000,6.4103
+263,16400,7.161
+268,16800,6.8279
+273,17200,6.076
+278,17600,6.2577
+282,18000,6.3496
+287,18400,6.1414
+292,18800,6.9705
+298,19200,4.5771
+302,19600,7.4473
+306,20000,7.2296
+310,20400,7.5263
+315,20800,5.9007
+319,21200,7.457
+323,21600,7.0129
+327,22000,7.5711
+331,22400,7.4269
+335,22800,7.6694
+339,23200,7.4846
+346,23600,4.3524
+350,24000,7.8691
+354,24400,7.6455
+358,24800,7.412
+363,25200,6.1338
+367,25600,7.6492
+372,26000,7.0091
+376,26400,7.6802
+380,26800,7.2539
+384,27200,7.7116
+388,27600,7.6681
+394,28000,4.7247
+400,28400,4.8051
+404,28800,7.619
+410,29200,6.0282
+414,29600,7.0344
+419,30000,6.6914
+424,30400,5.9087
+428,30800,6.2718
+434,31200,6.0214
+438,31600,7.7125
+444,32000,5.1105
+448,32400,6.5991
+452,32800,7.6193
+458,33200,5.5863
+464,33600,4.8851
+468,34000,6.7645
+474,34400,5.4311
+479,34800,6.3864
+487,35200,3.7461
+494,35600,4.4228
+500,36000,4.2904
+506,36400,5.8171
+510,36800,7.8094
+514,37200,7.1195
+519,37600,6.8759
+524,38000,5.3536
+535,38400,2.7984
+542,38800,4.5735
+549,39200,4.2022
+557,39600,3.886
+562,40000,5.98
+567,40400,5.694
+572,40800,6.5385
+577,41200,5.555
+581,41600,7.3993
+586,42000,5.9369
+591,42400,6.7453
+595,42800,7.2735
+599,43200,7.2897
+604,43600,5.4537
+609,44000,5.0761
+615,44400,5.6804
+620,44800,5.6548
+624,45200,7.1059
+628,45600,7.38
+632,46000,6.6452
+637,46400,6.9389
+641,46800,7.5441
+645,47200,7.4802
+653,47600,3.7831
+659,48000,4.9193
+665,48400,4.5841
+670,48800,5.2625
+676,49200,5.5027
+681,49600,6.0169
+689,50000,3.5328
+694,50400,6.1266
+700,50800,4.9486
+707,51200,4.3697
+713,51600,5.4997
+719,52000,4.4305
+724,52400,5.7043
+731,52800,5.1429
+739,53200,3.6578
+745,53600,5.5292
+751,54000,4.4774
+757,54400,5.5445
+761,54800,7.1383
+766,55200,6.4968
+770,55600,7.5131
+774,56000,7.5454
+779,56400,5.1354
+783,56800,6.7832
+789,57200,5.6604
+793,57600,7.4366
+797,58000,7.4868
+801,58400,7.4437
+805,58800,7.4123
+812,59200,4.6388
+816,59600,7.4665
+820,60000,7.4706
+824,60400,6.3475
+828,60800,7.5096
+832,61200,7.1812
+837,61600,6.3239
+841,62000,6.8265
+847,62400,5.2963
+853,62800,4.758
+857,63200,7.345
+862,63600,6.2916
+866,64000,7.0709
+874,64400,3.7731
+880,64800,4.6813
+888,65200,4.1506
+893,65600,5.6099
+897,66000,6.8958
+903,66400,5.7361
+910,66800,4.2763
+915,67200,5.9698
+922,67600,3.6519
+926,68000,7.6562
+932,68400,6.2379
+936,68800,7.0506
+942,69200,5.6851
+946,69600,7.5529
+950,70000,7.6569
+954,70400,7.489
+958,70800,6.3997
+962,71200,7.6004
+967,71600,6.2412
+971,72000,7.6256
+977,72400,5.6385
+981,72800,7.6622
+985,73200,6.9613
+990,73600,6.4998
+994,74000,7.4835
+998,74400,6.362
+1003,74800,6.4975
+1009,75200,6.0932
+1013,75600,7.3275
+1018,76000,6.3136
+1022,76400,7.918
+1026,76800,7.8966
+1030,77200,8.1455
+1035,77600,6.7381
+1040,78000,6.5592
+1045,78400,6.0788
+1049,78800,7.1046
+1054,79200,7.1041
+1058,79600,7.6982
+1062,80000,8.7302
+1066,80400,6.6165
+1071,80800,6.9873
+1077,81200,6.9054
+1082,81600,7.456
+1087,82000,8.4276
+1093,82400,6.369
+1097,82800,8.6679
+1101,83200,6.9598
+1107,83600,5.4595
+1112,84000,7.8471
+1116,84400,6.8466
+1121,84800,6.1909
+1127,85200,6.153
+1131,85600,7.6274
+1136,86000,6.3568
+1140,86400,7.0727
+1144,86800,8.7111
+1148,87200,7.0302
+1152,87600,7.515
+1157,88000,6.5863
+1162,88400,6.3346
+1166,88800,6.8116
+1171,89200,6.1335
+1175,89600,7.3719
+1179,90000,6.3049
+1183,90400,7.4515
+1188,90800,6.5131
+1193,91200,7.0855
+1198,91600,5.4287
+1203,92000,6.8094
+1209,92400,5.0978
+1214,92800,5.9272
+1220,93200,4.52
+1225,93600,6.2555
+1231,94000,5.4839
+1236,94400,6.4755
+1245,94800,3.9834
+1251,95200,4.4569
+1257,95600,5.508
+1261,96000,7.6455
+1265,96400,7.3739
+1269,96800,7.8249
+1275,97200,5.2307
+1279,97600,7.6121
+1284,98000,6.1267
+1288,98400,7.3416
+1292,98800,7.5182
+1296,99200,7.3272
+1301,99600,6.8465
+1306,100000,6.4113
+1311,100400,5.7506
+1316,100800,7.3543
+1320,101200,7.6077
+1326,101600,5.5944
+1331,102000,7.3631
+1337,102400,5.2
+1343,102800,4.8723
+1348,103200,6.1586
+1355,103600,4.7696
+1360,104000,6.1973
+1364,104400,6.7792
+1368,104800,7.2198
+1373,105200,7.1729
+1379,105600,5.4069
+1385,106000,4.6109
+1390,106400,7.1239
+1395,106800,5.8867
+1399,107200,7.495
+1403,107600,6.6621
+1407,108000,7.4027
+1412,108400,6.3171
+1416,108800,5.6887
+1421,109200,6.6554
+1425,109600,6.75
+1431,110000,5.4458
+1435,110400,7.2516
+1439,110800,6.3838
+1443,111200,6.9752
+1447,111600,7.2899
+1451,112000,7.285
+1455,112400,6.905
+1460,112800,6.7635
+1464,113200,7.2151
+1468,113600,7.3228
+1472,114000,7.2727
+1477,114400,5.1432
+1484,114800,4.5119
+1488,115200,7.5247
+1493,115600,6.6993
+1497,116000,6.583
+1502,116400,5.6599
+1506,116800,7.3572
+1512,117200,4.6733
+1517,117600,6.9534
+1522,118000,5.8504
+1527,118400,5.7037
+1532,118800,5.3677
+1537,119200,6.6015
+1545,119600,4.1934
+1550,120000,6.0054
+1555,120400,6.3411
+1559,120800,7.6877
+1563,121200,7.6184
+1569,121600,4.5052
+1573,122000,7.209
+1579,122400,6.2706
+1584,122800,6.129
+1588,123200,9.4211
+1595,123600,5.3557
+1601,124000,5.9072
+1608,124400,4.905
+1613,124800,8.0799
+1619,125200,5.1058
+1625,125600,5.3964
+1630,126000,6.4494
+1634,126400,6.4167
+1640,126800,5.6178
+1644,127200,6.6824
+1650,127600,5.9365
+1656,128000,4.972
+1663,128400,4.6724
+1669,128800,4.5071
+1676,129200,4.7821
+1683,129600,4.2253
+1689,130000,5.028
+1693,130400,6.5605
+1699,130800,6.1124
+1705,131200,5.982
+1711,131600,5.4071
+1716,132000,6.5883
+1720,132400,8.6701
+1725,132800,5.8316
+1730,133200,6.5105
+1734,133600,8.651
+1739,134000,7.6199
+1743,134400,7.9109
+1749,134800,7.469
+1753,135200,7.2578
+1760,135600,7.0038
+1765,136000,8.4482
+1770,136400,10.9777
+1776,136800,7.8633
+1781,137200,10.7422
+1786,137600,7.7002
+1791,138000,9.8498
+1796,138400,8.3565
+1802,138800,7.2121
+1808,139200,7.437
+1815,139600,5.7484
+1822,140000,7.3232
+1828,140400,8.0316
+1836,140800,5.3387
+1841,141200,7.3611
+1849,141600,6.0461
+1857,142000,5.8818
+1861,142400,8.7706
+1867,142800,12.6086
+1875,143200,4.9306
+1881,143600,8.4787
+1889,144000,6.6935
+1894,144400,9.2201
+1899,144800,9.3988
+1904,145200,11.3811
+1909,145600,8.435
+1916,146000,7.3488
+1922,146400,10.672
+1927,146800,11.5718
+1934,147200,7.5663
+1940,147600,8.4638
+1944,148000,10.8035
+1954,148400,5.3482
+1961,148800,7.711
+1967,149200,11.0491
+1975,149600,6.2135
+1981,150000,10.6626
+1991,150400,5.506
+1996,150800,9.1663
+2004,151200,5.6931
+2009,151600,11.9436
+2018,152000,8.0473
+2025,152400,7.8392
+2032,152800,8.6977
+2041,153200,5.9514
+2045,153600,15.2402
+2052,154000,9.4472
+2059,154400,5.8932
+2065,154800,9.2129
+2074,155200,6.222
+2078,155600,15.2026
+2090,156000,6.9467
+2099,156400,6.1649
+2108,156800,7.4966
+2117,157200,6.7418
+2126,157600,8.2676
+2133,158000,9.3655
+2141,158400,6.6173
+2145,158800,17.2168
+2154,159200,8.7089
+2160,159600,11.2436
+2165,160000,12.7216
+2169,160400,21.6313
+2174,160800,23.2383
+2179,161200,17.6359
+2185,161600,16.5861
+2191,162000,16.226
+2197,162400,17.9568
+2204,162800,12.871
+2209,163200,13.3918
+2216,163600,15.4123
+2223,164000,11.4062
+2228,164400,14.1367
+2237,164800,8.0556
+2244,165200,8.6179
+2256,165600,6.4841
+2266,166000,9.5171
+2274,166400,11.5372
+2283,166800,9.7408
+2287,167200,21.5043
+2291,167600,23.8779
+2296,168000,20.5695
+2301,168400,23.0533
+2309,168800,11.9759
+2314,169200,23.7854
+2320,169600,17.3165
+2327,170000,15.7337
+2332,170400,21.5258
+2337,170800,22.1934
+2344,171200,18.7651
+2351,171600,10.7436
+2360,172000,10.7671
+2369,172400,10.4788
+2382,172800,7.1276
+2386,173200,21.2732
+2392,173600,18.6327
+2397,174000,19.4716
+2404,174400,12.1719
+2409,174800,14.6277
+2416,175200,14.5393
+2422,175600,15.4089
+2429,176000,14.0683
+2435,176400,15.4539
+2441,176800,13.9568
+2446,177200,19.1191
+2456,177600,9.5304
+2466,178000,7.2711
+2472,178400,10.6102
+2478,178800,11.5563
+2482,179200,19.9556
+2490,179600,9.0751
+2497,180000,6.2264
+2502,180400,18.2541
+2507,180800,9.1061
+2513,181200,15.3433
+2517,181600,19.7729
+2522,182000,13.1969
+2527,182400,18.7246
+2531,182800,14.651
+2537,183200,13.6844
+2543,183600,13.292
+2550,184000,16.4504
+2559,184400,8.7936
+2563,184800,22.5295
+2569,185200,15.0557
+2577,185600,10.3489
+2585,186000,11.0047
+2595,186400,9.2111
+2604,186800,11.11
+2611,187200,10.4671
+2617,187600,12.7915
+2624,188000,13.962
+2633,188400,10.0636
+2637,188800,18.8204
+2642,189200,19.8767
+2647,189600,17.3735
+2654,190000,12.2333
+2664,190400,5.6146
+2670,190800,14.4501
+2675,191200,15.3957
+2685,191600,6.7485
+2689,192000,20.351
+2702,192400,4.2184
+2707,192800,16.1191
+2714,193200,11.0071
+2722,193600,6.7454
+2730,194000,4.1497
+2735,194400,6.5892
+2743,194800,10.7816
+2754,195200,3.9205
+2761,195600,10.0928
+2767,196000,14.8486
+2772,196400,17.4269
+2779,196800,10.2417
+2783,197200,13.4426
+2790,197600,10.6094
+2794,198000,17.6529
+2801,198400,7.4825
+2807,198800,9.2602
+2816,199200,4.0249
+2824,199600,6.6758
+2829,200000,16.2271

code/Lake application/logs/results_1/PPO_frozen_lake_log_2.csv

@@ -0,0 +1,501 @@
+episode,timestep,reward
+28,400,1.0712
+50,800,1.338
+64,1200,2.2055
+75,1600,2.7995
+81,2000,5.4941
+92,2400,3.3249
+96,2800,6.413
+101,3200,6.3269
+106,3600,5.3282
+112,4000,5.6833
+117,4400,5.6105
+123,4800,5.1394
+127,5200,7.3238
+131,5600,7.129
+137,6000,7.7241
+142,6400,4.6711
+148,6800,7.9164
+152,7200,6.7923
+157,7600,7.6097
+162,8000,5.8306
+169,8400,5.6442
+173,8800,8.2231
+179,9200,8.5511
+185,9600,6.1404
+189,10000,10.3032
+194,10400,16.185
+198,10800,20.0498
+202,11200,16.4263
+212,11600,6.5354
+217,12000,13.0853
+222,12400,19.0939
+232,12800,7.205
+241,13200,8.4478
+251,13600,7.8135
+261,14000,6.7748
+271,14400,8.0458
+277,14800,15.136
+287,15200,8.4088
+296,15600,9.5744
+301,16000,17.8938
+307,16400,14.878
+317,16800,9.3055
+323,17200,13.5246
+328,17600,18.3417
+334,18000,12.9367
+341,18400,14.7582
+345,18800,19.8885
+352,19200,11.2014
+359,19600,12.73
+365,20000,11.939
+373,20400,10.1943
+380,20800,12.1353
+394,21200,5.5529
+402,21600,8.7544
+411,22000,7.7306
+422,22400,6.3769
+431,22800,8.1486
+441,23200,6.2044
+446,23600,13.7331
+454,24000,10.2596
+460,24400,10.6293
+469,24800,8.1278
+473,25200,20.7824
+480,25600,14.1365
+484,26000,18.8189
+489,26400,15.9557
+499,26800,6.2855
+504,27200,16.2915
+511,27600,7.7517
+518,28000,11.3698
+522,28400,20.7648
+527,28800,19.0116
+533,29200,16.3582
+539,29600,16.8797
+545,30000,12.4403
+552,30400,16.3851
+557,30800,21.8059
+561,31200,23.8707
+569,31600,9.6084
+577,32000,13.8038
+589,32400,5.1129
+597,32800,13.5648
+605,33200,12.1758
+612,33600,11.9826
+620,34000,8.8992
+628,34400,10.1155
+636,34800,7.6714
+643,35200,9.6841
+648,35600,18.1897
+654,36000,10.757
+664,36400,8.2339
+670,36800,14.7556
+677,37200,12.949
+683,37600,12.4321
+694,38000,7.0777
+699,38400,16.2456
+712,38800,7.5439
+720,39200,13.3273
+725,39600,20.2522
+731,40000,20.2798
+737,40400,19.7969
+742,40800,23.1972
+747,41200,21.0181
+754,41600,15.7453
+764,42000,10.3466
+775,42400,9.1961
+782,42800,13.7833
+787,43200,22.5446
+793,43600,20.7285
+797,44000,20.3011
+803,44400,18.0368
+810,44800,17.1513
+816,45200,16.1653
+822,45600,20.1987
+827,46000,17.5718
+834,46400,15.5697
+844,46800,10.5578
+849,47200,22.3577
+854,47600,22.7982
+862,48000,11.8983
+873,48400,8.1632
+880,48800,13.9313
+888,49200,11.4741
+892,49600,24.8084
+899,50000,17.164
+904,50400,19.6258
+911,50800,15.8299
+918,51200,9.6712
+926,51600,10.8926
+933,52000,12.6279
+941,52400,10.5669
+946,52800,19.1119
+955,53200,6.5965
+962,53600,14.0385
+972,54000,5.9704
+977,54400,18.2561
+984,54800,11.3252
+994,55200,7.534
+999,55600,13.1451
+1005,56000,17.4585
+1011,56400,10.8942
+1017,56800,16.6066
+1023,57200,16.5959
+1027,57600,24.5556
+1037,58000,8.4738
+1042,58400,17.3071
+1049,58800,9.78
+1055,59200,15.1944
+1067,59600,5.3205
+1075,60000,11.3803
+1092,60400,3.6245
+1102,60800,5.8269
+1111,61200,9.5032
+1121,61600,5.4864
+1128,62000,6.7626
+1137,62400,9.5821
+1142,62800,16.4103
+1148,63200,11.2262
+1156,63600,8.8738
+1164,64000,9.1704
+1169,64400,10.2279
+1176,64800,11.3308
+1183,65200,10.4271
+1191,65600,7.8356
+1196,66000,13.8851
+1202,66400,18.9843
+1206,66800,24.7673
+1211,67200,18.6684
+1218,67600,16.0951
+1222,68000,22.9115
+1226,68400,23.6065
+1233,68800,11.1388
+1238,69200,14.0507
+1244,69600,15.8132
+1249,70000,13.9042
+1253,70400,24.436
+1263,70800,9.9095
+1268,71200,15.1151
+1273,71600,18.5476
+1278,72000,18.5341
+1283,72400,16.7705
+1290,72800,12.7882
+1297,73200,14.3823
+1303,73600,11.6998
+1309,74000,17.2901
+1315,74400,15.6958
+1323,74800,10.383
+1329,75200,11.5377
+1334,75600,18.225
+1340,76000,16.318
+1347,76400,11.497
+1353,76800,13.7727
+1360,77200,9.5276
+1367,77600,12.1409
+1373,78000,7.7239
+1383,78400,9.336
+1388,78800,14.569
+1392,79200,17.7591
+1396,79600,21.1039
+1401,80000,15.0516
+1407,80400,14.6828
+1412,80800,15.2347
+1422,81200,8.8604
+1429,81600,10.5288
+1436,82000,14.4741
+1443,82400,9.9426
+1450,82800,13.5522
+1457,83200,12.1773
+1466,83600,12.6106
+1473,84000,15.5286
+1483,84400,9.095
+1489,84800,16.7606
+1493,85200,16.0469
+1499,85600,11.1895
+1504,86000,16.8995
+1509,86400,14.7574
+1514,86800,17.2417
+1521,87200,8.4962
+1533,87600,5.2503
+1539,88000,13.6948
+1546,88400,10.7947
+1553,88800,11.7415
+1563,89200,5.8864
+1572,89600,7.1169
+1577,90000,11.0446
+1585,90400,7.4355
+1591,90800,7.9803
+1598,91200,8.522
+1603,91600,10.4511
+1608,92000,12.7399
+1614,92400,8.2374
+1619,92800,12.8639
+1623,93200,11.7356
+1629,93600,12.5085
+1635,94000,8.7022
+1640,94400,13.0884
+1644,94800,18.9683
+1648,95200,17.3632
+1654,95600,13.1206
+1659,96000,16.7717
+1667,96400,7.295
+1672,96800,14.4554
+1679,97200,10.9414
+1686,97600,11.4577
+1694,98000,10.4921
+1699,98400,11.4891
+1706,98800,12.5354
+1715,99200,9.2944
+1721,99600,15.9709
+1725,100000,17.2995
+1734,100400,6.9714
+1738,100800,15.647
+1746,101200,8.3676
+1754,101600,9.6176
+1761,102000,7.512
+1768,102400,10.3183
+1774,102800,13.3747
+1780,103200,11.1231
+1785,103600,13.8813
+1791,104000,17.1172
+1796,104400,12.4113
+1801,104800,16.1103
+1809,105200,6.39
+1818,105600,8.2949
+1825,106000,11.5045
+1831,106400,10.9331
+1837,106800,11.3621
+1842,107200,18.0518
+1848,107600,12.5318
+1855,108000,13.3056
+1861,108400,12.4526
+1867,108800,12.8945
+1876,109200,8.2563
+1882,109600,11.2807
+1888,110000,12.192
+1896,110400,7.9317
+1903,110800,11.2608
+1908,111200,18.0233
+1912,111600,17.8104
+1918,112000,13.7657
+1925,112400,7.795
+1931,112800,13.0999
+1936,113200,12.6435
+1942,113600,9.7847
+1947,114000,15.6282
+1953,114400,13.0466
+1959,114800,13.1941
+1963,115200,17.225
+1970,115600,10.4272
+1976,116000,10.2674
+1984,116400,7.0663
+1991,116800,7.4921
+1996,117200,12.674
+2005,117600,9.7954
+2010,118000,15.9189
+2016,118400,13.78
+2023,118800,10.5507
+2030,119200,7.0213
+2036,119600,16.1446
+2040,120000,20.1122
+2044,120400,17.4097
+2049,120800,13.002
+2054,121200,15.4103
+2058,121600,13.7301
+2062,122000,17.3055
+2069,122400,12.1489
+2075,122800,10.2724
+2084,123200,8.3912
+2088,123600,20.9833
+2094,124000,9.3535
+2102,124400,12.5569
+2106,124800,22.0025
+2111,125200,15.3937
+2117,125600,13.1022
+2127,126000,7.0628
+2132,126400,19.1772
+2138,126800,12.9516
+2146,127200,9.1282
+2154,127600,8.3276
+2160,128000,9.1102
+2167,128400,8.3557
+2175,128800,9.8998
+2181,129200,8.9511
+2190,129600,9.2503
+2197,130000,8.1281
+2203,130400,15.1956
+2207,130800,16.8667
+2221,131200,4.3684
+2232,131600,6.8054
+2237,132000,14.4013
+2244,132400,16.4632
+2254,132800,7.111
+2261,133200,12.7119
+2268,133600,11.1943
+2277,134000,10.9562
+2284,134400,10.6057
+2291,134800,16.0464
+2298,135200,12.4611
+2302,135600,22.5867
+2308,136000,14.8831
+2315,136400,10.9013
+2319,136800,17.5025
+2327,137200,13.1141
+2333,137600,16.9285
+2343,138000,12.4663
+2350,138400,13.3258
+2357,138800,16.7013
+2361,139200,22.2587
+2366,139600,25.103
+2371,140000,25.5381
+2376,140400,16.0396
+2381,140800,12.2065
+2387,141200,22.5733
+2392,141600,25.1197
+2399,142000,13.928
+2403,142400,22.8321
+2412,142800,11.8089
+2419,143200,12.0245
+2426,143600,11.2781
+2433,144000,9.814
+2440,144400,11.4397
+2445,144800,15.5671
+2451,145200,11.2167
+2457,145600,11.7414
+2465,146000,11.176
+2470,146400,17.8477
+2474,146800,23.4131
+2479,147200,17.3667
+2484,147600,15.4173
+2490,148000,15.5236
+2496,148400,11.9975
+2500,148800,15.5211
+2506,149200,9.2949
+2512,149600,13.2005
+2522,150000,6.7119
+2526,150400,13.047
+2532,150800,10.975
+2537,151200,12.4601
+2545,151600,10.3919
+2549,152000,17.1736
+2555,152400,10.9481
+2561,152800,6.5293
+2565,153200,16.5226
+2574,153600,6.976
+2580,154000,9.178
+2586,154400,9.0912
+2593,154800,8.2036
+2599,155200,11.2736
+2604,155600,11.8792
+2612,156000,8.9914
+2619,156400,7.5987
+2624,156800,16.6534
+2631,157200,9.9542
+2637,157600,11.2406
+2646,158000,5.5051
+2651,158400,13.1209
+2659,158800,11.2929
+2666,159200,9.2889
+2673,159600,6.4488
+2679,160000,8.9596
+2685,160400,12.8978
+2691,160800,12.8375
+2696,161200,16.7479
+2701,161600,10.707
+2709,162000,7.5865
+2720,162400,5.9772
+2727,162800,14.3091
+2735,163200,10.0529
+2741,163600,16.2379
+2750,164000,9.3317
+2761,164400,9.3682
+2772,164800,7.5992
+2777,165200,13.7422
+2785,165600,11.4302
+2796,166000,5.5863
+2800,166400,20.5644
+2809,166800,9.1758
+2819,167200,10.6074
+2824,167600,19.2911
+2829,168000,14.7837
+2834,168400,20.5648
+2839,168800,21.9035
+2846,169200,12.8552
+2852,169600,14.4815
+2860,170000,10.327
+2867,170400,16.0522
+2877,170800,7.9339
+2885,171200,9.3754
+2895,171600,8.846
+2901,172000,18.2031
+2911,172400,9.8781
+2920,172800,8.4863
+2925,173200,14.941
+2933,173600,12.9186
+2941,174000,10.819
+2946,174400,17.9883
+2951,174800,16.5405
+2956,175200,18.8158
+2960,175600,23.2525
+2967,176000,13.0999
+2973,176400,13.952
+2977,176800,21.9684
+2985,177200,10.9067
+2991,177600,11.2303
+2997,178000,13.2209
+3005,178400,11.2834
+3009,178800,16.5292
+3013,179200,21.2372
+3022,179600,11.0619
+3031,180000,7.3537
+3036,180400,14.5772
+3045,180800,9.2121
+3057,181200,5.2037
+3067,181600,6.669
+3075,182000,8.7553
+3081,182400,10.2706
+3087,182800,11.9766
+3098,183200,7.7188
+3104,183600,10.9377
+3112,184000,8.698
+3118,184400,8.7785
+3126,184800,7.6664
+3131,185200,13.2695
+3138,185600,8.0803
+3145,186000,12.0663
+3152,186400,9.1547
+3158,186800,10.6509
+3165,187200,11.0623
+3169,187600,19.4497
+3173,188000,18.2049
+3178,188400,14.0501
+3185,188800,11.2683
+3194,189200,6.596
+3202,189600,11.3825
+3210,190000,7.7631
+3214,190400,12.7471
+3222,190800,12.4214
+3228,191200,11.1834
+3234,191600,12.8345
+3242,192000,10.284
+3248,192400,7.1057
+3253,192800,11.0905
+3258,193200,18.9309
+3263,193600,18.4142
+3269,194000,14.4793
+3273,194400,21.8498
+3277,194800,19.3599
+3283,195200,19.47
+3287,195600,26.6692
+3291,196000,21.4347
+3296,196400,20.3488
+3301,196800,18.7587
+3309,197200,8.8581
+3314,197600,8.1858
+3321,198000,11.0028
+3325,198400,17.812
+3329,198800,21.4082
+3333,199200,20.2986
+3337,199600,25.3884
+3342,200000,18.7209

code/Lake application/logs/results_1/PPO_frozen_lake_log_3.csv

@@ -0,0 +1,501 @@
+episode,timestep,reward
+28,400,0.9697
+49,800,1.4299
+59,1200,3.1168
+67,1600,3.4994
+74,2000,4.3512
+78,2400,6.7968
+84,2800,4.9979
+90,3200,4.8983
+97,3600,5.1283
+104,4000,4.4929
+113,4400,3.0465
+121,4800,3.6254
+127,5200,5.7239
+131,5600,6.4944
+138,6000,4.5873
+143,6400,5.5102
+148,6800,6.1279
+155,7200,4.6213
+160,7600,5.1025
+164,8000,7.2864
+168,8400,7.325
+174,8800,5.1145
+179,9200,5.9985
+183,9600,7.3481
+189,10000,5.0562
+194,10400,5.74
+198,10800,6.7543
+205,11200,4.0238
+210,11600,7.0676
+215,12000,4.578
+220,12400,6.2174
+225,12800,6.4531
+231,13200,4.6471
+237,13600,5.7269
+241,14000,7.5008
+247,14400,5.3946
+251,14800,6.9517
+255,15200,7.4972
+261,15600,5.5219
+265,16000,6.6453
+271,16400,5.998
+279,16800,3.6128
+285,17200,4.8651
+290,17600,7.1438
+295,18000,6.1627
+300,18400,6.7364
+305,18800,5.0593
+311,19200,5.698
+317,19600,5.0444
+322,20000,6.426
+329,20400,5.4239
+334,20800,5.0805
+339,21200,7.5365
+343,21600,6.5818
+348,22000,6.7635
+353,22400,6.5158
+360,22800,4.8907
+365,23200,6.5301
+369,23600,7.0247
+376,24000,4.7928
+381,24400,6.8979
+390,24800,3.7186
+397,25200,4.4294
+403,25600,5.0304
+408,26000,6.3346
+412,26400,8.2039
+417,26800,6.1453
+422,27200,7.8806
+429,27600,4.8061
+434,28000,6.9867
+441,28400,4.2048
+447,28800,6.2038
+454,29200,5.0084
+462,29600,4.0663
+466,30000,6.9928
+471,30400,6.8849
+478,30800,4.8687
+482,31200,7.0303
+487,31600,6.6575
+492,32000,7.9083
+496,32400,7.7999
+502,32800,7.0862
+506,33200,7.493
+511,33600,7.9607
+515,34000,10.1515
+519,34400,10.5739
+523,34800,7.9336
+529,35200,8.113
+535,35600,7.7863
+541,36000,7.2677
+546,36400,7.9735
+552,36800,7.6501
+557,37200,7.2945
+565,37600,4.6367
+569,38000,8.4382
+578,38400,4.0414
+583,38800,7.4847
+588,39200,5.5446
+594,39600,6.5673
+599,40000,6.8408
+604,40400,6.5415
+609,40800,6.8515
+613,41200,7.0606
+617,41600,8.0479
+623,42000,5.6694
+627,42400,7.9936
+632,42800,6.7543
+637,43200,5.8305
+643,43600,6.281
+650,44000,3.5108
+656,44400,5.8348
+660,44800,7.7714
+664,45200,7.4986
+671,45600,4.8578
+679,46000,3.9887
+684,46400,6.534
+689,46800,6.7332
+693,47200,7.6821
+698,47600,6.0771
+704,48000,5.4061
+712,48400,3.1593
+717,48800,7.2291
+723,49200,5.0261
+728,49600,8.1633
+733,50000,6.6328
+738,50400,7.6968
+742,50800,9.8616
+747,51200,8.1682
+753,51600,6.0632
+757,52000,6.9269
+761,52400,8.9436
+768,52800,5.1674
+775,53200,5.4459
+779,53600,8.1192
+785,54000,6.4501
+791,54400,5.4877
+795,54800,9.5873
+800,55200,8.2846
+806,55600,6.569
+812,56000,8.7981
+816,56400,9.4428
+823,56800,5.6861
+829,57200,6.7182
+835,57600,6.808
+840,58000,9.3055
+845,58400,7.6702
+851,58800,6.8338
+855,59200,10.6899
+859,59600,9.518
+864,60000,8.8663
+869,60400,7.8428
+873,60800,8.1849
+879,61200,5.1644
+886,61600,6.1759
+893,62000,4.8597
+901,62400,4.9238
+908,62800,4.5957
+913,63200,5.2139
+920,63600,4.4207
+929,64000,3.8686
+933,64400,7.7389
+937,64800,7.038
+942,65200,6.4901
+946,65600,8.3237
+951,66000,6.4676
+956,66400,7.9483
+960,66800,7.4556
+966,67200,5.8683
+972,67600,4.9513
+977,68000,6.3514
+982,68400,7.6564
+989,68800,3.9463
+996,69200,4.8304
+1000,69600,7.4777
+1007,70000,5.3967
+1012,70400,5.9162
+1016,70800,9.1032
+1022,71200,5.6404
+1027,71600,7.1579
+1033,72000,5.5076
+1039,72400,5.1194
+1044,72800,5.7785
+1049,73200,7.2613
+1054,73600,6.0494
+1059,74000,7.0271
+1063,74400,7.5499
+1067,74800,7.0446
+1072,75200,6.0907
+1076,75600,6.7148
+1081,76000,7.3562
+1086,76400,5.7122
+1092,76800,4.6709
+1099,77200,4.9494
+1104,77600,5.152
+1109,78000,7.1534
+1115,78400,5.9638
+1119,78800,7.1621
+1123,79200,8.3579
+1128,79600,8.5829
+1137,80000,4.075
+1144,80400,3.9743
+1149,80800,7.1503
+1155,81200,5.8118
+1161,81600,5.5331
+1167,82000,5.4701
+1176,82400,4.1435
+1180,82800,6.5754
+1186,83200,5.7
+1192,83600,6.4401
+1198,84000,5.9802
+1202,84400,7.2848
+1207,84800,6.2365
+1213,85200,6.6385
+1218,85600,7.7489
+1224,86000,5.8286
+1229,86400,6.656
+1234,86800,7.9111
+1240,87200,5.2668
+1244,87600,8.2528
+1250,88000,5.2783
+1254,88400,7.6867
+1258,88800,7.9634
+1263,89200,7.2483
+1270,89600,6.3149
+1274,90000,7.6798
+1281,90400,6.6529
+1286,90800,6.4735
+1296,91200,3.4323
+1301,91600,7.3159
+1306,92000,8.4848
+1311,92400,6.9908
+1316,92800,8.4663
+1326,93200,3.5854
+1331,93600,8.0148
+1336,94000,7.4098
+1341,94400,6.6913
+1348,94800,4.7574
+1355,95200,5.4236
+1361,95600,6.3331
+1369,96000,4.811
+1373,96400,8.9366
+1377,96800,8.122
+1383,97200,6.3531
+1388,97600,5.7501
+1395,98000,5.7337
+1400,98400,6.524
+1405,98800,5.8413
+1414,99200,3.6238
+1419,99600,6.8038
+1427,100000,4.1713
+1431,100400,6.4416
+1435,100800,7.8524
+1440,101200,7.2999
+1444,101600,7.8906
+1448,102000,7.2445
+1452,102400,7.8529
+1457,102800,7.3003
+1463,103200,5.6059
+1467,103600,7.8356
+1472,104000,6.6397
+1477,104400,6.4739
+1481,104800,6.9889
+1486,105200,6.2231
+1490,105600,7.6594
+1495,106000,6.1959
+1501,106400,5.118
+1507,106800,4.5929
+1513,107200,5.9776
+1518,107600,6.0486
+1522,108000,7.2906
+1526,108400,6.2824
+1531,108800,5.9725
+1537,109200,5.3613
+1541,109600,6.6071
+1546,110000,6.5305
+1551,110400,5.922
+1555,110800,6.7349
+1561,111200,5.5868
+1565,111600,6.026
+1570,112000,6.0956
+1575,112400,6.1964
+1580,112800,6.6359
+1584,113200,6.554
+1590,113600,6.2213
+1598,114000,4.2887
+1604,114400,4.9337
+1610,114800,5.8288
+1614,115200,8.2018
+1618,115600,7.1357
+1623,116000,7.2247
+1629,116400,5.1857
+1633,116800,6.6046
+1638,117200,5.8635
+1642,117600,7.5453
+1648,118000,6.806
+1652,118400,7.8409
+1656,118800,8.5085
+1661,119200,6.7829
+1666,119600,6.8868
+1675,120000,4.2945
+1682,120400,4.4064
+1689,120800,5.0664
+1694,121200,6.5454
+1701,121600,5.6222
+1708,122000,5.4888
+1712,122400,11.0368
+1717,122800,7.0496
+1722,123200,8.6749
+1729,123600,6.9978
+1737,124000,5.4495
+1747,124400,4.0671
+1753,124800,7.6818
+1759,125200,6.4432
+1763,125600,10.7613
+1770,126000,6.4108
+1778,126400,4.3716
+1784,126800,6.7246
+1788,127200,10.4382
+1794,127600,7.1896
+1800,128000,7.588
+1805,128400,9.4403
+1809,128800,11.0176
+1813,129200,11.7979
+1818,129600,8.75
+1827,130000,4.6859
+1834,130400,6.7395
+1839,130800,8.993
+1843,131200,11.3917
+1849,131600,7.4701
+1855,132000,5.7999
+1862,132400,8.5014
+1866,132800,11.0123
+1879,133200,3.7213
+1884,133600,10.6388
+1892,134000,5.8487
+1898,134400,7.1305
+1904,134800,8.8896
+1910,135200,6.4877
+1918,135600,6.8896
+1924,136000,6.4605
+1930,136400,9.0044
+1934,136800,10.5351
+1940,137200,9.2294
+1945,137600,10.9699
+1949,138000,10.0906
+1954,138400,10.2141
+1959,138800,11.3224
+1964,139200,9.5563
+1969,139600,6.9219
+1974,140000,11.1704
+1978,140400,11.1142
+1982,140800,11.9859
+1986,141200,11.8602
+1990,141600,11.6316
+1994,142000,11.6433
+2000,142400,9.2738
+2006,142800,7.5194
+2013,143200,6.4846
+2019,143600,7.8079
+2023,144000,9.8408
+2031,144400,6.9516
+2035,144800,9.8839
+2041,145200,7.7938
+2049,145600,7.6839
+2054,146000,7.4943
+2061,146400,8.0578
+2070,146800,4.8268
+2075,147200,10.4681
+2081,147600,7.9008
+2086,148000,11.0989
+2091,148400,11.994
+2095,148800,9.9046
+2105,149200,4.7372
+2111,149600,9.8328
+2116,150000,11.0763
+2122,150400,10.9598
+2129,150800,7.7438
+2135,151200,8.6623
+2140,151600,12.6654
+2146,152000,8.1182
+2151,152400,9.0062
+2157,152800,8.5988
+2163,153200,8.9284
+2167,153600,14.2702
+2173,154000,10.585
+2178,154400,11.7006
+2186,154800,6.7155
+2192,155200,7.7146
+2198,155600,8.8118
+2203,156000,11.7081
+2212,156400,4.9161
+2217,156800,11.632
+2222,157200,12.2152
+2227,157600,12.1379
+2236,158000,4.0621
+2244,158400,7.7841
+2250,158800,8.4301
+2260,159200,5.0246
+2266,159600,7.0603
+2274,160000,6.5755
+2282,160400,7.0485
+2287,160800,9.5418
+2297,161200,5.8611
+2305,161600,7.2947
+2312,162000,7.2398
+2316,162400,13.7584
+2322,162800,11.7808
+2329,163200,11.604
+2334,163600,13.0979
+2342,164000,8.5548
+2347,164400,14.6783
+2353,164800,12.7043
+2358,165200,13.3911
+2363,165600,11.383
+2371,166000,8.1752
+2376,166400,11.7496
+2382,166800,12.7269
+2388,167200,8.3856
+2397,167600,7.333
+2406,168000,6.2778
+2411,168400,11.0526
+2418,168800,9.0626
+2424,169200,7.5325
+2436,169600,4.0862
+2441,170000,10.966
+2446,170400,9.7353
+2452,170800,8.3233
+2459,171200,5.748
+2465,171600,7.1726
+2472,172000,7.998
+2481,172400,6.561
+2488,172800,6.2343
+2495,173200,7.5135
+2504,173600,4.6772
+2509,174000,9.5455
+2516,174400,7.1717
+2521,174800,9.9983
+2526,175200,8.8316
+2535,175600,4.6613
+2543,176000,4.4261
+2551,176400,7.2283
+2556,176800,14.7301
+2562,177200,11.5262
+2569,177600,9.4323
+2574,178000,14.3105
+2580,178400,11.4739
+2584,178800,14.4407
+2590,179200,13.8673
+2596,179600,13.3536
+2600,180000,12.2952
+2607,180400,11.8378
+2611,180800,18.9288
+2615,181200,13.6898
+2622,181600,10.7245
+2626,182000,17.0192
+2631,182400,17.5475
+2636,182800,14.2294
+2640,183200,18.8072
+2646,183600,10.0944
+2650,184000,14.8466
+2655,184400,12.8789
+2663,184800,7.384
+2668,185200,12.7785
+2680,185600,5.2209
+2687,186000,8.8431
+2695,186400,8.6145
+2703,186800,6.0656
+2708,187200,11.2342
+2714,187600,9.9512
+2723,188000,6.9898
+2729,188400,10.1041
+2739,188800,6.2203
+2744,189200,12.3465
+2754,189600,5.1606
+2761,190000,10.6421
+2767,190400,10.2982
+2777,190800,5.2088
+2782,191200,10.9174
+2786,191600,13.2552
+2791,192000,12.1285
+2798,192400,8.8285
+2805,192800,7.9108
+2814,193200,7.3404
+2819,193600,12.2333
+2825,194000,9.6662
+2832,194400,8.0742
+2837,194800,12.2136
+2843,195200,12.3488
+2850,195600,10.1218
+2856,196000,12.0153
+2862,196400,11.8017
+2869,196800,10.381
+2874,197200,14.0652
+2879,197600,14.9152
+2887,198000,8.0053
+2893,198400,12.0596
+2897,198800,15.0738
+2902,199200,16.9263
+2907,199600,12.2609
+2912,200000,15.858

code/Lake application/logs/results_1/PPO_frozen_lake_log_4.csv

@@ -0,0 +1,501 @@
+episode,timestep,reward
+22,400,1.3382
+40,800,1.8287
+52,1200,2.5231
+66,1600,2.103
+77,2000,3.0155
+85,2400,3.7714
+90,2800,5.8035
+96,3200,5.2274
+100,3600,6.3303
+107,4000,4.8019
+111,4400,6.4724
+117,4800,5.5563
+122,5200,6.2998
+128,5600,5.1646
+132,6000,7.1287
+138,6400,5.8314
+142,6800,7.4052
+147,7200,6.2875
+152,7600,5.6015
+157,8000,6.1754
+162,8400,7.082
+167,8800,6.4028
+171,9200,7.2559
+176,9600,6.9444
+181,10000,6.2933
+186,10400,5.2366
+195,10800,3.9987
+201,11200,5.395
+207,11600,4.9618
+213,12000,5.1983
+219,12400,4.4254
+225,12800,5.7096
+229,13200,7.1898
+235,13600,5.3824
+242,14000,4.5048
+250,14400,3.7881
+255,14800,5.2267
+260,15200,7.031
+264,15600,6.9285
+268,16000,7.5476
+273,16400,5.6767
+278,16800,6.5262
+283,17200,6.1492
+288,17600,5.6468
+292,18000,7.6928
+297,18400,6.6286
+301,18800,7.7454
+305,19200,6.5254
+312,19600,4.8168
+317,20000,6.5406
+322,20400,6.5504
+326,20800,6.7464
+331,21200,6.2836
+336,21600,6.7817
+340,22000,6.6796
+344,22400,7.2005
+351,22800,4.3842
+355,23200,7.6366
+359,23600,7.4904
+365,24000,6.1521
+369,24400,7.6973
+373,24800,7.0403
+377,25200,6.9523
+382,25600,7.1983
+387,26000,6.4586
+391,26400,6.4252
+397,26800,5.499
+402,27200,5.0302
+407,27600,6.8429
+413,28000,4.7347
+418,28400,6.183
+423,28800,6.8443
+427,29200,7.5077
+431,29600,6.0028
+437,30000,6.0114
+441,30400,6.4562
+446,30800,6.7718
+450,31200,6.276
+454,31600,7.5279
+458,32000,7.9159
+462,32400,8.0556
+468,32800,5.5321
+473,33200,5.7863
+479,33600,5.8919
+484,34000,5.3791
+488,34400,7.7837
+494,34800,5.618
+498,35200,7.0261
+502,35600,7.828
+507,36000,6.4049
+512,36400,6.4361
+517,36800,5.7041
+522,37200,7.132
+526,37600,7.4088
+530,38000,7.8924
+535,38400,5.7086
+540,38800,6.5723
+544,39200,7.588
+548,39600,6.1673
+552,40000,7.8064
+557,40400,6.5526
+562,40800,7.5729
+566,41200,6.3143
+570,41600,8.4047
+575,42000,7.2824
+579,42400,7.7924
+583,42800,7.4971
+587,43200,6.6631
+592,43600,7.1095
+596,44000,6.4556
+600,44400,7.6746
+604,44800,7.6558
+609,45200,6.194
+614,45600,7.1185
+619,46000,5.7002
+624,46400,6.5519
+629,46800,6.8268
+633,47200,6.9014
+638,47600,6.301
+644,48000,6.1982
+651,48400,4.9436
+655,48800,6.7595
+661,49200,6.5292
+666,49600,5.158
+671,50000,6.6904
+675,50400,8.0493
+679,50800,7.9991
+687,51200,4.3244
+692,51600,7.5411
+696,52000,9.858
+700,52400,8.5624
+704,52800,7.9155
+709,53200,8.9057
+713,53600,9.5471
+718,54000,7.1981
+722,54400,8.6158
+726,54800,10.1275
+736,55200,4.2344
+742,55600,7.8258
+747,56000,7.7521
+753,56400,9.0208
+759,56800,7.7654
+765,57200,8.6142
+771,57600,6.8121
+776,58000,10.9618
+781,58400,10.3827
+785,58800,10.0289
+791,59200,9.8831
+800,59600,6.9521
+806,60000,7.7517
+815,60400,8.8211
+821,60800,8.9909
+827,61200,8.7221
+836,61600,7.7684
+841,62000,9.6272
+845,62400,14.9976
+850,62800,11.6372
+857,63200,8.8488
+865,63600,7.6496
+871,64000,10.0977
+876,64400,10.3548
+883,64800,10.7205
+889,65200,12.5606
+896,65600,9.843
+902,66000,11.6978
+909,66400,10.94
+915,66800,11.5443
+920,67200,14.4685
+927,67600,7.6312
+934,68000,9.3931
+941,68400,7.5232
+946,68800,13.3119
+952,69200,15.4347
+956,69600,14.1605
+963,70000,12.82
+968,70400,14.5149
+973,70800,14.5627
+979,71200,12.8107
+984,71600,16.1239
+992,72000,8.8389
+997,72400,15.4108
+1008,72800,3.5878
+1015,73200,11.6583
+1023,73600,4.6755
+1029,74000,7.929
+1034,74400,8.3184
+1043,74800,4.4589
+1049,75200,5.8413
+1055,75600,5.8525
+1061,76000,5.7395
+1066,76400,7.915
+1072,76800,6.9911
+1077,77200,9.5236
+1085,77600,6.0534
+1091,78000,6.3831
+1098,78400,5.0208
+1104,78800,8.7623
+1109,79200,6.9585
+1117,79600,6.3551
+1121,80000,14.1976
+1127,80400,9.871
+1131,80800,13.4573
+1138,81200,7.47
+1143,81600,7.7708
+1149,82000,10.2425
+1154,82400,13.5466
+1160,82800,8.8403
+1166,83200,9.1941
+1171,83600,12.7902
+1179,84000,6.3707
+1186,84400,8.839
+1195,84800,6.0659
+1202,85200,9.7067
+1210,85600,5.3875
+1219,86000,7.3249
+1228,86400,8.1348
+1240,86800,4.2997
+1246,87200,7.7715
+1253,87600,9.9431
+1261,88000,5.8097
+1268,88400,9.7243
+1275,88800,9.9173
+1281,89200,8.526
+1291,89600,6.5095
+1296,90000,10.6242
+1305,90400,6.8888
+1313,90800,6.5251
+1320,91200,7.6252
+1330,91600,6.4161
+1339,92000,6.192
+1345,92400,10.4853
+1352,92800,6.5388
+1358,93200,12.3501
+1365,93600,7.43
+1373,94000,7.5638
+1378,94400,15.0633
+1384,94800,14.7513
+1396,95200,6.5226
+1403,95600,8.8394
+1410,96000,7.9987
+1420,96400,7.5662
+1427,96800,8.6258
+1435,97200,9.7086
+1444,97600,8.0059
+1451,98000,10.653
+1459,98400,7.4015
+1471,98800,4.6864
+1479,99200,7.8408
+1485,99600,10.5011
+1491,100000,14.4978
+1498,100400,13.2709
+1504,100800,11.8503
+1511,101200,10.111
+1518,101600,7.1249
+1524,102000,11.0255
+1533,102400,9.3004
+1545,102800,4.3136
+1549,103200,20.1579
+1562,103600,5.6588
+1567,104000,11.4403
+1574,104400,7.7448
+1582,104800,8.4585
+1591,105200,7.9123
+1596,105600,12.7572
+1602,106000,9.7458
+1613,106400,6.0184
+1621,106800,7.3456
+1627,107200,10.2534
+1632,107600,14.4112
+1637,108000,10.8765
+1647,108400,8.215
+1652,108800,14.8476
+1658,109200,13.0967
+1666,109600,9.5115
+1672,110000,12.5662
+1680,110400,9.9636
+1685,110800,17.9725
+1690,111200,21.6135
+1695,111600,16.7148
+1702,112000,11.3757
+1709,112400,11.6943
+1714,112800,20.2134
+1718,113200,15.6719
+1724,113600,17.1174
+1728,114000,21.9856
+1733,114400,14.545
+1741,114800,9.2946
+1745,115200,13.9682
+1751,115600,11.2764
+1758,116000,10.046
+1765,116400,6.6395
+1770,116800,12.6278
+1778,117200,7.5994
+1786,117600,5.1644
+1794,118000,6.6251
+1801,118400,5.2048
+1806,118800,8.7313
+1813,119200,6.7291
+1821,119600,5.8351
+1829,120000,8.9222
+1835,120400,8.7741
+1842,120800,7.2133
+1849,121200,8.8234
+1855,121600,9.6317
+1862,122000,8.2559
+1867,122400,11.0191
+1872,122800,13.0496
+1878,123200,12.1102
+1884,123600,15.2622
+1890,124000,14.949
+1897,124400,14.1959
+1905,124800,10.2858
+1912,125200,10.4039
+1920,125600,9.2182
+1925,126000,18.0285
+1931,126400,14.5642
+1936,126800,15.7313
+1942,127200,16.071
+1949,127600,12.4836
+1955,128000,14.5803
+1964,128400,6.5171
+1973,128800,10.3231
+1979,129200,11.6592
+1985,129600,13.9441
+1991,130000,11.8742
+1997,130400,14.2094
+2002,130800,16.5023
+2008,131200,7.9427
+2013,131600,13.3196
+2017,132000,22.0345
+2023,132400,9.6949
+2029,132800,8.3864
+2033,133200,15.5384
+2038,133600,14.2444
+2043,134000,16.1699
+2047,134400,14.6686
+2052,134800,11.1139
+2057,135200,15.3049
+2061,135600,18.3799
+2066,136000,10.9603
+2070,136400,12.5289
+2076,136800,12.3745
+2080,137200,11.4958
+2086,137600,11.5674
+2094,138000,8.5145
+2099,138400,15.1147
+2104,138800,12.0235
+2110,139200,10.5698
+2116,139600,11.9205
+2123,140000,6.9545
+2130,140400,10.1053
+2136,140800,11.9632
+2143,141200,10.4299
+2151,141600,8.3429
+2165,142000,4.3628
+2172,142400,6.8153
+2179,142800,12.8716
+2187,143200,5.8168
+2196,143600,8.4523
+2202,144000,11.4057
+2208,144400,10.0338
+2219,144800,6.4849
+2224,145200,12.969
+2228,145600,18.6417
+2233,146000,12.536
+2238,146400,15.4672
+2243,146800,13.8039
+2248,147200,14.223
+2254,147600,12.1347
+2261,148000,8.5235
+2267,148400,15.2563
+2272,148800,11.314
+2278,149200,12.9462
+2285,149600,9.7554
+2291,150000,14.0195
+2296,150400,15.3385
+2303,150800,9.973
+2308,151200,18.1009
+2312,151600,20.9784
+2321,152000,10.5002
+2335,152400,4.4978
+2342,152800,10.5598
+2350,153200,9.1714
+2357,153600,12.176
+2364,154000,11.5123
+2368,154400,20.0547
+2373,154800,17.2117
+2379,155200,15.459
+2387,155600,11.3856
+2393,156000,10.4216
+2398,156400,18.6778
+2406,156800,9.8121
+2413,157200,15.4442
+2420,157600,8.2398
+2429,158000,7.9804
+2436,158400,10.0033
+2444,158800,11.684
+2451,159200,11.7212
+2458,159600,11.1657
+2463,160000,12.9049
+2474,160400,7.2923
+2480,160800,11.1158
+2488,161200,7.0536
+2496,161600,12.9054
+2503,162000,10.8589
+2510,162400,7.6154
+2516,162800,15.1175
+2526,163200,6.0549
+2535,163600,6.8174
+2543,164000,11.3806
+2548,164400,14.9169
+2555,164800,8.7519
+2561,165200,12.8943
+2568,165600,7.4888
+2577,166000,12.9756
+2586,166400,9.7037
+2595,166800,7.6698
+2604,167200,8.0272
+2609,167600,17.4422
+2614,168000,21.842
+2619,168400,24.0025
+2624,168800,21.9548
+2629,169200,16.2903
+2636,169600,15.3797
+2644,170000,12.3509
+2651,170400,17.7225
+2657,170800,17.4392
+2667,171200,7.6192
+2672,171600,19.9947
+2677,172000,21.4869
+2682,172400,16.9837
+2689,172800,15.026
+2698,173200,8.2427
+2704,173600,9.9491
+2712,174000,12.5259
+2717,174400,17.4708
+2723,174800,12.1165
+2729,175200,15.3909
+2736,175600,12.777
+2741,176000,21.2902
+2745,176400,26.1009
+2751,176800,17.7154
+2757,177200,14.3114
+2762,177600,21.2451
+2766,178000,24.6657
+2773,178400,11.323
+2778,178800,18.9868
+2789,179200,10.2768
+2795,179600,16.099
+2802,180000,9.7036
+2811,180400,10.1027
+2817,180800,14.6335
+2821,181200,20.4911
+2825,181600,23.7041
+2830,182000,19.1257
+2834,182400,21.7177
+2838,182800,22.6593
+2843,183200,21.3371
+2849,183600,12.1633
+2855,184000,18.8629
+2859,184400,16.2873
+2866,184800,16.9789
+2871,185200,13.7611
+2878,185600,12.7885
+2883,186000,18.4539
+2888,186400,19.4485
+2894,186800,17.3323
+2899,187200,16.1448
+2909,187600,9.09
+2921,188000,5.7227
+2930,188400,9.5406
+2939,188800,8.9963
+2948,189200,9.0039
+2959,189600,6.1213
+2968,190000,10.7098
+2973,190400,27.5424
+2981,190800,11.4104
+2988,191200,14.7688
+2996,191600,8.1336
+3006,192000,9.3062
+3013,192400,9.5427
+3020,192800,12.1444
+3027,193200,13.9121
+3033,193600,11.6091
+3043,194000,9.565
+3050,194400,12.3977
+3058,194800,13.0409
+3064,195200,19.9469
+3071,195600,12.6553
+3078,196000,11.394
+3082,196400,23.2945
+3087,196800,25.3409
+3093,197200,20.015
+3099,197600,16.1347
+3103,198000,15.8059
+3108,198400,16.2553
+3116,198800,12.0655
+3123,199200,7.5759
+3133,199600,8.3166
+3146,200000,5.2906

code/Lake application/logs/results_1/PPO_frozen_lake_log_5.csv

@@ -0,0 +1,501 @@
+episode,timestep,reward
+24,400,1.1896
+45,800,1.4148
+59,1200,2.3332
+66,1600,5.1201
+70,2000,6.5847
+78,2400,5.1845
+84,2800,5.4028
+91,3200,4.5706
+98,3600,4.8473
+103,4000,6.01
+108,4400,5.9648
+114,4800,5.9022
+118,5200,7.4097
+123,5600,6.5452
+128,6000,6.3157
+133,6400,7.2965
+143,6800,3.5149
+147,7200,7.2122
+151,7600,7.6719
+156,8000,7.8449
+160,8400,6.3771
+168,8800,4.1202
+172,9200,7.7297
+179,9600,5.2521
+184,10000,5.0265
+190,10400,6.301
+195,10800,6.4812
+200,11200,5.9884
+204,11600,8.2287
+208,12000,6.8769
+213,12400,8.2376
+217,12800,6.7639
+222,13200,7.491
+226,13600,7.14
+234,14000,4.8056
+238,14400,8.2315
+244,14800,5.6201
+250,15200,5.6978
+254,15600,8.6572
+259,16000,7.1277
+264,16400,6.3588
+276,16800,3.2533
+285,17200,4.354
+295,17600,3.6256
+302,18000,5.482
+312,18400,4.9095
+324,18800,3.2072
+332,19200,4.2376
+337,19600,7.9829
+343,20000,7.3154
+352,20400,5.8945
+359,20800,5.4014
+367,21200,6.2058
+372,21600,8.3138
+377,22000,7.7193
+385,22400,5.7761
+392,22800,6.301
+397,23200,7.7678
+402,23600,7.7383
+408,24000,8.0968
+414,24400,7.2615
+420,24800,8.6811
+428,25200,6.1812
+434,25600,5.421
+439,26000,10.3048
+444,26400,8.6396
+450,26800,5.8203
+455,27200,8.955
+460,27600,10.1203
+464,28000,10.9113
+469,28400,10.8469
+475,28800,8.0835
+481,29200,8.4523
+486,29600,9.6786
+494,30000,5.8029
+499,30400,9.7283
+503,30800,10.3865
+509,31200,7.4219
+516,31600,6.7019
+521,32000,8.9637
+526,32400,8.0468
+531,32800,9.7588
+538,33200,7.8747
+542,33600,8.1689
+546,34000,11.1529
+550,34400,11.3951
+555,34800,9.1521
+559,35200,10.907
+563,35600,11.4909
+569,36000,8.8749
+573,36400,11.4543
+577,36800,10.4455
+583,37200,6.9897
+587,37600,9.4309
+592,38000,10.4979
+596,38400,10.9061
+600,38800,11.3928
+606,39200,9.6524
+610,39600,11.4807
+616,40000,7.0779
+620,40400,7.8871
+630,40800,5.189
+637,41200,4.9294
+645,41600,5.8549
+649,42000,9.1055
+655,42400,7.5844
+660,42800,7.9374
+665,43200,8.81
+669,43600,11.0942
+675,44000,8.1994
+681,44400,5.7097
+686,44800,5.6185
+693,45200,6.8876
+697,45600,9.3624
+702,46000,10.4714
+707,46400,8.134
+711,46800,8.3317
+717,47200,8.2672
+722,47600,8.0033
+729,48000,5.1115
+737,48400,6.7697
+743,48800,6.1466
+750,49200,6.2071
+756,49600,8.233
+761,50000,6.4818
+766,50400,7.5639
+771,50800,8.9587
+778,51200,7.244
+783,51600,8.6157
+790,52000,6.8854
+796,52400,6.3743
+800,52800,7.0537
+806,53200,5.9831
+812,53600,7.1028
+817,54000,7.4769
+822,54400,6.6734
+828,54800,7.8816
+833,55200,7.9481
+837,55600,8.0056
+842,56000,8.3952
+847,56400,7.193
+852,56800,6.8126
+857,57200,6.8457
+861,57600,8.5708
+867,58000,7.7666
+872,58400,8.4442
+877,58800,8.728
+882,59200,6.7609
+887,59600,6.3465
+892,60000,9.3179
+896,60400,10.0812
+901,60800,7.6496
+906,61200,8.6439
+914,61600,5.8768
+921,62000,5.4985
+928,62400,6.8348
+932,62800,11.426
+938,63200,7.9831
+942,63600,11.7064
+947,64000,8.5804
+956,64400,4.5109
+963,64800,6.216
+969,65200,6.0191
+976,65600,5.3416
+983,66000,5.4151
+989,66400,6.187
+993,66800,8.3532
+998,67200,5.5044
+1003,67600,7.4823
+1009,68000,6.0798
+1015,68400,5.3716
+1021,68800,6.6076
+1029,69200,4.1927
+1035,69600,5.568
+1040,70000,8.2748
+1044,70400,8.0315
+1049,70800,7.3339
+1054,71200,8.792
+1059,71600,5.9685
+1064,72000,9.0921
+1070,72400,6.5296
+1075,72800,6.9532
+1080,73200,7.8572
+1087,73600,6.9338
+1092,74000,7.5691
+1096,74400,7.7686
+1105,74800,5.2799
+1111,75200,6.5266
+1121,75600,4.676
+1131,76000,4.0666
+1136,76400,7.5432
+1142,76800,9.6043
+1147,77200,10.452
+1153,77600,9.5079
+1161,78000,5.3496
+1168,78400,7.87
+1172,78800,12.6175
+1177,79200,11.3827
+1184,79600,9.1898
+1189,80000,9.0837
+1194,80400,10.6039
+1204,80800,5.8527
+1213,81200,4.7043
+1221,81600,5.8605
+1228,82000,6.6981
+1233,82400,9.605
+1241,82800,5.4104
+1246,83200,8.432
+1251,83600,7.9885
+1255,84000,10.6539
+1261,84400,7.0366
+1268,84800,5.8069
+1273,85200,6.9464
+1280,85600,5.2223
+1286,86000,5.3272
+1292,86400,6.4975
+1301,86800,3.7054
+1307,87200,5.0905
+1314,87600,5.9251
+1320,88000,5.9757
+1325,88400,7.8605
+1330,88800,6.1258
+1335,89200,6.4772
+1342,89600,4.6821
+1348,90000,6.1312
+1356,90400,4.4604
+1360,90800,8.8793
+1366,91200,5.4831
+1372,91600,6.6693
+1377,92000,6.0235
+1382,92400,7.023
+1386,92800,7.991
+1391,93200,9.647
+1399,93600,4.4356
+1404,94000,7.8905
+1413,94400,4.2128
+1418,94800,7.0636
+1423,95200,8.5693
+1430,95600,6.583
+1436,96000,7.1519
+1444,96400,4.0687
+1454,96800,3.9372
+1460,97200,6.4719
+1467,97600,7.3715
+1477,98000,4.4424
+1486,98400,5.2201
+1495,98800,5.0595
+1500,99200,8.9895
+1507,99600,5.1904
+1512,100000,7.0454
+1520,100400,5.0524
+1526,100800,6.979
+1530,101200,8.5726
+1538,101600,5.8817
+1545,102000,4.4648
+1552,102400,6.082
+1560,102800,5.4946
+1567,103200,5.3969
+1574,103600,5.1586
+1580,104000,5.9475
+1588,104400,5.6989
+1595,104800,7.2335
+1602,105200,5.8916
+1606,105600,9.3439
+1613,106000,9.341
+1622,106400,6.1297
+1626,106800,12.5539
+1631,107200,8.9207
+1637,107600,9.1629
+1643,108000,8.1442
+1650,108400,6.787
+1656,108800,8.6608
+1664,109200,5.6852
+1673,109600,5.7596
+1678,110000,11.7301
+1684,110400,6.4299
+1691,110800,8.9632
+1700,111200,5.1691
+1707,111600,6.5305
+1717,112000,4.326
+1723,112400,8.8668
+1731,112800,6.4173
+1737,113200,8.5648
+1741,113600,11.226
+1750,114000,5.9103
+1757,114400,8.2391
+1761,114800,11.72
+1765,115200,12.409
+1770,115600,9.8668
+1775,116000,9.7725
+1783,116400,5.4277
+1787,116800,11.838
+1793,117200,8.9856
+1802,117600,4.9619
+1809,118000,7.6212
+1814,118400,9.6064
+1820,118800,9.063
+1826,119200,8.8093
+1832,119600,8.2819
+1839,120000,6.0866
+1844,120400,9.4807
+1849,120800,11.4309
+1854,121200,8.7056
+1859,121600,9.2045
+1866,122000,7.9125
+1873,122400,7.3911
+1883,122800,3.7769
+1889,123200,7.0604
+1896,123600,5.8807
+1902,124000,8.3782
+1907,124400,9.9949
+1911,124800,12.1608
+1916,125200,10.2111
+1923,125600,6.3728
+1927,126000,11.6701
+1932,126400,10.0707
+1937,126800,9.8646
+1943,127200,8.6133
+1948,127600,6.8825
+1952,128000,11.6098
+1957,128400,10.7833
+1967,128800,4.9028
+1974,129200,4.8043
+1981,129600,7.1724
+1988,130000,6.6265
+1992,130400,12.4505
+1997,130800,10.4089
+2002,131200,9.3429
+2008,131600,8.49
+2016,132000,5.6689
+2022,132400,7.8804
+2027,132800,10.9685
+2033,133200,7.3669
+2040,133600,7.722
+2046,134000,7.4917
+2050,134400,12.4121
+2055,134800,8.4809
+2060,135200,8.2016
+2065,135600,9.6964
+2069,136000,11.8806
+2074,136400,10.897
+2081,136800,6.2054
+2088,137200,6.4216
+2097,137600,5.6088
+2105,138000,6.3134
+2111,138400,7.3101
+2118,138800,7.7567
+2122,139200,11.3647
+2127,139600,7.5965
+2133,140000,8.5471
+2140,140400,7.6505
+2148,140800,5.5044
+2153,141200,10.6672
+2158,141600,9.3921
+2164,142000,9.4556
+2169,142400,8.4184
+2174,142800,12.2423
+2180,143200,10.4702
+2192,143600,3.7695
+2199,144000,7.5407
+2209,144400,5.2107
+2218,144800,5.2279
+2223,145200,10.1944
+2232,145600,6.5609
+2239,146000,7.3471
+2246,146400,9.7593
+2256,146800,6.1584
+2265,147200,6.7017
+2272,147600,8.3735
+2278,148000,12.8356
+2284,148400,10.6073
+2290,148800,11.7075
+2294,149200,15.3994
+2298,149600,14.141
+2304,150000,12.4809
+2310,150400,9.9578
+2319,150800,5.6223
+2327,151200,9.5834
+2333,151600,10.7389
+2341,152000,7.72
+2346,152400,7.7508
+2352,152800,12.4856
+2357,153200,11.9052
+2363,153600,10.7156
+2373,154000,4.0923
+2382,154400,6.618
+2388,154800,7.4782
+2399,155200,5.56
+2409,155600,4.0632
+2418,156000,6.6121
+2425,156400,8.951
+2433,156800,6.1173
+2439,157200,10.4851
+2449,157600,6.8041
+2455,158000,9.7919
+2461,158400,14.9732
+2466,158800,17.2664
+2476,159200,7.0055
+2480,159600,18.4795
+2488,160000,7.4399
+2493,160400,13.4248
+2497,160800,17.9206
+2505,161200,10.3671
+2513,161600,7.8602
+2520,162000,7.5847
+2530,162400,6.8028
+2537,162800,8.7183
+2544,163200,10.3386
+2551,163600,11.3944
+2556,164000,10.885
+2566,164400,7.0053
+2575,164800,7.0606
+2580,165200,14.0458
+2584,165600,16.733
+2595,166000,6.4971
+2606,166400,5.5624
+2612,166800,12.141
+2616,167200,20.5602
+2625,167600,7.7261
+2634,168000,7.5007
+2639,168400,13.3516
+2645,168800,13.6817
+2651,169200,10.8423
+2657,169600,11.7492
+2664,170000,9.1222
+2670,170400,10.5513
+2677,170800,10.756
+2682,171200,14.8444
+2687,171600,13.6451
+2692,172000,11.6598
+2699,172400,10.1388
+2708,172800,8.8513
+2720,173200,4.2224
+2727,173600,8.3196
+2732,174000,11.9491
+2740,174400,6.6871
+2748,174800,7.4226
+2753,175200,10.0185
+2760,175600,10.3176
+2764,176000,11.3026
+2770,176400,9.9799
+2776,176800,9.911
+2782,177200,7.9342
+2789,177600,6.5082
+2796,178000,6.9011
+2801,178400,11.9291
+2806,178800,9.9604
+2814,179200,7.9044
+2820,179600,13.3649
+2826,180000,9.8404
+2832,180400,12.8414
+2836,180800,18.641
+2843,181200,11.7401
+2850,181600,12.4612
+2859,182000,8.9368
+2864,182400,12.2839
+2869,182800,15.863
+2877,183200,8.5494
+2884,183600,11.6935
+2896,184000,6.2298
+2903,184400,10.0828
+2909,184800,11.6092
+2914,185200,11.3742
+2922,185600,10.9442
+2932,186000,5.9435
+2936,186400,15.5259
+2944,186800,8.2002
+2950,187200,12.5212
+2957,187600,9.7587
+2962,188000,12.3502
+2969,188400,10.8997
+2976,188800,8.21
+2984,189200,8.5636
+2990,189600,12.3993
+2996,190000,12.8622
+3000,190400,19.3607
+3006,190800,9.842
+3010,191200,14.5131
+3020,191600,5.0736
+3030,192000,4.7978
+3042,192400,3.6577
+3049,192800,8.4608
+3056,193200,7.3668
+3068,193600,4.3313
+3075,194000,7.6891
+3080,194400,12.0177
+3086,194800,8.6174
+3093,195200,7.7047
+3099,195600,8.0317
+3105,196000,8.0931
+3114,196400,5.4667
+3122,196800,6.6043
+3132,197200,5.0706
+3140,197600,8.1802
+3149,198000,6.0144
+3160,198400,5.1743
+3168,198800,6.8536
+3178,199200,3.8362
+3186,199600,6.7096
+3192,200000,8.9841

code/Lake application/logs/results_2/PDPPO_frozen_lake_log_1.csv

@@ -0,0 +1,501 @@
+episode,timestep,reward
+22,400,1.2526
+48,800,1.1762
+66,1200,1.698
+86,1600,1.5699
+100,2000,2.2196
+114,2400,2.2243
+130,2800,1.9494
+144,3200,2.1269
+161,3600,1.8416
+173,4000,2.4108
+186,4400,2.3831
+195,4800,3.7798
+204,5200,3.3905
+212,5600,3.7893
+221,6000,2.6417
+230,6400,3.9088
+245,6800,2.3282
+256,7200,2.7405
+263,7600,4.6528
+273,8000,3.0146
+281,8400,3.2375
+291,8800,3.553
+297,9200,4.7911
+307,9600,3.4636
+314,10000,4.4748
+320,10400,4.6664
+329,10800,3.4853
+336,11200,4.7085
+341,11600,6.6859
+349,12000,3.6321
+355,12400,5.3428
+362,12800,4.1236
+367,13200,6.1173
+373,13600,5.3324
+377,14000,7.2656
+381,14400,7.7223
+386,14800,5.9847
+390,15200,7.6658
+395,15600,5.8334
+400,16000,7.1939
+404,16400,7.1478
+408,16800,7.5988
+414,17200,6.1732
+419,17600,6.1007
+423,18000,7.2673
+427,18400,7.41
+432,18800,7.0193
+438,19200,5.3807
+443,19600,5.9541
+447,20000,7.6451
+452,20400,7.0583
+457,20800,7.3468
+461,21200,7.6243
+466,21600,6.6661
+473,22000,4.9625
+479,22400,4.5857
+484,22800,6.9212
+488,23200,6.1686
+493,23600,7.897
+497,24000,7.9243
+501,24400,8.608
+506,24800,6.6952
+511,25200,6.2059
+516,25600,7.1384
+522,26000,5.7323
+527,26400,5.9145
+534,26800,5.6198
+538,27200,7.6158
+542,27600,7.7017
+546,28000,7.8191
+551,28400,8.2935
+556,28800,7.3137
+562,29200,5.8557
+570,29600,4.3915
+576,30000,4.9182
+581,30400,6.3954
+585,30800,7.2091
+590,31200,9.0975
+594,31600,5.9126
+600,32000,5.1805
+606,32400,6.9075
+611,32800,6.2059
+616,33200,4.6794
+621,33600,7.0498
+626,34000,7.4649
+631,34400,6.6183
+635,34800,8.8603
+639,35200,6.275
+643,35600,9.1126
+648,36000,6.9308
+653,36400,6.7373
+657,36800,7.6857
+662,37200,7.836
+666,37600,6.3725
+671,38000,6.2922
+676,38400,6.6979
+680,38800,7.6388
+684,39200,7.0079
+688,39600,7.5892
+692,40000,7.7355
+697,40400,6.5238
+702,40800,6.181
+706,41200,6.5753
+711,41600,6.499
+716,42000,6.1642
+721,42400,6.3709
+725,42800,6.5148
+730,43200,6.1071
+734,43600,7.2231
+739,44000,6.9884
+743,44400,7.6752
+747,44800,7.645
+751,45200,7.6253
+756,45600,6.9888
+760,46000,6.551
+764,46400,8.8765
+768,46800,7.644
+772,47200,7.7078
+776,47600,7.7402
+780,48000,7.7096
+786,48400,6.4542
+790,48800,6.8511
+795,49200,6.7727
+799,49600,7.6417
+803,50000,7.6993
+807,50400,7.7043
+812,50800,6.8261
+816,51200,6.92
+820,51600,7.9919
+825,52000,6.4103
+830,52400,5.7618
+834,52800,7.0424
+838,53200,7.6928
+842,53600,7.7354
+847,54000,7.3411
+852,54400,6.3078
+856,54800,7.4612
+860,55200,6.6696
+865,55600,6.7569
+869,56000,6.9279
+874,56400,6.5996
+878,56800,7.7271
+882,57200,7.6932
+886,57600,7.5902
+890,58000,7.1683
+897,58400,4.5812
+903,58800,6.1682
+908,59200,6.1325
+912,59600,7.8386
+916,60000,6.3943
+921,60400,6.8943
+925,60800,7.8821
+932,61200,4.9096
+937,61600,6.3275
+941,62000,8.5893
+945,62400,7.9071
+949,62800,7.8808
+953,63200,7.8324
+957,63600,8.5609
+961,64000,8.4952
+965,64400,7.8793
+971,64800,5.9811
+975,65200,6.3148
+979,65600,8.3047
+985,66000,6.6049
+991,66400,5.5465
+995,66800,6.7472
+1000,67200,6.688
+1004,67600,8.5059
+1008,68000,8.0416
+1012,68400,9.3594
+1017,68800,7.135
+1022,69200,7.7882
+1028,69600,5.1304
+1032,70000,7.9267
+1036,70400,7.9253
+1040,70800,7.522
+1045,71200,6.7979
+1050,71600,7.0769
+1055,72000,7.1814
+1059,72400,7.9156
+1063,72800,6.9876
+1069,73200,5.6939
+1073,73600,7.603
+1078,74000,6.9544
+1085,74400,5.1872
+1089,74800,8.0712
+1094,75200,5.4866
+1099,75600,6.856
+1104,76000,6.6695
+1111,76400,5.2366
+1115,76800,8.231
+1120,77200,6.3017
+1125,77600,6.1984
+1129,78000,8.0981
+1134,78400,5.6874
+1139,78800,6.7436
+1144,79200,6.7781
+1148,79600,8.035
+1153,80000,7.0241
+1157,80400,8.1216
+1161,80800,7.7847
+1165,81200,7.0458
+1172,81600,4.9159
+1176,82000,8.8252
+1180,82400,8.1435
+1184,82800,8.0794
+1188,83200,8.4439
+1194,83600,6.5179
+1198,84000,9.5129
+1202,84400,9.5982
+1206,84800,8.1605
+1213,85200,5.1735
+1218,85600,6.3955
+1222,86000,6.5836
+1227,86400,7.8827
+1231,86800,9.0583
+1236,87200,7.4464
+1242,87600,6.9449
+1247,88000,7.6785
+1251,88400,7.7551
+1255,88800,9.1958
+1260,89200,7.8329
+1265,89600,9.6998
+1269,90000,12.2941
+1274,90400,7.8743
+1281,90800,6.7261
+1285,91200,10.3406
+1289,91600,10.3629
+1294,92000,12.8416
+1298,92400,11.8637
+1302,92800,10.95
+1307,93200,12.6815
+1312,93600,10.2359
+1316,94000,14.5616
+1320,94400,12.2057
+1325,94800,10.3496
+1331,95200,10.0749
+1336,95600,12.5332
+1341,96000,7.9488
+1347,96400,11.6734
+1353,96800,8.6258
+1359,97200,8.6982
+1363,97600,16.8782
+1369,98000,13.076
+1374,98400,12.6862
+1380,98800,13.0069
+1385,99200,13.2064
+1394,99600,7.5732
+1399,100000,13.571
+1406,100400,11.8885
+1411,100800,14.0567
+1416,101200,14.4459
+1422,101600,14.7572
+1430,102000,9.0038
+1436,102400,11.7543
+1443,102800,12.3206
+1448,103200,13.6686
+1454,103600,14.6563
+1461,104000,10.5416
+1467,104400,11.8187
+1473,104800,12.6766
+1478,105200,13.234
+1487,105600,10.4377
+1494,106000,10.4682
+1499,106400,14.2136
+1505,106800,13.3214
+1512,107200,9.9131
+1518,107600,11.3795
+1524,108000,13.6525
+1531,108400,10.5166
+1536,108800,15.6816
+1541,109200,13.5686
+1546,109600,15.8877
+1552,110000,10.798
+1556,110400,15.505
+1564,110800,8.7457
+1571,111200,9.8442
+1576,111600,16.783
+1581,112000,16.6507
+1587,112400,12.5058
+1597,112800,7.0879
+1604,113200,9.4795
+1612,113600,8.8247
+1619,114000,9.8475
+1626,114400,9.1637
+1631,114800,14.0166
+1638,115200,8.0027
+1642,115600,17.5209
+1650,116000,7.7073
+1654,116400,20.1086
+1661,116800,7.9623
+1668,117200,12.2066
+1675,117600,8.7988
+1681,118000,14.56
+1686,118400,16.3382
+1691,118800,12.1992
+1699,119200,10.4266
+1703,119600,19.3562
+1708,120000,11.3536
+1712,120400,20.2356
+1716,120800,19.5323
+1721,121200,17.4551
+1728,121600,10.1801
+1732,122000,16.947
+1739,122400,9.827
+1745,122800,13.3383
+1749,123200,19.4348
+1755,123600,12.4996
+1761,124000,13.8883
+1765,124400,20.16
+1769,124800,19.6822
+1776,125200,10.0647
+1780,125600,18.9827
+1785,126000,10.8816
+1791,126400,12.6382
+1797,126800,11.2938
+1806,127200,8.8997
+1811,127600,15.7514
+1817,128000,9.8611
+1825,128400,10.0803
+1833,128800,9.5121
+1841,129200,7.1428
+1850,129600,8.5931
+1858,130000,9.1178
+1866,130400,10.6061
+1871,130800,15.7645
+1875,131200,19.0111
+1881,131600,10.6172
+1885,132000,20.6998
+1889,132400,20.2484
+1894,132800,16.4684
+1900,133200,13.5329
+1904,133600,20.9186
+1910,134000,15.0977
+1915,134400,14.7358
+1920,134800,15.5285
+1928,135200,8.3647
+1934,135600,11.9479
+1940,136000,14.4666
+1947,136400,9.6302
+1953,136800,13.6634
+1958,137200,14.0396
+1966,137600,9.1948
+1973,138000,9.3788
+1982,138400,9.0964
+1991,138800,7.105
+1998,139200,11.6642
+2005,139600,8.2024
+2014,140000,7.7126
+2021,140400,11.0422
+2027,140800,11.2108
+2032,141200,15.9589
+2039,141600,8.9692
+2046,142000,12.0016
+2052,142400,13.3516
+2056,142800,19.3885
+2061,143200,14.0276
+2066,143600,14.164
+2073,144000,11.3607
+2082,144400,6.5908
+2089,144800,10.36
+2094,145200,14.2824
+2105,145600,4.9849
+2111,146000,11.1523
+2116,146400,12.4602
+2123,146800,10.201
+2129,147200,11.5534
+2138,147600,5.9679
+2142,148000,16.4718
+2147,148400,17.5833
+2152,148800,16.768
+2156,149200,19.1331
+2162,149600,12.3424
+2166,150000,18.0845
+2170,150400,18.9557
+2175,150800,14.0846
+2179,151200,20.1474
+2184,151600,15.3493
+2188,152000,18.1315
+2193,152400,15.8044
+2197,152800,17.7751
+2203,153200,11.6984
+2209,153600,13.0837
+2213,154000,15.855
+2218,154400,16.8814
+2223,154800,16.0311
+2227,155200,15.3192
+2234,155600,11.457
+2239,156000,16.1934
+2246,156400,11.9881
+2252,156800,9.2926
+2257,157200,16.73
+2261,157600,14.6868
+2268,158000,7.9747
+2273,158400,10.7901
+2277,158800,18.3051
+2283,159200,14.0301
+2288,159600,15.5108
+2294,160000,10.0966
+2300,160400,8.0898
+2306,160800,8.0323
+2315,161200,5.836
+2321,161600,10.9128
+2329,162000,5.4973
+2334,162400,11.0091
+2340,162800,8.3576
+2349,163200,6.2928
+2354,163600,13.008
+2358,164000,15.7582
+2364,164400,10.6657
+2372,164800,7.3352
+2378,165200,7.9416
+2385,165600,9.7033
+2390,166000,14.8003
+2395,166400,13.5949
+2399,166800,16.5582
+2405,167200,13.1281
+2410,167600,13.6583
+2415,168000,13.3859
+2422,168400,12.7336
+2427,168800,13.4928
+2432,169200,14.222
+2438,169600,12.2052
+2442,170000,17.538
+2449,170400,13.4975
+2454,170800,13.4316
+2460,171200,14.3904
+2466,171600,14.1914
+2472,172000,10.7212
+2478,172400,13.2043
+2484,172800,12.3505
+2488,173200,18.4707
+2493,173600,13.7594
+2498,174000,15.2434
+2503,174400,17.6234
+2509,174800,11.5482
+2513,175200,18.5919
+2518,175600,18.2289
+2525,176000,10.1992
+2529,176400,19.2187
+2533,176800,20.1763
+2538,177200,15.8706
+2542,177600,20.1437
+2547,178000,14.6606
+2552,178400,16.5006
+2557,178800,17.2947
+2561,179200,20.4127
+2566,179600,16.8026
+2571,180000,16.4537
+2575,180400,19.0096
+2580,180800,12.4961
+2586,181200,13.702
+2590,181600,19.4121
+2595,182000,19.1734
+2599,182400,15.6252
+2604,182800,17.4543
+2608,183200,18.4527
+2613,183600,17.0079
+2617,184000,15.8574
+2623,184400,14.6486
+2627,184800,16.7431
+2632,185200,16.3204
+2636,185600,18.1816
+2640,186000,19.3138
+2646,186400,13.6693
+2652,186800,12.1288
+2656,187200,19.5385
+2662,187600,12.7194
+2666,188000,18.8978
+2671,188400,17.7266
+2675,188800,20.3572
+2680,189200,16.1169
+2685,189600,14.712
+2692,190000,11.1334
+2696,190400,17.4639
+2701,190800,16.8716
+2705,191200,19.8286
+2710,191600,18.7337
+2714,192000,19.8553
+2719,192400,14.2238
+2725,192800,12.8373
+2735,193200,7.0151
+2739,193600,13.3644
+2745,194000,13.3385
+2753,194400,8.739
+2758,194800,13.8917
+2764,195200,10.3893
+2773,195600,4.8033
+2784,196000,5.5534
+2792,196400,7.2659
+2796,196800,13.9818
+2802,197200,13.2211
+2807,197600,16.0152
+2814,198000,9.9562
+2823,198400,6.7525
+2828,198800,14.4238
+2833,199200,10.7252
+2838,199600,13.7153
+2842,200000,19.2825

code/Lake application/logs/results_2/PDPPO_frozen_lake_log_2.csv

@@ -0,0 +1,501 @@
+episode,timestep,reward
+25,400,1.2001
+50,800,1.1974
+67,1200,1.7766
+87,1600,1.4873
+104,2000,1.7077
+114,2400,2.5908
+134,2800,1.8992
+145,3200,2.8642
+154,3600,2.9615
+163,4000,3.7629
+170,4400,4.1044
+180,4800,3.4985
+187,5200,4.3872
+192,5600,5.6852
+198,6000,5.7852
+205,6400,4.5497
+210,6800,5.1203
+215,7200,6.3622
+222,7600,4.9101
+230,8000,3.7216
+237,8400,3.9412
+243,8800,5.1797
+250,9200,4.9764
+255,9600,5.0597
+260,10000,6.6622
+266,10400,5.4034
+274,10800,3.7803
+279,11200,4.8601
+284,11600,6.8704
+289,12000,5.884
+298,12400,3.1085
+306,12800,3.7402
+317,13200,2.4474
+325,13600,4.697
+333,14000,3.7155
+340,14400,4.1133
+345,14800,6.3342
+354,15200,3.4713
+363,15600,2.973
+369,16000,5.3844
+376,16400,3.9942
+384,16800,4.3917
+390,17200,3.9032
+396,17600,5.5522
+402,18000,5.0959
+410,18400,4.0555
+419,18800,3.8592
+424,19200,5.5717
+430,19600,5.4533
+436,20000,4.4112
+442,20400,5.2485
+453,20800,3.6337
+458,21200,5.7439
+464,21600,5.8554
+470,22000,6.1356
+478,22400,4.1016
+483,22800,6.9287
+487,23200,7.9029
+492,23600,7.2121
+496,24000,6.8316
+501,24400,7.3779
+506,24800,7.0025
+511,25200,7.3224
+516,25600,6.0262
+521,26000,7.3493
+525,26400,8.5734
+530,26800,7.505
+536,27200,5.6146
+541,27600,7.32
+547,28000,5.8791
+551,28400,8.3691
+555,28800,8.5242
+560,29200,6.8016
+564,29600,6.4966
+570,30000,6.5884
+574,30400,7.9429
+579,30800,8.2591
+586,31200,5.1781
+590,31600,6.725
+595,32000,7.3339
+602,32400,5.5153
+606,32800,7.294
+611,33200,7.861
+617,33600,5.8614
+622,34000,6.2895
+626,34400,7.641
+631,34800,7.1765
+635,35200,8.3742
+639,35600,8.5644
+644,36000,7.5061
+648,36400,7.137
+654,36800,6.317
+659,37200,7.5171
+663,37600,8.4671
+667,38000,8.8055
+672,38400,7.2286
+677,38800,7.0709
+682,39200,6.0232
+687,39600,7.7556
+691,40000,7.7135
+695,40400,8.3451
+701,40800,6.0775
+705,41200,8.3363
+709,41600,8.6071
+715,42000,5.3694
+720,42400,7.6011
+724,42800,8.7117
+729,43200,7.0684
+735,43600,5.433
+739,44000,8.6411
+743,44400,8.5924
+747,44800,8.9168
+752,45200,7.2905
+756,45600,8.9831
+761,46000,7.4007
+766,46400,6.664
+770,46800,7.3928
+777,47200,5.7024
+783,47600,5.151
+788,48000,7.2108
+793,48400,7.2763
+799,48800,7.0493
+803,49200,8.5485
+808,49600,7.5421
+812,50000,9.5257
+816,50400,9.1144
+822,50800,5.6572
+828,51200,7.8776
+832,51600,10.0686
+838,52000,6.2858
+842,52400,8.7905
+847,52800,6.9064
+852,53200,8.591
+857,53600,7.9326
+861,54000,10.7555
+865,54400,11.0084
+870,54800,8.5343
+874,55200,10.2597
+878,55600,11.0706
+883,56000,9.7567
+887,56400,11.0066
+891,56800,11.0205
+897,57200,8.6028
+902,57600,7.8706
+906,58000,11.6827
+911,58400,11.5678
+916,58800,7.3302
+920,59200,11.8686
+925,59600,9.6659
+929,60000,11.8218
+933,60400,11.753
+938,60800,10.8838
+942,61200,11.9606
+946,61600,11.9891
+950,62000,10.7408
+955,62400,9.9649
+959,62800,10.1192
+963,63200,11.9243
+967,63600,12.0996
+971,64000,12.2641
+975,64400,12.2119
+980,64800,9.6896
+984,65200,11.985
+988,65600,12.0389
+993,66000,9.7261
+998,66400,10.0918
+1002,66800,11.9409
+1007,67200,9.7976
+1011,67600,11.9562
+1015,68000,11.9393
+1019,68400,11.9489
+1023,68800,10.6665
+1029,69200,8.6481
+1035,69600,8.2744
+1040,70000,9.8761
+1044,70400,12.7648
+1048,70800,12.0824
+1053,71200,9.9615
+1059,71600,7.5596
+1063,72000,12.066
+1068,72400,10.3717
+1072,72800,11.9771
+1076,73200,11.9456
+1080,73600,9.1042
+1084,74000,12.076
+1090,74400,9.4681
+1094,74800,11.529
+1098,75200,12.0008
+1103,75600,10.1566
+1108,76000,9.751
+1112,76400,12.3104
+1116,76800,9.2405
+1120,77200,12.0348
+1125,77600,10.0069
+1130,78000,10.0386
+1136,78400,10.4839
+1142,78800,7.6819
+1149,79200,5.9398
+1156,79600,7.5437
+1163,80000,7.4114
+1167,80400,9.315
+1172,80800,11.2425
+1176,81200,10.7758
+1180,81600,12.0153
+1185,82000,10.7721
+1190,82400,9.8986
+1194,82800,11.9053
+1198,83200,11.9558
+1203,83600,9.3269
+1207,84000,12.2099
+1213,84400,8.4628
+1217,84800,10.7038
+1221,85200,12.2837
+1226,85600,9.9915
+1231,86000,9.9481
+1236,86400,10.1545
+1240,86800,12.2327
+1244,87200,12.075
+1248,87600,12.01
+1252,88000,10.5855
+1256,88400,12.134
+1260,88800,12.3757
+1264,89200,11.9803
+1269,89600,9.3804
+1274,90000,9.8415
+1278,90400,11.9713
+1284,90800,8.3152
+1289,91200,8.8216
+1293,91600,12.3776
+1299,92000,8.471
+1303,92400,12.5027
+1307,92800,12.517
+1313,93200,8.4122
+1317,93600,13.1839
+1322,94000,10.4515
+1326,94400,11.1728
+1330,94800,13.0548
+1334,95200,12.0536
+1338,95600,12.4526
+1344,96000,9.0872
+1351,96400,7.9055
+1355,96800,11.2378
+1360,97200,11.1121
+1364,97600,12.7524
+1368,98000,12.6111
+1373,98400,11.2153
+1378,98800,8.2927
+1382,99200,12.5971
+1388,99600,9.538
+1392,100000,12.5941
+1397,100400,10.2156
+1401,100800,12.5003
+1406,101200,9.1204
+1411,101600,11.3426
+1418,102000,6.7031
+1422,102400,13.0065
+1427,102800,10.4644
+1431,103200,12.6849
+1435,103600,12.3948
+1441,104000,9.7853
+1446,104400,9.4532
+1450,104800,10.5596
+1456,105200,8.6362
+1461,105600,11.1625
+1465,106000,9.7383
+1471,106400,9.5542
+1476,106800,10.1016
+1480,107200,11.3806
+1484,107600,12.8012
+1488,108000,12.754
+1493,108400,9.2045
+1497,108800,12.9866
+1504,109200,7.3446
+1509,109600,11.8225
+1514,110000,8.9346
+1518,110400,12.7821
+1523,110800,11.2293
+1527,111200,12.976
+1532,111600,8.6775
+1536,112000,12.7469
+1540,112400,12.7591
+1545,112800,10.3778
+1550,113200,10.3744
+1554,113600,12.7611
+1558,114000,11.8172
+1562,114400,12.8452
+1566,114800,12.6974
+1570,115200,12.7894
+1575,115600,10.9595
+1579,116000,13.0989
+1583,116400,12.9278
+1589,116800,9.0682
+1593,117200,12.9978
+1597,117600,10.7087
+1601,118000,13.5158
+1605,118400,13.435
+1610,118800,11.5973
+1614,119200,13.0533
+1618,119600,13.0017
+1623,120000,10.8166
+1629,120400,9.0293
+1634,120800,9.2539
+1639,121200,10.6134
+1644,121600,11.5216
+1648,122000,13.3206
+1652,122400,12.207
+1657,122800,11.3809
+1661,123200,10.1333
+1665,123600,13.4154
+1669,124000,13.7245
+1673,124400,13.2766
+1677,124800,13.6751
+1681,125200,13.3002
+1686,125600,11.3154
+1690,126000,14.442
+1697,126400,9.1752
+1701,126800,12.6002
+1710,127200,8.7327
+1715,127600,11.4596
+1722,128000,9.0605
+1730,128400,6.6071
+1738,128800,7.8556
+1743,129200,8.6705
+1748,129600,13.2619
+1757,130000,8.0688
+1762,130400,11.255
+1769,130800,10.1434
+1776,131200,7.9637
+1780,131600,16.234
+1784,132000,16.009
+1790,132400,11.5696
+1795,132800,13.03
+1801,133200,11.5952
+1807,133600,10.1351
+1811,134000,14.54
+1823,134400,4.607
+1829,134800,12.6576
+1833,135200,16.8332
+1838,135600,11.8201
+1842,136000,17.1005
+1846,136400,16.8673
+1851,136800,15.3565
+1857,137200,8.8723
+1862,137600,13.1234
+1868,138000,12.9135
+1876,138400,6.6299
+1883,138800,8.6404
+1892,139200,8.0842
+1896,139600,16.6064
+1900,140000,12.9285
+1912,140400,5.249
+1918,140800,9.7863
+1926,141200,7.8766
+1932,141600,7.9322
+1942,142000,5.3181
+1947,142400,12.7024
+1956,142800,8.2081
+1968,143200,2.9574
+1975,143600,6.7944
+1981,144000,11.6649
+1994,144400,5.4107
+2003,144800,6.3419
+2012,145200,6.9728
+2023,145600,5.0183
+2031,146000,7.8319
+2041,146400,4.6116
+2054,146800,4.337
+2063,147200,5.3691
+2069,147600,12.9849
+2078,148000,8.5405
+2091,148400,4.9827
+2108,148800,3.026
+2123,149200,3.3827
+2132,149600,6.8501
+2141,150000,7.5638
+2152,150400,5.2852
+2162,150800,7.5752
+2170,151200,7.9508
+2177,151600,10.2038
+2191,152000,3.0956
+2200,152400,7.7211
+2208,152800,6.9913
+2215,153200,10.9929
+2219,153600,15.1937
+2226,154000,12.1173
+2235,154400,6.3827
+2244,154800,9.904
+2251,155200,10.4535
+2257,155600,11.1928
+2262,156000,11.8628
+2269,156400,11.1129
+2275,156800,10.8419
+2282,157200,10.6974
+2288,157600,12.6491
+2293,158000,17.1227
+2297,158400,15.1925
+2302,158800,15.8225
+2310,159200,10.5351
+2314,159600,20.5124
+2318,160000,20.7472
+2323,160400,16.1619
+2328,160800,17.0157
+2332,161200,19.5865
+2336,161600,20.3359
+2340,162000,15.7826
+2344,162400,20.7786
+2349,162800,19.027
+2353,163200,16.8306
+2358,163600,15.0345
+2364,164000,13.7065
+2370,164400,15.8193
+2375,164800,15.9792
+2379,165200,16.8467
+2384,165600,17.832
+2388,166000,20.4626
+2393,166400,14.8119
+2399,166800,12.4114
+2403,167200,20.9186
+2408,167600,13.3934
+2412,168000,20.5788
+2418,168400,16.2933
+2422,168800,18.8223
+2427,169200,19.3578
+2432,169600,13.3396
+2437,170000,17.3548
+2443,170400,16.2848
+2448,170800,18.1538
+2453,171200,16.7561
+2457,171600,17.7607
+2463,172000,15.1953
+2469,172400,13.134
+2476,172800,12.5457
+2482,173200,14.6165
+2487,173600,18.5189
+2492,174000,19.1314
+2496,174400,17.353
+2500,174800,22.18
+2506,175200,15.5115
+2511,175600,19.8744
+2515,176000,18.1736
+2519,176400,21.4187
+2525,176800,17.5231
+2531,177200,12.1789
+2536,177600,17.784
+2540,178000,20.6193
+2545,178400,17.4618
+2549,178800,21.499
+2553,179200,17.4586
+2557,179600,20.6227
+2561,180000,20.1806
+2567,180400,13.3524
+2573,180800,14.6734
+2577,181200,21.1076
+2582,181600,17.1151
+2587,182000,12.7979
+2592,182400,15.9493
+2599,182800,14.3687
+2604,183200,15.514
+2610,183600,12.1476
+2616,184000,13.7643
+2622,184400,14.6858
+2627,184800,13.8197
+2631,185200,20.2988
+2637,185600,13.8708
+2641,186000,21.6494
+2646,186400,15.6542
+2653,186800,11.9435
+2659,187200,13.8883
+2666,187600,12.5814
+2671,188000,17.3771
+2678,188400,11.9801
+2683,188800,13.1353
+2692,189200,10.337
+2698,189600,11.3343
+2705,190000,12.8742
+2711,190400,14.6771
+2717,190800,11.9189
+2726,191200,8.7059
+2732,191600,13.1508
+2739,192000,9.7473
+2743,192400,19.009
+2748,192800,16.8758
+2754,193200,15.7392
+2758,193600,20.0703
+2764,194000,12.4249
+2768,194400,21.7073
+2773,194800,18.1804
+2779,195200,15.5854
+2785,195600,12.3091
+2790,196000,15.0442
+2794,196400,20.9159
+2802,196800,11.6643
+2811,197200,7.4276
+2819,197600,8.6076
+2826,198000,10.6653
+2831,198400,16.6902
+2837,198800,13.4796
+2843,199200,14.7899
+2848,199600,14.6657
+2855,200000,11.2761

code/Lake application/logs/results_2/PDPPO_frozen_lake_log_3.csv

@@ -0,0 +1,501 @@
+episode,timestep,reward
+20,400,1.4511
+42,800,1.4159
+60,1200,1.7077
+73,1600,2.1135
+84,2000,2.7468
+105,2400,1.7674
+113,2800,3.7814
+122,3200,3.7896
+134,3600,2.7067
+146,4000,2.5524
+152,4400,6.5864
+157,4800,6.5195
+162,5200,6.1349
+168,5600,5.317
+177,6000,3.6724
+183,6400,6.1446
+189,6800,5.5608
+195,7200,6.1726
+199,7600,6.344
+205,8000,5.8716
+213,8400,4.0247
+217,8800,6.9902
+224,9200,6.2118
+229,9600,6.7983
+236,10000,6.4537
+241,10400,9.274
+246,10800,7.4492
+253,11200,6.7536
+257,11600,10.0378
+264,12000,5.3493
+270,12400,6.8639
+274,12800,9.7271
+278,13200,10.6744
+284,13600,7.8632
+290,14000,8.3459
+296,14400,6.5538
+301,14800,9.4489
+307,15200,8.4332
+313,15600,7.6024
+318,16000,9.6646
+323,16400,7.5682
+327,16800,12.2827
+332,17200,11.2367
+336,17600,10.6158
+341,18000,8.7318
+347,18400,8.9077
+352,18800,10.0014
+357,19200,9.9727
+362,19600,8.4138
+368,20000,8.7518
+372,20400,12.3645
+376,20800,10.8224
+381,21200,11.2952
+387,21600,7.5035
+392,22000,10.1362
+398,22400,8.0769
+402,22800,11.4559
+408,23200,10.9315
+412,23600,12.5869
+416,24000,9.7607
+421,24400,10.3972
+427,24800,9.1654
+434,25200,6.5726
+440,25600,7.9518
+445,26000,12.163
+449,26400,12.5389
+453,26800,12.4047
+457,27200,12.8434
+461,27600,12.508
+465,28000,12.7977
+469,28400,12.7722
+475,28800,7.5211
+481,29200,8.8252
+485,29600,11.5555
+489,30000,11.477
+493,30400,12.9982
+497,30800,13.1206
+504,31200,9.1732
+509,31600,9.6877
+515,32000,8.65
+522,32400,6.6807
+529,32800,7.7346
+534,33200,7.8476
+541,33600,7.9549
+548,34000,6.9946
+555,34400,5.7617
+562,34800,7.4759
+567,35200,9.4369
+574,35600,8.9111
+581,36000,6.5248
+586,36400,10.6508
+594,36800,4.8766
+606,37200,5.0255
+611,37600,8.7737
+619,38000,7.297
+625,38400,8.7064
+631,38800,7.6392
+637,39200,8.1329
+643,39600,8.6996
+648,40000,12.1593
+654,40400,8.0097
+658,40800,13.7355
+663,41200,10.8767
+667,41600,13.4623
+671,42000,13.7128
+677,42400,7.9521
+685,42800,7.0389
+693,43200,7.1228
+699,43600,7.1906
+704,44000,10.2499
+711,44400,8.8439
+715,44800,10.723
+722,45200,8.0561
+726,45600,12.8837
+730,46000,11.6005
+735,46400,10.5382
+740,46800,10.9579
+744,47200,12.8439
+749,47600,8.5832
+755,48000,10.2625
+759,48400,11.0394
+764,48800,10.5021
+771,49200,7.5662
+775,49600,11.8596
+779,50000,10.9197
+785,50400,9.6613
+791,50800,7.1842
+797,51200,8.9205
+810,51600,3.7799
+814,52000,11.7011
+818,52400,13.1067
+825,52800,6.8794
+837,53200,4.1435
+844,53600,7.8758
+850,54000,6.8512
+857,54400,7.279
+863,54800,8.8461
+870,55200,5.1664
+876,55600,8.291
+882,56000,7.4186
+886,56400,12.7099
+895,56800,5.8991
+900,57200,10.3267
+905,57600,10.4081
+910,58000,8.2201
+917,58400,7.2555
+923,58800,8.6902
+929,59200,8.4311
+935,59600,9.3899
+940,60000,9.48
+947,60400,8.8701
+952,60800,10.6712
+957,61200,12.1303
+963,61600,10.0041
+967,62000,11.8506
+975,62400,7.9768
+981,62800,10.121
+985,63200,12.7484
+992,63600,8.6506
+996,64000,12.0166
+1003,64400,8.6989
+1007,64800,13.5841
+1012,65200,11.8948
+1020,65600,7.5362
+1024,66000,14.5183
+1031,66400,8.4667
+1039,66800,6.099
+1044,67200,9.9433
+1051,67600,10.2187
+1056,68000,12.4386
+1065,68400,4.846
+1071,68800,12.9973
+1076,69200,11.6068
+1084,69600,9.6249
+1090,70000,8.821
+1098,70400,8.9721
+1104,70800,10.8627
+1110,71200,12.9207
+1117,71600,9.6223
+1124,72000,9.5217
+1129,72400,12.7009
+1137,72800,7.1291
+1142,73200,15.8915
+1147,73600,12.773
+1152,74000,14.0152
+1158,74400,12.8197
+1165,74800,7.6715
+1171,75200,11.6273
+1176,75600,14.7481
+1182,76000,10.9798
+1186,76400,17.6803
+1193,76800,10.2361
+1198,77200,11.135
+1203,77600,13.5392
+1207,78000,16.3684
+1213,78400,12.9063
+1218,78800,11.9918
+1226,79200,8.0133
+1231,79600,13.8717
+1236,80000,13.592
+1240,80400,16.6494
+1247,80800,9.0394
+1253,81200,11.6721
+1257,81600,12.4117
+1263,82000,13.1805
+1268,82400,12.8848
+1273,82800,13.0017
+1279,83200,11.932
+1286,83600,10.3167
+1292,84000,11.0747
+1298,84400,12.7847
+1302,84800,16.4195
+1307,85200,14.5389
+1313,85600,11.3515
+1317,86000,19.0045
+1324,86400,12.204
+1328,86800,18.172
+1334,87200,11.4799
+1339,87600,15.2817
+1346,88000,12.8543
+1351,88400,15.2124
+1355,88800,20.012
+1360,89200,15.6753
+1364,89600,18.9953
+1369,90000,14.7316
+1373,90400,18.3781
+1379,90800,12.0495
+1383,91200,19.1038
+1388,91600,15.5228
+1394,92000,13.1508
+1399,92400,13.739
+1404,92800,15.4669
+1409,93200,12.2052
+1413,93600,18.4534
+1417,94000,18.4736
+1421,94400,19.451
+1426,94800,17.0253
+1431,95200,13.6257
+1436,95600,15.849
+1447,96000,6.1302
+1452,96400,13.5995
+1460,96800,10.4263
+1466,97200,12.8175
+1470,97600,19.8985
+1474,98000,19.4499
+1480,98400,12.2035
+1485,98800,15.5921
+1493,99200,9.945
+1500,99600,8.3822
+1504,100000,20.1108
+1511,100400,13.2678
+1517,100800,13.3653
+1521,101200,20.5694
+1526,101600,12.9576
+1531,102000,18.6283
+1537,102400,13.6185
+1542,102800,13.5109
+1547,103200,14.7459
+1555,103600,11.7803
+1562,104000,8.6873
+1572,104400,8.425
+1577,104800,12.4127
+1583,105200,11.9189
+1590,105600,12.8694
+1597,106000,10.6233
+1604,106400,8.3289
+1613,106800,9.3008
+1622,107200,7.9757
+1627,107600,15.1024
+1633,108000,12.5267
+1641,108400,10.7986
+1646,108800,10.9844
+1650,109200,20.3921
+1657,109600,12.9882
+1664,110000,8.8107
+1671,110400,13.2145
+1677,110800,13.7578
+1683,111200,11.6441
+1691,111600,10.6352
+1698,112000,9.8465
+1704,112400,13.4316
+1708,112800,19.3443
+1715,113200,11.1754
+1722,113600,10.6371
+1728,114000,9.1292
+1734,114400,14.4382
+1741,114800,13.7379
+1745,115200,20.8923
+1749,115600,16.3165
+1755,116000,15.3684
+1762,116400,9.6267
+1766,116800,17.2582
+1772,117200,14.4907
+1777,117600,16.4743
+1783,118000,13.0177
+1792,118400,7.393
+1799,118800,10.4686
+1804,119200,17.1223
+1814,119600,7.6813
+1820,120000,10.9129
+1826,120400,15.8259
+1830,120800,20.2207
+1837,121200,9.1836
+1843,121600,14.5043
+1848,122000,12.8295
+1853,122400,18.3265
+1858,122800,16.1205
+1863,123200,19.6436
+1869,123600,11.276
+1873,124000,20.749
+1882,124400,9.1372
+1888,124800,13.6346
+1893,125200,17.135
+1899,125600,10.655
+1904,126000,20.1275
+1908,126400,19.847
+1913,126800,15.7564
+1918,127200,14.3012
+1923,127600,15.9916
+1928,128000,15.8541
+1932,128400,20.358
+1937,128800,16.4342
+1941,129200,21.0778
+1946,129600,15.8157
+1952,130000,14.1684
+1956,130400,17.5727
+1962,130800,13.636
+1967,131200,16.7871
+1971,131600,19.9122
+1976,132000,17.2468
+1984,132400,10.4792
+1989,132800,15.3806
+1994,133200,15.9361
+1998,133600,20.595
+2003,134000,14.5683
+2010,134400,12.9443
+2017,134800,11.1499
+2021,135200,16.4114
+2028,135600,11.15
+2033,136000,16.1641
+2039,136400,11.8905
+2045,136800,12.094
+2051,137200,13.1055
+2057,137600,12.3794
+2062,138000,16.7795
+2066,138400,15.2186
+2071,138800,17.5386
+2075,139200,19.7387
+2080,139600,17.7345
+2087,140000,9.286
+2091,140400,20.1843
+2095,140800,20.6579
+2099,141200,20.4451
+2105,141600,12.451
+2111,142000,13.8034
+2116,142400,16.789
+2121,142800,17.3337
+2127,143200,12.512
+2132,143600,18.0873
+2137,144000,16.5148
+2143,144400,14.2245
+2150,144800,11.4969
+2158,145200,6.4741
+2167,145600,10.4108
+2173,146000,9.8208
+2180,146400,14.0351
+2186,146800,11.3705
+2191,147200,17.1942
+2196,147600,16.5483
+2201,148000,17.2101
+2206,148400,15.8254
+2211,148800,16.944
+2216,149200,13.2711
+2222,149600,16.0584
+2227,150000,15.4412
+2231,150400,20.0904
+2235,150800,18.9388
+2241,151200,10.2277
+2248,151600,9.7938
+2257,152000,6.8651
+2265,152400,10.0151
+2269,152800,16.5873
+2275,153200,13.0192
+2280,153600,16.4787
+2284,154000,19.9945
+2288,154400,21.1409
+2293,154800,17.4874
+2298,155200,16.9422
+2302,155600,16.5356
+2307,156000,17.7096
+2314,156400,11.4627
+2318,156800,20.033
+2323,157200,14.9603
+2327,157600,19.5649
+2334,158000,10.3648
+2339,158400,17.1039
+2343,158800,21.143
+2348,159200,17.2194
+2352,159600,21.0071
+2356,160000,21.8045
+2361,160400,17.6473
+2367,160800,15.5177
+2371,161200,20.4818
+2376,161600,17.1564
+2381,162000,14.4102
+2386,162400,18.4027
+2390,162800,22.3808
+2395,163200,14.4186
+2400,163600,13.4343
+2404,164000,19.605
+2409,164400,14.7853
+2415,164800,11.0992
+2421,165200,12.6908
+2426,165600,17.289
+2431,166000,17.1976
+2435,166400,21.6014
+2441,166800,12.793
+2445,167200,14.232
+2450,167600,15.6217
+2457,168000,14.2949
+2461,168400,16.1677
+2465,168800,21.0584
+2470,169200,19.4555
+2474,169600,16.2403
+2481,170000,10.8878
+2491,170400,4.966
+2497,170800,11.4854
+2509,171200,4.8386
+2517,171600,9.0965
+2532,172000,4.4732
+2539,172400,10.3322
+2548,172800,6.617
+2559,173200,6.6052
+2566,173600,8.1639
+2571,174000,14.7295
+2575,174400,19.1853
+2581,174800,11.4953
+2586,175200,15.2959
+2591,175600,17.3706
+2597,176000,16.6239
+2602,176400,16.1638
+2608,176800,13.7811
+2614,177200,11.7461
+2624,177600,8.2969
+2629,178000,16.7226
+2636,178400,10.9358
+2642,178800,16.5572
+2649,179200,10.4538
+2655,179600,12.1145
+2660,180000,17.3922
+2669,180400,10.0921
+2675,180800,14.2412
+2680,181200,16.8869
+2684,181600,17.4712
+2689,182000,19.3122
+2693,182400,22.4468
+2699,182800,15.3699
+2704,183200,18.0721
+2710,183600,13.5026
+2717,184000,12.2414
+2721,184400,17.901
+2726,184800,16.2374
+2731,185200,17.4196
+2736,185600,18.1275
+2740,186000,20.9471
+2745,186400,20.6423
+2750,186800,18.2358
+2755,187200,17.1191
+2760,187600,17.9405
+2766,188000,12.8474
+2770,188400,19.6427
+2776,188800,15.7317
+2780,189200,19.7856
+2787,189600,10.9979
+2792,190000,18.7782
+2798,190400,13.124
+2803,190800,15.1497
+2808,191200,15.8551
+2818,191600,6.6157
+2824,192000,15.3035
+2831,192400,11.4718
+2835,192800,22.8164
+2841,193200,16.0213
+2846,193600,17.6302
+2851,194000,14.8716
+2855,194400,20.6271
+2862,194800,14.8134
+2868,195200,10.8108
+2872,195600,22.52
+2879,196000,13.9515
+2885,196400,12.6253
+2893,196800,8.5209
+2900,197200,11.9898
+2906,197600,13.4315
+2911,198000,18.833
+2918,198400,11.5714
+2923,198800,15.1536
+2931,199200,9.8851
+2937,199600,12.3961
+2943,200000,10.9256

code/Lake application/logs/results_2/PDPPO_frozen_lake_log_4.csv

@@ -0,0 +1,501 @@
+episode,timestep,reward
+27,400,1.0785
+45,800,1.6504
+65,1200,1.4004
+75,1600,2.9704
+92,2000,1.7748
+101,2400,3.44
+112,2800,2.7486
+118,3200,4.5942
+124,3600,5.5212
+132,4000,3.7269
+139,4400,4.3077
+146,4800,4.8111
+151,5200,5.5228
+157,5600,6.1189
+163,6000,5.2543
+168,6400,5.2306
+175,6800,5.3845
+182,7200,4.3935
+187,7600,5.7774
+192,8000,6.1537
+197,8400,5.7906
+202,8800,7.0905
+209,9200,4.5122
+213,9600,6.5891
+218,10000,7.7331
+223,10400,6.6175
+227,10800,10.1737
+231,11200,8.0792
+238,11600,5.0695
+243,12000,6.8807
+247,12400,6.8492
+252,12800,7.4324
+256,13200,7.9897
+263,13600,4.5812
+270,14000,4.8277
+275,14400,6.4227
+281,14800,7.0743
+287,15200,5.8331
+292,15600,5.6509
+297,16000,7.8764
+303,16400,4.8805
+308,16800,6.1394
+312,17200,7.3848
+319,17600,5.4064
+326,18000,4.2903
+333,18400,4.4865
+338,18800,6.5078
+344,19200,5.3164
+348,19600,6.5957
+352,20000,8.4949
+357,20400,8.0414
+361,20800,8.259
+366,21200,8.7431
+371,21600,8.1425
+375,22000,7.6031
+379,22400,10.6469
+384,22800,7.4268
+389,23200,7.8948
+393,23600,9.7095
+399,24000,7.9959
+403,24400,12.6126
+407,24800,10.4783
+414,25200,8.8362
+418,25600,12.1951
+422,26000,12.6242
+426,26400,13.1493
+430,26800,11.1338
+434,27200,11.6175
+440,27600,7.7618
+445,28000,10.8874
+451,28400,9.7606
+456,28800,11.5253
+462,29200,6.7856
+467,29600,11.7964
+472,30000,8.8168
+476,30400,13.8556
+481,30800,11.2407
+485,31200,13.1211
+491,31600,10.3695
+496,32000,8.6598
+500,32400,13.3756
+506,32800,9.0459
+511,33200,10.4668
+516,33600,9.0482
+522,34000,9.8797
+526,34400,13.0444
+532,34800,7.3031
+536,35200,12.3843
+541,35600,11.6548
+545,36000,12.1808
+550,36400,8.937
+554,36800,13.32
+559,37200,10.2187
+563,37600,11.1425
+569,38000,10.4837
+573,38400,9.4084
+579,38800,9.5927
+585,39200,8.1512
+591,39600,7.4306
+598,40000,7.4929
+603,40400,12.0014
+609,40800,6.8877
+615,41200,8.7736
+622,41600,6.575
+632,42000,3.9331
+638,42400,9.7754
+644,42800,8.4288
+649,43200,9.7733
+656,43600,6.2225
+666,44000,4.1808
+672,44400,6.0964
+685,44800,3.3983
+694,45200,5.6596
+702,45600,6.6803
+708,46000,7.2517
+716,46400,5.9717
+729,46800,3.6724
+740,47200,4.6841
+744,47600,12.0424
+751,48000,8.4528
+757,48400,7.1028
+761,48800,12.6503
+770,49200,4.9012
+775,49600,9.6465
+782,50000,7.875
+787,50400,12.0186
+793,50800,9.2714
+798,51200,9.0638
+804,51600,9.1859
+808,52000,13.3962
+814,52400,9.1845
+819,52800,8.6704
+823,53200,12.922
+827,53600,13.1999
+832,54000,10.8399
+837,54400,12.6242
+842,54800,11.2661
+847,55200,10.5994
+853,55600,8.1275
+858,56000,12.1206
+866,56400,6.3821
+871,56800,8.825
+875,57200,13.9679
+880,57600,13.0994
+884,58000,14.0452
+888,58400,10.9111
+892,58800,13.7891
+897,59200,12.1729
+902,59600,12.7809
+907,60000,10.9993
+914,60400,6.609
+922,60800,6.1909
+927,61200,10.7844
+934,61600,7.0903
+945,62000,4.2601
+953,62400,5.6534
+958,62800,10.075
+963,63200,10.4977
+967,63600,11.9044
+971,64000,13.6212
+975,64400,13.086
+980,64800,12.0809
+987,65200,8.3579
+993,65600,8.2868
+997,66000,14.3792
+1001,66400,11.228
+1008,66800,8.2525
+1017,67200,6.1501
+1022,67600,9.1965
+1027,68000,11.0591
+1032,68400,11.1621
+1038,68800,10.1783
+1042,69200,10.0313
+1052,69600,5.234
+1060,70000,7.2449
+1064,70400,11.1294
+1070,70800,7.2192
+1075,71200,10.5247
+1082,71600,8.6408
+1087,72000,12.3906
+1091,72400,10.4689
+1095,72800,14.3503
+1100,73200,13.3414
+1104,73600,14.6343
+1108,74000,12.0234
+1113,74400,10.4043
+1118,74800,13.1958
+1122,75200,14.1113
+1128,75600,8.9121
+1133,76000,9.4241
+1138,76400,12.1937
+1142,76800,10.8917
+1148,77200,9.6075
+1152,77600,12.5465
+1157,78000,12.2911
+1161,78400,14.3991
+1166,78800,11.7178
+1170,79200,14.2131
+1176,79600,9.4352
+1182,80000,9.8783
+1187,80400,8.9671
+1192,80800,11.4625
+1197,81200,13.3021
+1201,81600,14.9308
+1205,82000,11.3369
+1212,82400,9.6094
+1217,82800,11.86
+1221,83200,12.7279
+1227,83600,10.7913
+1232,84000,11.0473
+1236,84400,14.2463
+1242,84800,9.8461
+1246,85200,14.7624
+1252,85600,10.4951
+1259,86000,8.9123
+1265,86400,10.1258
+1271,86800,9.0527
+1276,87200,12.5009
+1280,87600,14.9112
+1286,88000,9.7683
+1291,88400,12.2766
+1295,88800,14.6523
+1303,89200,8.0044
+1307,89600,14.4251
+1311,90000,14.7435
+1316,90400,13.5846
+1322,90800,11.1363
+1326,91200,13.1161
+1330,91600,14.8782
+1335,92000,12.9898
+1341,92400,10.5745
+1349,92800,7.7201
+1355,93200,9.7362
+1362,93600,9.0283
+1371,94000,5.595
+1376,94400,9.584
+1384,94800,7.7162
+1389,95200,9.5471
+1395,95600,11.8725
+1400,96000,11.4859
+1406,96400,10.0133
+1411,96800,10.0898
+1419,97200,7.1929
+1425,97600,10.2308
+1431,98000,10.8156
+1436,98400,11.1904
+1441,98800,13.0126
+1448,99200,9.5278
+1456,99600,7.7219
+1460,100000,11.838
+1467,100400,10.3963
+1471,100800,15.1954
+1477,101200,9.7102
+1483,101600,10.8302
+1487,102000,13.6986
+1491,102400,17.6513
+1497,102800,11.7696
+1502,103200,14.6363
+1506,103600,12.3268
+1510,104000,15.2753
+1515,104400,13.8737
+1520,104800,9.0522
+1524,105200,15.7886
+1529,105600,14.429
+1534,106000,12.0745
+1539,106400,13.0085
+1543,106800,16.5044
+1547,107200,12.8213
+1552,107600,10.883
+1557,108000,12.4299
+1561,108400,12.7274
+1565,108800,15.701
+1571,109200,11.4077
+1576,109600,13.4852
+1580,110000,15.1326
+1585,110400,12.6496
+1592,110800,11.1331
+1598,111200,10.001
+1604,111600,9.8463
+1610,112000,9.5269
+1615,112400,12.4523
+1623,112800,8.4184
+1627,113200,14.1469
+1634,113600,7.9029
+1640,114000,11.989
+1646,114400,10.0607
+1654,114800,7.935
+1658,115200,15.8146
+1662,115600,16.4529
+1666,116000,16.031
+1670,116400,16.961
+1674,116800,16.9752
+1680,117200,11.4042
+1687,117600,10.9484
+1693,118000,11.3393
+1697,118400,19.1626
+1702,118800,16.0651
+1707,119200,12.5378
+1713,119600,14.3033
+1720,120000,11.5588
+1724,120400,15.8731
+1729,120800,16.0878
+1734,121200,16.3894
+1739,121600,14.4359
+1745,122000,14.7653
+1752,122400,10.6896
+1756,122800,17.8908
+1762,123200,14.3452
+1768,123600,13.2548
+1772,124000,19.6647
+1776,124400,19.1847
+1780,124800,16.8379
+1786,125200,14.3642
+1790,125600,16.4554
+1794,126000,19.7687
+1801,126400,10.1183
+1806,126800,18.1265
+1812,127200,11.8105
+1818,127600,11.0402
+1823,128000,16.0082
+1830,128400,9.7205
+1836,128800,10.7887
+1841,129200,12.7927
+1845,129600,17.2088
+1852,130000,11.4728
+1857,130400,13.019
+1861,130800,18.7369
+1868,131200,11.284
+1873,131600,13.2389
+1878,132000,15.4627
+1882,132400,19.8252
+1889,132800,10.8438
+1894,133200,12.0874
+1901,133600,11.009
+1906,134000,13.7517
+1915,134400,8.6863
+1921,134800,10.9831
+1928,135200,11.1524
+1934,135600,11.4495
+1941,136000,9.4189
+1945,136400,18.9608
+1952,136800,9.5868
+1958,137200,10.5218
+1964,137600,13.0402
+1972,138000,7.7796
+1980,138400,7.3364
+1986,138800,13.7623
+1991,139200,16.2419
+1996,139600,11.4699
+2003,140000,10.983
+2012,140400,7.5045
+2016,140800,20.0741
+2022,141200,12.6223
+2027,141600,15.3017
+2033,142000,12.7772
+2037,142400,20.7311
+2041,142800,18.8427
+2046,143200,19.43
+2051,143600,15.0906
+2056,144000,13.0377
+2065,144400,7.2051
+2071,144800,13.2424
+2078,145200,11.0955
+2088,145600,5.3895
+2093,146000,12.318
+2100,146400,12.8436
+2105,146800,14.4873
+2109,147200,19.8853
+2117,147600,10.7438
+2123,148000,13.8923
+2128,148400,15.6226
+2132,148800,18.0464
+2137,149200,16.3704
+2144,149600,12.576
+2148,150000,20.3183
+2156,150400,9.807
+2161,150800,12.9068
+2165,151200,21.0426
+2171,151600,15.371
+2175,152000,17.7154
+2180,152400,17.2565
+2185,152800,18.8765
+2190,153200,17.2862
+2194,153600,19.4167
+2200,154000,13.2546
+2205,154400,15.271
+2211,154800,14.3329
+2215,155200,21.1205
+2221,155600,12.1206
+2227,156000,14.4072
+2232,156400,16.5247
+2236,156800,21.231
+2240,157200,20.8302
+2244,157600,19.4892
+2248,158000,20.936
+2252,158400,21.2178
+2256,158800,21.1056
+2261,159200,16.8762
+2267,159600,14.0483
+2272,160000,13.3591
+2279,160400,12.6142
+2285,160800,9.0069
+2291,161200,13.7843
+2300,161600,7.8473
+2307,162000,9.9078
+2312,162400,15.3595
+2317,162800,14.6767
+2324,163200,10.7115
+2331,163600,9.728
+2336,164000,14.8431
+2341,164400,15.2026
+2346,164800,12.322
+2351,165200,11.1342
+2356,165600,14.9371
+2361,166000,13.2367
+2365,166400,20.3463
+2371,166800,10.1723
+2377,167200,10.4504
+2381,167600,18.0134
+2388,168000,10.336
+2395,168400,11.1223
+2403,168800,6.6491
+2410,169200,9.6499
+2418,169600,7.0802
+2423,170000,10.1417
+2431,170400,6.2054
+2440,170800,7.8601
+2449,171200,7.7198
+2454,171600,14.5012
+2459,172000,15.7605
+2467,172400,7.7054
+2473,172800,12.5347
+2478,173200,15.5908
+2486,173600,5.7416
+2494,174000,9.3572
+2501,174400,9.0669
+2505,174800,11.2773
+2512,175200,6.6277
+2518,175600,4.9397
+2524,176000,6.4387
+2529,176400,9.8694
+2534,176800,8.5331
+2538,177200,8.5411
+2545,177600,6.2453
+2549,178000,11.7878
+2554,178400,10.175
+2558,178800,12.4336
+2565,179200,6.5908
+2570,179600,6.219
+2577,180000,6.0009
+2582,180400,5.8638
+2588,180800,6.9441
+2592,181200,6.8218
+2598,181600,9.0203
+2604,182000,7.8072
+2609,182400,6.4259
+2616,182800,6.3331
+2622,183200,9.3925
+2631,183600,6.0891
+2639,184000,5.3467
+2645,184400,5.1621
+2652,184800,7.0998
+2656,185200,8.041
+2663,185600,4.7481
+2670,186000,5.0002
+2677,186400,4.4118
+2682,186800,5.4232
+2689,187200,7.2963
+2693,187600,6.6073
+2699,188000,8.326
+2707,188400,5.5025
+2713,188800,5.5795
+2721,189200,3.796
+2728,189600,5.6303
+2734,190000,6.2013
+2740,190400,6.0202
+2746,190800,4.2373
+2751,191200,8.5135
+2758,191600,3.9532
+2763,192000,7.6742
+2770,192400,4.637
+2776,192800,4.4964
+2781,193200,8.2152
+2789,193600,4.0498
+2793,194000,7.944
+2798,194400,6.7629
+2802,194800,6.5532
+2807,195200,7.4252
+2811,195600,8.0144
+2815,196000,7.9792
+2819,196400,7.9971
+2824,196800,5.5573
+2828,197200,7.3163
+2833,197600,8.0791
+2838,198000,5.7746
+2843,198400,7.1306
+2848,198800,7.198
+2852,199200,8.5707
+2858,199600,5.5788
+2862,200000,8.7377

code/Lake application/logs/results_2/PDPPO_frozen_lake_log_5.csv

@@ -0,0 +1,501 @@
+episode,timestep,reward
+22,400,1.2943
+41,800,1.6015
+55,1200,1.9737
+64,1600,3.5487
+73,2000,3.146
+82,2400,3.2991
+92,2800,2.7673
+100,3200,3.8523
+107,3600,3.7979
+115,4000,4.256
+119,4400,6.4793
+124,4800,5.5414
+128,5200,7.3476
+133,5600,5.8006
+138,6000,6.1344
+144,6400,5.8326
+150,6800,4.8366
+154,7200,6.7648
+159,7600,6.5947
+163,8000,7.3957
+167,8400,7.3241
+172,8800,5.9961
+177,9200,6.0296
+182,9600,7.2685
+187,10000,6.637
+192,10400,5.999
+196,10800,6.4386
+201,11200,6.6646
+207,11600,5.4069
+212,12000,6.6252
+216,12400,6.4882
+223,12800,5.5298
+228,13200,6.2035
+233,13600,5.9536
+237,14000,6.9952
+244,14400,4.9771
+249,14800,6.1157
+253,15200,7.8858
+257,15600,7.8068
+261,16000,7.9837
+266,16400,6.6484
+270,16800,8.0041
+274,17200,6.6656
+278,17600,8.0284
+283,18000,7.3533
+290,18400,4.5723
+295,18800,6.8589
+299,19200,8.1765
+303,19600,8.1
+307,20000,8.1549
+312,20400,5.818
+318,20800,6.0381
+323,21200,5.8448
+327,21600,8.4165
+332,22000,6.3011
+336,22400,8.2937
+343,22800,5.503
+347,23200,8.5968
+353,23600,5.1292
+358,24000,7.4338
+363,24400,6.6991
+368,24800,6.5642
+373,25200,6.8225
+377,25600,8.3731
+382,26000,7.4055
+387,26400,7.1132
+393,26800,5.6948
+399,27200,5.2845
+404,27600,5.9669
+408,28000,8.3363
+415,28400,5.6672
+420,28800,5.4611
+425,29200,7.714
+429,29600,7.5512
+434,30000,7.2347
+439,30400,6.5507
+443,30800,8.4242
+449,31200,6.0604
+453,31600,8.2476
+458,32000,8.0065
+462,32400,8.2966
+467,32800,6.7805
+471,33200,9.484
+475,33600,8.7649
+481,34000,8.2474
+485,34400,9.5795
+490,34800,8.6045
+496,35200,7.4124
+501,35600,7.6157
+505,36000,10.9111
+510,36400,8.8556
+514,36800,9.9951
+519,37200,11.3025
+523,37600,11.5043
+527,38000,11.7302
+532,38400,10.3442
+536,38800,12.0948
+541,39200,10.2846
+545,39600,12.2089
+549,40000,11.8314
+554,40400,9.8942
+559,40800,10.0485
+563,41200,12.4629
+567,41600,12.2332
+571,42000,11.1847
+575,42400,12.4524
+580,42800,10.2353
+587,43200,6.303
+591,43600,12.3469
+596,44000,11.1024
+600,44400,12.4106
+604,44800,11.5728
+608,45200,10.9579
+614,45600,9.6399
+618,46000,11.3217
+622,46400,12.529
+628,46800,9.0714
+633,47200,9.9013
+638,47600,10.0751
+643,48000,10.2499
+647,48400,12.5708
+652,48800,8.7569
+656,49200,12.3604
+660,49600,12.5165
+664,50000,12.3671
+668,50400,12.618
+672,50800,12.6543
+677,51200,10.6911
+681,51600,11.8317
+685,52000,12.6955
+690,52400,10.1157
+695,52800,10.0116
+700,53200,10.2901
+705,53600,8.1824
+710,54000,9.8285
+716,54400,8.8717
+721,54800,9.9854
+725,55200,12.9736
+731,55600,8.6973
+737,56000,8.4719
+742,56400,10.3744
+747,56800,8.9466
+752,57200,10.6086
+758,57600,10.0045
+763,58000,9.8052
+768,58400,11.2697
+774,58800,7.2144
+780,59200,9.6594
+784,59600,12.3888
+789,60000,10.4215
+795,60400,8.8573
+800,60800,10.8871
+807,61200,5.8992
+812,61600,10.12
+817,62000,10.8824
+821,62400,13.1221
+826,62800,10.7535
+830,63200,13.1014
+835,63600,10.7345
+839,64000,13.082
+843,64400,13.2581
+847,64800,13.4413
+851,65200,13.5097
+855,65600,13.3641
+859,66000,13.3583
+863,66400,13.1701
+868,66800,11.1134
+872,67200,13.6245
+880,67600,5.9886
+886,68000,10.2475
+890,68400,14.6202
+894,68800,10.0685
+900,69200,11.0053
+904,69600,13.6849
+909,70000,9.6346
+913,70400,13.5021
+917,70800,13.1414
+922,71200,10.8646
+926,71600,13.4908
+932,72000,9.1396
+936,72400,11.6489
+940,72800,13.2407
+944,73200,13.0342
+948,73600,13.5329
+952,74000,13.4403
+956,74400,13.7446
+960,74800,13.1605
+964,75200,13.3433
+969,75600,10.9952
+974,76000,11.5741
+980,76400,9.3918
+985,76800,11.7496
+992,77200,8.3268
+997,77600,10.0094
+1002,78000,10.6563
+1007,78400,11.5348
+1011,78800,15.119
+1017,79200,12.0912
+1023,79600,8.9712
+1027,80000,11.6409
+1031,80400,15.629
+1036,80800,13.3766
+1040,81200,15.2536
+1045,81600,12.2289
+1049,82000,15.4839
+1055,82400,9.1355
+1060,82800,12.202
+1064,83200,14.748
+1068,83600,14.5001
+1072,84000,15.0432
+1080,84400,7.0194
+1085,84800,10.1806
+1090,85200,11.795
+1095,85600,12.7322
+1100,86000,10.2449
+1109,86400,6.4535
+1117,86800,6.893
+1123,87200,8.8297
+1129,87600,9.9537
+1134,88000,9.4762
+1138,88400,15.7652
+1144,88800,12.4334
+1153,89200,6.025
+1157,89600,14.0408
+1162,90000,12.8277
+1167,90400,12.4492
+1173,90800,8.5448
+1180,91200,9.9864
+1184,91600,15.1804
+1190,92000,8.7061
+1196,92400,11.9629
+1202,92800,9.8801
+1207,93200,10.6953
+1211,93600,16.5854
+1221,94000,6.0867
+1229,94400,9.2746
+1234,94800,12.5134
+1240,95200,8.38
+1245,95600,14.4995
+1249,96000,16.0927
+1255,96400,8.763
+1261,96800,11.1095
+1265,97200,15.5608
+1271,97600,12.1043
+1278,98000,9.2772
+1283,98400,10.4808
+1291,98800,9.4265
+1296,99200,13.4881
+1303,99600,8.7073
+1311,100000,7.7693
+1323,100400,5.2588
+1328,100800,12.0091
+1338,101200,6.3879
+1346,101600,9.2148
+1354,102000,6.5712
+1359,102400,15.4022
+1363,102800,17.8751
+1372,103200,5.6517
+1379,103600,11.7892
+1383,104000,13.7202
+1390,104400,12.0935
+1398,104800,8.3031
+1403,105200,11.4273
+1409,105600,13.6581
+1414,106000,12.8389
+1421,106400,10.5737
+1426,106800,14.0672
+1431,107200,15.1905
+1435,107600,16.499
+1441,108000,14.9027
+1447,108400,12.2743
+1453,108800,12.5351
+1458,109200,13.3324
+1464,109600,10.1987
+1470,110000,14.6681
+1480,110400,6.3981
+1484,110800,17.443
+1489,111200,11.7544
+1494,111600,12.3627
+1500,112000,14.2433
+1507,112400,9.9183
+1514,112800,10.1194
+1520,113200,11.6173
+1526,113600,12.6407
+1530,114000,14.9395
+1537,114400,10.6247
+1546,114800,8.2285
+1555,115200,7.4419
+1565,115600,6.9423
+1571,116000,11.8016
+1576,116400,13.4006
+1582,116800,12.1714
+1589,117200,9.5481
+1594,117600,17.1002
+1599,118000,13.3729
+1605,118400,14.4748
+1610,118800,15.9086
+1616,119200,13.515
+1621,119600,13.6216
+1627,120000,13.3638
+1631,120400,15.8724
+1637,120800,13.1813
+1642,121200,16.7492
+1647,121600,16.2201
+1652,122000,14.2471
+1658,122400,11.9562
+1663,122800,14.3429
+1670,123200,8.2829
+1675,123600,12.7689
+1679,124000,17.9555
+1686,124400,10.3992
+1693,124800,9.3774
+1700,125200,10.5229
+1705,125600,13.095
+1709,126000,16.827
+1716,126400,11.5151
+1720,126800,15.0954
+1725,127200,17.1223
+1729,127600,19.2025
+1734,128000,16.3688
+1740,128400,13.0476
+1744,128800,17.5761
+1751,129200,9.9195
+1758,129600,12.3492
+1763,130000,12.9946
+1768,130400,15.0853
+1774,130800,12.7493
+1780,131200,13.7049
+1784,131600,16.4027
+1791,132000,10.02
+1796,132400,13.2953
+1802,132800,13.2571
+1807,133200,16.9227
+1816,133600,7.5362
+1823,134000,9.0337
+1831,134400,8.2112
+1839,134800,9.5222
+1845,135200,10.5068
+1851,135600,15.2168
+1860,136000,7.9646
+1866,136400,9.9186
+1872,136800,11.7983
+1878,137200,15.5265
+1884,137600,11.4403
+1889,138000,14.1125
+1893,138400,18.7814
+1898,138800,15.9716
+1907,139200,6.6898
+1916,139600,8.6765
+1926,140000,6.0023
+1930,140400,18.1008
+1938,140800,8.4123
+1946,141200,8.4666
+1953,141600,9.8084
+1958,142000,12.6225
+1962,142400,16.6253
+1969,142800,11.879
+1979,143200,5.891
+1983,143600,14.6341
+1991,144000,10.6154
+1995,144400,19.3371
+2000,144800,14.6999
+2005,145200,17.3212
+2010,145600,13.511
+2019,146000,8.9176
+2025,146400,11.9234
+2029,146800,18.7508
+2035,147200,11.3652
+2042,147600,10.7875
+2047,148000,17.2345
+2052,148400,13.469
+2057,148800,15.0187
+2062,149200,18.2483
+2067,149600,15.1589
+2072,150000,13.8506
+2078,150400,13.5154
+2083,150800,17.4931
+2088,151200,13.0434
+2094,151600,12.4448
+2098,152000,16.2641
+2103,152400,18.5631
+2109,152800,13.4721
+2114,153200,17.5024
+2118,153600,17.2629
+2123,154000,16.0443
+2130,154400,12.4068
+2134,154800,19.7433
+2141,155200,12.1725
+2145,155600,17.4482
+2151,156000,14.9725
+2158,156400,8.3286
+2163,156800,15.3569
+2168,157200,16.8249
+2176,157600,10.7159
+2181,158000,13.5756
+2186,158400,16.8722
+2191,158800,16.1566
+2196,159200,16.4402
+2200,159600,19.2901
+2206,160000,13.408
+2216,160400,6.5277
+2221,160800,16.5398
+2225,161200,17.9313
+2229,161600,16.0893
+2234,162000,13.5297
+2239,162400,18.5255
+2244,162800,15.412
+2249,163200,16.7656
+2254,163600,12.2883
+2260,164000,12.8931
+2268,164400,9.994
+2273,164800,14.903
+2278,165200,14.9247
+2284,165600,13.8873
+2288,166000,15.893
+2294,166400,16.0733
+2298,166800,20.3023
+2306,167200,8.823
+2311,167600,14.7959
+2316,168000,15.5506
+2320,168400,18.9412
+2328,168800,9.9269
+2337,169200,7.8662
+2342,169600,13.8098
+2347,170000,16.0699
+2352,170400,14.2326
+2357,170800,14.3457
+2365,171200,7.755
+2372,171600,11.1969
+2377,172000,17.7461
+2382,172400,14.5072
+2387,172800,15.6355
+2392,173200,16.9889
+2397,173600,16.7953
+2402,174000,14.2381
+2407,174400,15.2855
+2414,174800,9.9614
+2422,175200,6.716
+2429,175600,9.1914
+2433,176000,15.9262
+2439,176400,13.2029
+2444,176800,12.0649
+2448,177200,11.2086
+2453,177600,17.596
+2458,178000,10.4265
+2464,178400,12.7507
+2470,178800,10.5018
+2477,179200,9.6174
+2484,179600,12.3463
+2493,180000,7.463
+2502,180400,7.0977
+2511,180800,6.566
+2516,181200,12.4002
+2522,181600,12.8125
+2526,182000,16.8705
+2532,182400,15.6621
+2537,182800,15.5005
+2545,183200,7.1256
+2549,183600,19.091
+2553,184000,20.7269
+2558,184400,17.8736
+2562,184800,16.5145
+2568,185200,12.29
+2574,185600,11.165
+2580,186000,14.862
+2585,186400,12.9498
+2590,186800,16.6379
+2597,187200,12.0999
+2601,187600,19.986
+2605,188000,21.4165
+2610,188400,12.3688
+2614,188800,17.9821
+2618,189200,20.5196
+2623,189600,19.5269
+2629,190000,11.2376
+2633,190400,20.1738
+2637,190800,20.5599
+2642,191200,15.8952
+2649,191600,12.2397
+2654,192000,16.6287
+2659,192400,14.966
+2665,192800,13.1073
+2671,193200,11.1274
+2676,193600,15.6753
+2681,194000,15.8626
+2688,194400,9.1965
+2693,194800,15.0967
+2697,195200,19.8348
+2701,195600,14.855
+2706,196000,13.0618
+2715,196400,9.1275
+2721,196800,7.1574
+2726,197200,16.5246
+2732,197600,10.026
+2739,198000,11.8092
+2745,198400,10.2368
+2751,198800,11.7544
+2757,199200,11.6034
+2764,199600,9.3743
+2769,200000,15.3278

code/Lake application/logs/results_2/PPO_frozen_lake_log_1.csv

@@ -0,0 +1,501 @@
+episode,timestep,reward
+15,400,1.9345
+32,800,1.8784
+42,1200,3.0732
+52,1600,2.9835
+65,2000,2.3838
+69,2400,7.2159
+75,2800,4.758
+80,3200,5.6732
+85,3600,7.076
+89,4000,5.77
+95,4400,5.4889
+101,4800,4.8132
+106,5200,7.4812
+115,5600,2.816
+123,6000,4.0168
+129,6400,5.4723
+137,6800,3.675
+142,7200,5.8091
+147,7600,5.6523
+151,8000,7.3976
+158,8400,4.279
+164,8800,4.7596
+168,9200,7.4893
+173,9600,6.204
+177,10000,7.3342
+183,10400,4.9965
+188,10800,6.2773
+193,11200,6.1197
+197,11600,7.2967
+201,12000,7.4756
+205,12400,7.1791
+209,12800,7.3157
+213,13200,7.0592
+218,13600,5.8954
+222,14000,7.1992
+227,14400,6.5493
+232,14800,5.9315
+237,15200,7.0655
+243,15600,5.2935
+251,16000,3.6833
+258,16400,4.511
+267,16800,4.1031
+271,17200,7.4993
+275,17600,9.0943
+279,18000,7.6642
+285,18400,6.6167
+291,18800,5.4259
+295,19200,7.9226
+299,19600,8.2753
+304,20000,6.8015
+308,20400,7.8775
+315,20800,4.5316
+321,21200,5.4028
+326,21600,6.3314
+330,22000,8.1528
+334,22400,7.4973
+339,22800,6.6922
+343,23200,7.7072
+347,23600,7.2187
+351,24000,7.7859
+356,24400,7.0575
+360,24800,7.2281
+366,25200,5.8797
+372,25600,4.8101
+378,26000,5.0378
+384,26400,5.6277
+389,26800,5.2954
+400,27200,2.9769
+406,27600,4.6418
+416,28000,3.4303
+423,28400,4.0932
+430,28800,4.505
+435,29200,5.5546
+439,29600,10.5906
+445,30000,6.1133
+451,30400,5.9823
+455,30800,8.4017
+462,31200,5.0732
+468,31600,5.7932
+474,32000,6.708
+479,32400,6.8853
+485,32800,9.0958
+490,33200,6.4707
+495,33600,7.4192
+501,34000,6.0651
+506,34400,6.7035
+511,34800,6.8316
+518,35200,5.9218
+524,35600,7.2932
+528,36000,8.8676
+534,36400,8.8016
+538,36800,8.7465
+544,37200,8.41
+549,37600,8.3048
+555,38000,8.0676
+561,38400,6.0523
+566,38800,8.8973
+571,39200,8.7871
+577,39600,6.6108
+582,40000,10.0771
+587,40400,7.6992
+594,40800,5.8859
+602,41200,5.2757
+607,41600,7.4604
+614,42000,6.3297
+624,42400,4.3
+630,42800,7.3462
+636,43200,7.7832
+642,43600,7.2278
+649,44000,7.0903
+656,44400,7.2864
+660,44800,11.4337
+664,45200,9.4219
+669,45600,10.1337
+674,46000,10.8808
+679,46400,9.5832
+686,46800,6.8856
+692,47200,7.9853
+699,47600,5.4185
+705,48000,8.51
+711,48400,8.7795
+717,48800,8.1386
+721,49200,12.1496
+727,49600,9.671
+734,50000,4.7004
+742,50400,5.4595
+751,50800,7.1566
+756,51200,11.0405
+763,51600,7.8217
+769,52000,6.9623
+774,52400,9.7564
+780,52800,8.2737
+786,53200,7.7905
+792,53600,8.2016
+798,54000,11.1113
+803,54400,9.71
+811,54800,6.3473
+816,55200,10.3967
+822,55600,8.5624
+829,56000,7.9157
+836,56400,7.3681
+843,56800,7.9999
+848,57200,12.6498
+855,57600,6.7851
+864,58000,5.926
+871,58400,8.3211
+877,58800,8.4122
+885,59200,5.5055
+892,59600,4.8152
+896,60000,10.2298
+903,60400,8.8526
+908,60800,10.4385
+914,61200,6.8265
+920,61600,7.4231
+926,62000,7.7046
+932,62400,8.744
+939,62800,6.815
+944,63200,8.9794
+950,63600,9.0955
+960,64000,3.9011
+966,64400,9.1169
+970,64800,9.8831
+975,65200,10.7317
+983,65600,5.231
+988,66000,9.8312
+993,66400,8.1545
+999,66800,8.3283
+1008,67200,3.3866
+1014,67600,6.2135
+1018,68000,8.4834
+1025,68400,4.2607
+1031,68800,7.755
+1035,69200,6.7764
+1043,69600,4.8484
+1049,70000,6.8767
+1055,70400,7.2876
+1061,70800,6.5337
+1066,71200,8.7095
+1071,71600,8.5315
+1076,72000,8.1106
+1082,72400,6.5303
+1088,72800,5.1858
+1092,73200,8.2166
+1097,73600,7.7642
+1103,74000,5.7689
+1109,74400,6.5443
+1113,74800,6.0629
+1119,75200,6.9554
+1124,75600,6.5107
+1131,76000,5.8779
+1136,76400,7.0196
+1141,76800,8.221
+1146,77200,7.0804
+1150,77600,7.0824
+1158,78000,4.9257
+1165,78400,4.3861
+1171,78800,5.412
+1178,79200,5.8524
+1186,79600,4.6331
+1193,80000,5.5782
+1199,80400,7.2403
+1204,80800,8.5925
+1208,81200,9.0404
+1212,81600,10.5095
+1220,82000,7.0118
+1225,82400,8.1882
+1229,82800,10.863
+1233,83200,10.7497
+1239,83600,6.4536
+1243,84000,11.2907
+1247,84400,11.783
+1252,84800,11.0893
+1257,85200,9.8707
+1261,85600,13.0076
+1265,86000,12.8858
+1269,86400,13.1408
+1273,86800,11.8421
+1277,87200,9.9549
+1284,87600,6.7063
+1288,88000,12.4549
+1292,88400,13.2602
+1299,88800,7.2956
+1303,89200,15.6022
+1308,89600,12.0928
+1317,90000,3.9969
+1323,90400,6.6326
+1331,90800,6.7157
+1337,91200,6.7272
+1346,91600,4.2204
+1351,92000,10.5966
+1357,92400,6.7511
+1364,92800,6.8195
+1371,93200,6.92
+1381,93600,4.6135
+1387,94000,9.0048
+1395,94400,4.8047
+1401,94800,9.5174
+1407,95200,11.6457
+1414,95600,6.9109
+1421,96000,5.7336
+1430,96400,5.3074
+1434,96800,9.2165
+1439,97200,9.5083
+1447,97600,5.2046
+1456,98000,4.3764
+1462,98400,5.923
+1467,98800,8.1644
+1473,99200,6.7967
+1480,99600,7.7647
+1484,100000,12.2281
+1488,100400,9.1569
+1494,100800,8.54
+1499,101200,9.4563
+1503,101600,9.5965
+1509,102000,8.6554
+1514,102400,11.0492
+1518,102800,10.4916
+1525,103200,5.7317
+1534,103600,5.4161
+1539,104000,9.7729
+1546,104400,6.9356
+1552,104800,7.0423
+1556,105200,12.5092
+1563,105600,8.7948
+1567,106000,12.3567
+1573,106400,7.8177
+1577,106800,12.8039
+1584,107200,6.8714
+1589,107600,8.3869
+1593,108000,12.6683
+1601,108400,6.5902
+1607,108800,8.628
+1615,109200,5.4762
+1621,109600,9.3018
+1627,110000,7.4751
+1640,110400,4.139
+1647,110800,7.7327
+1652,111200,10.3567
+1662,111600,4.9881
+1668,112000,8.8983
+1673,112400,9.1326
+1683,112800,3.9693
+1693,113200,4.1639
+1699,113600,8.137
+1705,114000,8.6208
+1710,114400,9.1387
+1717,114800,6.4969
+1724,115200,5.8328
+1729,115600,9.6428
+1736,116000,7.7885
+1742,116400,6.1858
+1751,116800,5.3759
+1760,117200,4.9088
+1764,117600,12.8121
+1773,118000,5.528
+1781,118400,4.8728
+1786,118800,9.4107
+1794,119200,5.6412
+1800,119600,8.3779
+1804,120000,14.4553
+1814,120400,4.32
+1819,120800,12.7509
+1828,121200,6.4023
+1833,121600,15.743
+1841,122000,6.9661
+1847,122400,11.3894
+1852,122800,12.2086
+1857,123200,16.0304
+1862,123600,16.1933
+1870,124000,8.6231
+1875,124400,13.6078
+1880,124800,14.3245
+1889,125200,5.8726
+1896,125600,9.655
+1904,126000,6.397
+1912,126400,5.4703
+1917,126800,13.7367
+1923,127200,10.1322
+1928,127600,12.9031
+1932,128000,11.6047
+1941,128400,8.1528
+1948,128800,10.0696
+1953,129200,12.8557
+1964,129600,6.1629
+1974,130000,4.7264
+1983,130400,5.0263
+1989,130800,13.0645
+2000,131200,4.9296
+2006,131600,9.139
+2016,132000,6.4189
+2025,132400,4.758
+2033,132800,7.2135
+2038,133200,8.544
+2045,133600,9.7969
+2051,134000,7.3074
+2063,134400,4.5675
+2072,134800,4.937
+2080,135200,6.392
+2088,135600,4.6451
+2093,136000,9.6252
+2100,136400,5.984
+2106,136800,7.3098
+2114,137200,6.8205
+2122,137600,5.1513
+2130,138000,6.8115
+2140,138400,4.8287
+2149,138800,3.6011
+2157,139200,5.5075
+2163,139600,9.1481
+2171,140000,6.788
+2178,140400,7.9736
+2186,140800,8.5882
+2191,141200,9.2546
+2196,141600,15.4158
+2206,142000,7.1935
+2211,142400,13.2408
+2223,142800,4.9066
+2235,143200,4.5763
+2241,143600,10.3808
+2246,144000,10.8564
+2254,144400,7.7899
+2261,144800,8.8404
+2268,145200,8.8255
+2276,145600,7.4552
+2282,146000,11.216
+2289,146400,10.299
+2295,146800,9.2089
+2302,147200,8.5813
+2311,147600,5.5125
+2318,148000,7.7001
+2323,148400,11.1793
+2333,148800,3.7442
+2340,149200,10.413
+2345,149600,11.7132
+2350,150000,9.5357
+2357,150400,8.7914
+2363,150800,8.3319
+2370,151200,7.1484
+2376,151600,5.6768
+2385,152000,4.1424
+2390,152400,6.528
+2394,152800,7.7268
+2399,153200,8.4871
+2406,153600,6.8809
+2411,154000,8.3506
+2415,154400,11.0307
+2419,154800,10.397
+2427,155200,5.1522
+2434,155600,5.8036
+2439,156000,10.0986
+2445,156400,7.7494
+2452,156800,8.7844
+2458,157200,9.1953
+2464,157600,9.7125
+2469,158000,14.3923
+2475,158400,10.5696
+2479,158800,19.5742
+2488,159200,6.9904
+2494,159600,12.7756
+2499,160000,12.9402
+2506,160400,11.8509
+2510,160800,14.1441
+2522,161200,5.8916
+2530,161600,8.0032
+2536,162000,11.8227
+2540,162400,14.6518
+2548,162800,9.0002
+2555,163200,5.4754
+2560,163600,15.7747
+2568,164000,9.1886
+2572,164400,18.4005
+2580,164800,6.8589
+2586,165200,13.1998
+2591,165600,12.0732
+2599,166000,7.623
+2608,166400,6.2975
+2614,166800,10.325
+2622,167200,7.3189
+2628,167600,11.0537
+2636,168000,8.6657
+2642,168400,13.2533
+2650,168800,5.9973
+2660,169200,7.9659
+2665,169600,14.8961
+2673,170000,9.1592
+2680,170400,12.4329
+2687,170800,12.6719
+2694,171200,11.0606
+2702,171600,10.5086
+2709,172000,10.1179
+2716,172400,9.1879
+2723,172800,11.2291
+2728,173200,12.3325
+2734,173600,12.1718
+2741,174000,10.1937
+2748,174400,9.3497
+2753,174800,10.5969
+2761,175200,10.1798
+2767,175600,10.9461
+2772,176000,13.9387
+2777,176400,14.2562
+2783,176800,8.5326
+2789,177200,9.8383
+2794,177600,12.2766
+2800,178000,14.2094
+2804,178400,15.156
+2811,178800,6.4663
+2816,179200,15.0295
+2823,179600,10.1727
+2828,180000,11.4667
+2834,180400,11.0373
+2840,180800,11.0221
+2844,181200,18.4976
+2850,181600,9.7416
+2857,182000,8.77
+2863,182400,7.9617
+2867,182800,21.1088
+2872,183200,15.0862
+2880,183600,9.2334
+2885,184000,15.0575
+2891,184400,8.7998
+2898,184800,9.4749
+2903,185200,15.2583
+2908,185600,15.9073
+2913,186000,18.1103
+2920,186400,10.0962
+2925,186800,14.1606
+2931,187200,13.2483
+2936,187600,9.6115
+2942,188000,10.5395
+2948,188400,13.0603
+2952,188800,15.9296
+2958,189200,11.0247
+2964,189600,13.2235
+2973,190000,6.3575
+2978,190400,11.5839
+2983,190800,13.9557
+2989,191200,9.2667
+2995,191600,10.0113
+3000,192000,10.4833
+3006,192400,11.3706
+3012,192800,10.4763
+3018,193200,10.3391
+3022,193600,16.8413
+3030,194000,9.8189
+3042,194400,4.0255
+3049,194800,8.3469
+3058,195200,6.7786
+3066,195600,9.455
+3072,196000,9.361
+3078,196400,11.942
+3086,196800,8.1976
+3093,197200,8.8319
+3098,197600,10.2572
+3105,198000,10.4746
+3111,198400,9.7314
+3117,198800,13.439
+3121,199200,15.2623
+3127,199600,15.3384
+3137,200000,8.1123

code/Lake application/logs/results_2/PPO_frozen_lake_log_2.csv

@@ -0,0 +1,501 @@
+episode,timestep,reward
+19,400,1.6097
+38,800,1.6416
+61,1200,1.4003
+73,1600,2.5901
+81,2000,3.7322
+90,2400,3.1764
+95,2800,6.9871
+102,3200,4.2802
+109,3600,4.7436
+116,4000,4.2818
+120,4400,7.5307
+124,4800,6.3049
+132,5200,4.3085
+137,5600,6.2239
+141,6000,7.0954
+145,6400,6.7039
+150,6800,6.2469
+155,7200,6.36
+160,7600,6.1549
+164,8000,6.9545
+169,8400,7.0148
+175,8800,4.7251
+179,9200,7.9361
+184,9600,6.5351
+189,10000,6.2774
+195,10400,5.7017
+199,10800,7.825
+203,11200,6.8724
+207,11600,7.7336
+211,12000,7.5364
+215,12400,7.8172
+219,12800,7.9367
+223,13200,7.7311
+228,13600,7.2277
+232,14000,7.0049
+238,14400,5.1928
+242,14800,7.6765
+246,15200,7.4842
+250,15600,7.5239
+254,16000,7.1122
+260,16400,5.5361
+265,16800,7.1787
+272,17200,4.4245
+276,17600,6.2406
+280,18000,7.7256
+286,18400,5.6842
+291,18800,6.646
+297,19200,5.5573
+302,19600,4.9828
+308,20000,5.9812
+313,20400,6.4346
+320,20800,4.4117
+324,21200,6.6082
+330,21600,6.2485
+334,22000,7.8137
+339,22400,5.5175
+343,22800,7.8811
+347,23200,7.1971
+353,23600,5.196
+358,24000,6.0231
+364,24400,6.6306
+369,24800,5.7511
+374,25200,6.1309
+378,25600,7.6626
+382,26000,6.8129
+387,26400,6.4101
+391,26800,8.1621
+395,27200,7.6024
+399,27600,7.6139
+404,28000,7.434
+409,28400,7.5108
+413,28800,10.0048
+417,29200,8.9736
+423,29600,5.5721
+428,30000,5.6607
+433,30400,7.3198
+437,30800,7.5887
+441,31200,7.1714
+446,31600,5.5443
+452,32000,6.0095
+456,32400,6.5919
+461,32800,6.9332
+465,33200,5.9742
+472,33600,4.6549
+476,34000,6.7065
+481,34400,6.6351
+485,34800,7.3781
+491,35200,5.3901
+495,35600,7.481
+499,36000,7.3904
+504,36400,5.5203
+508,36800,6.6256
+513,37200,6.2715
+519,37600,5.4467
+523,38000,7.5914
+528,38400,5.8271
+535,38800,4.7497
+539,39200,6.705
+543,39600,7.2193
+547,40000,7.6591
+553,40400,5.4713
+559,40800,5.6278
+563,41200,7.0377
+567,41600,6.5975
+572,42000,6.4589
+577,42400,6.4962
+581,42800,7.5891
+585,43200,7.5598
+593,43600,3.9839
+597,44000,6.7188
+601,44400,6.9495
+606,44800,7.4883
+613,45200,3.7567
+617,45600,7.7218
+622,46000,6.69
+626,46400,7.9262
+630,46800,6.3034
+635,47200,6.0545
+639,47600,7.4195
+644,48000,7.13
+648,48400,7.0398
+652,48800,7.6392
+660,49200,3.7038
+667,49600,4.3578
+672,50000,5.9194
+678,50400,4.8665
+683,50800,6.3008
+691,51200,4.5975
+697,51600,5.1888
+702,52000,5.7734
+707,52400,7.4679
+715,52800,3.3284
+721,53200,6.0641
+727,53600,6.0441
+731,54000,7.6702
+736,54400,7.4219
+740,54800,6.222
+746,55200,6.4839
+751,55600,5.6146
+755,56000,7.5972
+761,56400,5.612
+765,56800,7.8148
+770,57200,5.7253
+774,57600,7.9334
+778,58000,8.3276
+783,58400,6.8913
+787,58800,7.2828
+793,59200,6.9596
+797,59600,8.2329
+803,60000,6.4227
+808,60400,6.7123
+812,60800,7.9976
+819,61200,5.5859
+826,61600,4.4553
+832,62000,6.7451
+837,62400,6.241
+843,62800,6.4673
+848,63200,6.9543
+855,63600,5.0913
+861,64000,7.2159
+865,64400,7.9432
+870,64800,7.8044
+875,65200,7.2243
+880,65600,7.7488
+886,66000,6.3843
+890,66400,9.9507
+894,66800,9.063
+899,67200,7.5112
+903,67600,7.6428
+908,68000,8.2787
+912,68400,8.5889
+918,68800,5.723
+922,69200,8.2975
+926,69600,7.3824
+931,70000,8.2068
+937,70400,7.7218
+941,70800,8.7229
+945,71200,10.2961
+950,71600,7.6121
+955,72000,7.8974
+960,72400,7.8978
+967,72800,5.6801
+979,73200,2.1174
+990,73600,3.7935
+995,74000,10.1194
+1000,74400,6.8977
+1006,74800,7.6098
+1011,75200,7.2811
+1017,75600,7.2474
+1023,76000,8.5673
+1029,76400,6.6197
+1035,76800,8.3262
+1041,77200,7.9771
+1047,77600,7.1542
+1051,78000,11.2335
+1056,78400,8.8322
+1062,78800,7.5779
+1066,79200,13.4042
+1071,79600,10.4175
+1076,80000,13.3333
+1080,80400,12.734
+1086,80800,11.1026
+1090,81200,14.7759
+1096,81600,8.6412
+1104,82000,6.3892
+1110,82400,9.8357
+1118,82800,7.1741
+1125,83200,7.8912
+1131,83600,8.455
+1135,84000,14.1645
+1140,84400,9.4645
+1146,84800,10.1664
+1153,85200,5.8022
+1160,85600,6.215
+1168,86000,4.1062
+1175,86400,6.1067
+1180,86800,6.3874
+1187,87200,6.3452
+1192,87600,6.9666
+1198,88000,5.1382
+1203,88400,6.1001
+1212,88800,4.8099
+1218,89200,7.4769
+1226,89600,7.5724
+1231,90000,10.0412
+1238,90400,8.6996
+1242,90800,13.2195
+1247,91200,11.4526
+1252,91600,12.0908
+1260,92000,6.7688
+1265,92400,10.449
+1270,92800,10.2558
+1277,93200,5.5992
+1284,93600,9.3672
+1292,94000,7.979
+1302,94400,6.4251
+1308,94800,8.6478
+1314,95200,12.1168
+1322,95600,6.8864
+1333,96000,5.5147
+1339,96400,9.0327
+1347,96800,6.0256
+1356,97200,4.7894
+1365,97600,5.5282
+1371,98000,7.7578
+1380,98400,8.3785
+1389,98800,5.239
+1395,99200,8.3681
+1400,99600,9.7949
+1406,100000,9.5216
+1414,100400,5.3299
+1420,100800,7.8871
+1428,101200,6.0441
+1434,101600,6.6909
+1444,102000,5.0341
+1450,102400,7.7649
+1461,102800,4.1021
+1472,103200,2.9832
+1482,103600,4.4576
+1489,104000,7.3442
+1497,104400,5.6498
+1505,104800,6.4131
+1511,105200,10.5803
+1516,105600,11.3073
+1520,106000,13.2675
+1527,106400,10.6688
+1535,106800,5.8265
+1540,107200,10.7056
+1545,107600,7.6742
+1553,108000,5.7264
+1560,108400,6.6821
+1567,108800,5.4216
+1573,109200,8.4766
+1578,109600,8.6739
+1582,110000,11.4666
+1586,110400,9.0832
+1594,110800,6.2276
+1599,111200,8.6866
+1606,111600,6.2615
+1614,112000,5.6982
+1621,112400,5.8051
+1626,112800,7.93
+1631,113200,10.595
+1636,113600,7.6407
+1640,114000,11.1847
+1647,114400,5.6078
+1651,114800,9.1446
+1658,115200,6.8432
+1662,115600,12.9911
+1671,116000,4.613
+1676,116400,11.0305
+1681,116800,8.694
+1688,117200,6.11
+1698,117600,6.4954
+1703,118000,9.7062
+1712,118400,5.8668
+1717,118800,7.5547
+1724,119200,8.3224
+1729,119600,8.329
+1737,120000,7.1094
+1742,120400,9.2663
+1747,120800,9.6127
+1757,121200,5.6769
+1762,121600,8.5658
+1770,122000,7.4468
+1776,122400,8.0238
+1782,122800,8.0927
+1790,123200,5.6296
+1798,123600,6.9246
+1806,124000,6.5561
+1813,124400,7.4058
+1821,124800,6.9491
+1833,125200,4.4584
+1839,125600,8.2732
+1848,126000,6.3892
+1853,126400,10.4589
+1858,126800,12.1972
+1867,127200,5.6719
+1874,127600,7.9428
+1880,128000,7.5333
+1885,128400,12.1847
+1889,128800,11.591
+1895,129200,9.8666
+1902,129600,9.6165
+1910,130000,8.469
+1917,130400,5.6391
+1924,130800,8.5008
+1931,131200,9.7077
+1935,131600,13.5229
+1939,132000,16.4664
+1944,132400,13.0046
+1949,132800,8.4371
+1955,133200,8.8647
+1959,133600,14.3521
+1963,134000,11.7871
+1968,134400,10.4688
+1974,134800,9.3431
+1979,135200,8.226
+1984,135600,10.8513
+1988,136000,10.6682
+1998,136400,4.4273
+2003,136800,12.778
+2011,137200,8.0067
+2017,137600,9.4886
+2027,138000,5.6532
+2033,138400,7.9827
+2039,138800,5.9282
+2045,139200,9.6567
+2053,139600,7.7935
+2059,140000,5.9489
+2064,140400,6.9939
+2070,140800,7.3466
+2075,141200,6.8939
+2079,141600,9.1796
+2085,142000,9.0719
+2089,142400,13.9325
+2094,142800,11.0623
+2099,143200,6.7284
+2107,143600,5.6728
+2115,144000,6.321
+2119,144400,10.3033
+2124,144800,8.3466
+2130,145200,6.2657
+2136,145600,5.7428
+2142,146000,7.4074
+2147,146400,7.8209
+2153,146800,7.3415
+2161,147200,5.2917
+2166,147600,12.5109
+2174,148000,5.0395
+2179,148400,10.3555
+2187,148800,5.349
+2192,149200,10.0253
+2198,149600,7.0379
+2205,150000,7.5125
+2211,150400,7.8038
+2217,150800,7.8167
+2221,151200,11.7039
+2229,151600,5.489
+2236,152000,6.0868
+2243,152400,6.2941
+2249,152800,6.6384
+2254,153200,8.3705
+2259,153600,8.5642
+2265,154000,6.8584
+2272,154400,7.3834
+2278,154800,8.2766
+2286,155200,5.1656
+2290,155600,12.9118
+2294,156000,11.8071
+2299,156400,8.9169
+2303,156800,8.9791
+2308,157200,9.3741
+2312,157600,11.1361
+2316,158000,11.7926
+2320,158400,11.7151
+2326,158800,8.0207
+2333,159200,6.0691
+2338,159600,9.577
+2344,160000,6.4232
+2350,160400,8.8049
+2354,160800,9.1235
+2359,161200,8.3861
+2365,161600,7.0742
+2369,162000,7.6221
+2373,162400,7.9897
+2378,162800,5.8857
+2384,163200,7.0723
+2389,163600,6.654
+2395,164000,7.1041
+2399,164400,8.4728
+2407,164800,4.6772
+2411,165200,7.5167
+2417,165600,7.8597
+2422,166000,9.3692
+2427,166400,8.0704
+2431,166800,10.7773
+2435,167200,9.905
+2440,167600,8.5513
+2444,168000,8.9629
+2450,168400,7.6352
+2457,168800,6.9678
+2463,169200,7.899
+2469,169600,7.6206
+2474,170000,8.0358
+2481,170400,7.7953
+2486,170800,10.2047
+2492,171200,12.0048
+2500,171600,5.3765
+2509,172000,5.1863
+2513,172400,12.2984
+2517,172800,11.7797
+2524,173200,7.875
+2534,173600,3.9569
+2539,174000,11.8781
+2544,174400,10.3608
+2550,174800,8.7275
+2556,175200,8.4987
+2561,175600,10.7181
+2566,176000,8.2619
+2571,176400,8.8454
+2575,176800,8.7559
+2580,177200,8.4039
+2584,177600,11.9876
+2592,178000,6.4634
+2597,178400,7.8467
+2602,178800,6.8881
+2607,179200,7.1522
+2612,179600,7.6053
+2620,180000,5.2739
+2628,180400,4.7562
+2636,180800,3.7301
+2641,181200,10.6831
+2645,181600,8.9943
+2652,182000,5.277
+2661,182400,4.275
+2671,182800,3.453
+2676,183200,8.5581
+2681,183600,7.2929
+2686,184000,12.0382
+2690,184400,12.133
+2694,184800,12.2167
+2701,185200,5.3062
+2706,185600,10.3605
+2714,186000,5.9351
+2718,186400,9.3307
+2723,186800,9.5625
+2730,187200,7.1449
+2735,187600,7.7182
+2741,188000,6.4756
+2745,188400,8.736
+2751,188800,8.123
+2757,189200,8.8964
+2761,189600,10.7888
+2765,190000,12.0779
+2769,190400,10.5991
+2775,190800,8.0868
+2780,191200,8.917
+2785,191600,9.9389
+2789,192000,11.5622
+2795,192400,8.386
+2800,192800,8.3721
+2805,193200,8.8166
+2811,193600,8.1763
+2817,194000,8.3255
+2823,194400,8.8563
+2827,194800,12.6773
+2833,195200,7.9132
+2837,195600,10.4681
+2844,196000,8.2131
+2849,196400,9.2795
+2855,196800,7.094
+2860,197200,10.5194
+2864,197600,12.9269
+2870,198000,7.9327
+2876,198400,7.223
+2881,198800,10.8269
+2885,199200,12.3889
+2892,199600,6.5769
+2897,200000,11.076

code/Lake application/logs/results_2/PPO_frozen_lake_log_3.csv

@@ -0,0 +1,501 @@
+episode,timestep,reward
+15,400,1.8261
+32,800,1.7685
+45,1200,2.4142
+51,1600,4.6704
+61,2000,3.1899
+67,2400,5.3735
+75,2800,4.1154
+82,3200,3.7725
+92,3600,3.7704
+103,4000,2.6905
+110,4400,4.0276
+117,4800,5.8064
+124,5200,4.4713
+132,5600,4.0093
+137,6000,5.9495
+142,6400,5.8014
+148,6800,5.8551
+155,7200,4.6169
+162,7600,4.2494
+170,8000,3.9722
+177,8400,3.9192
+186,8800,3.8894
+192,9200,5.2858
+196,9600,6.1068
+201,10000,7.4978
+206,10400,6.5813
+210,10800,7.0333
+216,11200,5.7853
+220,11600,6.9548
+226,12000,6.9295
+231,12400,8.6606
+235,12800,8.1602
+241,13200,5.5479
+248,13600,5.2991
+253,14000,5.5222
+259,14400,6.6097
+266,14800,5.7681
+272,15200,5.1424
+281,15600,3.9703
+289,16000,5.446
+296,16400,5.6469
+301,16800,8.1023
+309,17200,5.4118
+313,17600,10.0884
+318,18000,7.85
+324,18400,7.1867
+334,18800,4.1973
+341,19200,6.5775
+346,19600,8.8721
+353,20000,8.3356
+358,20400,8.4092
+364,20800,7.0598
+369,21200,9.7722
+375,21600,8.7407
+381,22000,7.1701
+387,22400,6.8198
+391,22800,11.2731
+399,23200,6.9213
+404,23600,7.6675
+408,24000,8.5927
+412,24400,9.5454
+416,24800,11.0606
+421,25200,11.0094
+426,25600,9.2321
+431,26000,7.2778
+436,26400,10.7602
+440,26800,10.2041
+444,27200,10.7586
+448,27600,8.5868
+452,28000,9.4227
+458,28400,8.901
+462,28800,9.059
+467,29200,9.2612
+472,29600,10.802
+478,30000,7.4041
+483,30400,8.8905
+489,30800,6.3011
+493,31200,12.0364
+497,31600,11.5981
+501,32000,11.5024
+505,32400,11.9104
+509,32800,11.9212
+513,33200,12.0185
+518,33600,9.4035
+522,34000,11.5176
+526,34400,11.0239
+534,34800,6.1402
+538,35200,9.7348
+546,35600,6.35
+551,36000,7.8883
+558,36400,7.7455
+563,36800,7.1019
+571,37200,6.6505
+576,37600,7.5348
+581,38000,12.1861
+586,38400,9.1601
+591,38800,8.1292
+596,39200,7.3226
+602,39600,9.3
+608,40000,10.1455
+614,40400,7.404
+620,40800,9.5543
+627,41200,8.0328
+636,41600,5.051
+648,42000,4.3144
+654,42400,8.7103
+661,42800,8.5619
+666,43200,9.0912
+671,43600,12.1562
+679,44000,6.8929
+683,44400,12.4673
+690,44800,7.4547
+700,45200,6.1627
+708,45600,5.2344
+712,46000,14.522
+718,46400,9.7264
+724,46800,9.4083
+731,47200,7.3673
+735,47600,10.918
+741,48000,9.7135
+746,48400,11.6226
+753,48800,6.5335
+760,49200,6.1922
+765,49600,11.59
+772,50000,7.6406
+779,50400,7.3931
+785,50800,8.8649
+790,51200,13.0236
+796,51600,9.1355
+802,52000,9.2798
+812,52400,4.6073
+818,52800,8.5625
+823,53200,8.0732
+829,53600,8.2494
+837,54000,5.0721
+849,54400,3.926
+857,54800,5.9843
+866,55200,5.4496
+872,55600,9.6436
+877,56000,9.8259
+882,56400,12.0831
+886,56800,11.8707
+892,57200,9.3723
+897,57600,8.75
+902,58000,9.1673
+908,58400,8.2213
+919,58800,3.6353
+929,59200,3.9628
+935,59600,6.8984
+942,60000,6.928
+948,60400,8.007
+954,60800,7.1696
+962,61200,6.8068
+970,61600,5.7813
+979,62000,6.7075
+990,62400,4.5979
+995,62800,11.0131
+1001,63200,9.9881
+1007,63600,8.771
+1013,64000,6.4708
+1020,64400,8.4602
+1024,64800,12.5658
+1029,65200,12.6734
+1033,65600,13.9195
+1037,66000,10.7454
+1043,66400,10.9443
+1048,66800,10.9429
+1053,67200,9.4126
+1059,67600,8.414
+1066,68000,6.8977
+1071,68400,8.4342
+1077,68800,6.9781
+1081,69200,9.3134
+1087,69600,8.1705
+1091,70000,8.8618
+1096,70400,10.7669
+1100,70800,10.662
+1104,71200,9.404
+1108,71600,10.9212
+1114,72000,7.7906
+1120,72400,6.9977
+1124,72800,10.0251
+1130,73200,8.4128
+1134,73600,9.9691
+1138,74000,11.4181
+1143,74400,8.6228
+1152,74800,5.3614
+1157,75200,8.1655
+1164,75600,7.1774
+1171,76000,5.9159
+1180,76400,4.0023
+1189,76800,4.7476
+1197,77200,5.5766
+1202,77600,8.0878
+1209,78000,6.6897
+1213,78400,13.7633
+1221,78800,6.482
+1230,79200,6.0141
+1234,79600,12.2026
+1242,80000,5.6537
+1251,80400,4.3695
+1259,80800,7.4921
+1264,81200,9.8077
+1269,81600,10.9606
+1275,82000,9.6273
+1280,82400,12.195
+1287,82800,7.4125
+1292,83200,12.1273
+1296,83600,13.0822
+1303,84000,7.0237
+1308,84400,11.6651
+1313,84800,9.4606
+1318,85200,12.5532
+1324,85600,9.9701
+1329,86000,11.7337
+1333,86400,14.7119
+1339,86800,9.0294
+1345,87200,7.5184
+1353,87600,7.3597
+1359,88000,10.1495
+1365,88400,8.7107
+1375,88800,4.0411
+1383,89200,5.6476
+1389,89600,6.5996
+1397,90000,8.0321
+1404,90400,8.9973
+1410,90800,8.9065
+1417,91200,5.712
+1427,91600,4.6027
+1436,92000,5.296
+1441,92400,7.6209
+1449,92800,7.4688
+1456,93200,5.0893
+1464,93600,6.9208
+1472,94000,7.4646
+1479,94400,8.2095
+1484,94800,9.7461
+1490,95200,8.781
+1494,95600,8.3202
+1499,96000,9.0412
+1506,96400,6.9789
+1511,96800,7.9199
+1517,97200,10.6694
+1524,97600,6.481
+1532,98000,6.7254
+1541,98400,6.2726
+1547,98800,7.5851
+1555,99200,5.3696
+1560,99600,10.4877
+1567,100000,8.8012
+1577,100400,5.5168
+1584,100800,10.155
+1588,101200,8.5114
+1595,101600,6.3359
+1602,102000,6.6452
+1608,102400,6.8052
+1612,102800,8.0109
+1617,103200,5.8693
+1621,103600,8.5857
+1626,104000,9.6799
+1634,104400,5.6426
+1639,104800,6.5151
+1644,105200,7.8849
+1648,105600,8.9338
+1654,106000,5.6678
+1659,106400,6.3244
+1665,106800,5.6218
+1669,107200,9.5578
+1676,107600,5.3338
+1686,108000,4.6969
+1693,108400,5.4701
+1698,108800,10.4455
+1704,109200,8.6891
+1710,109600,10.0138
+1717,110000,7.5448
+1726,110400,7.8171
+1731,110800,10.4446
+1739,111200,8.1385
+1748,111600,6.0488
+1756,112000,7.1346
+1761,112400,9.6157
+1767,112800,9.1395
+1774,113200,8.8882
+1782,113600,8.0369
+1789,114000,9.7018
+1795,114400,8.5519
+1802,114800,12.4735
+1810,115200,6.0752
+1815,115600,11.7469
+1824,116000,5.4505
+1829,116400,9.3351
+1835,116800,10.7987
+1840,117200,15.1342
+1846,117600,14.5398
+1853,118000,10.6334
+1861,118400,11.3101
+1866,118800,14.8907
+1873,119200,9.6076
+1882,119600,7.7126
+1893,120000,4.7907
+1901,120400,6.0066
+1906,120800,15.4955
+1911,121200,13.3978
+1919,121600,9.6642
+1924,122000,15.7393
+1928,122400,18.9361
+1932,122800,19.4331
+1940,123200,8.6073
+1947,123600,10.161
+1954,124000,8.8061
+1959,124400,13.5384
+1966,124800,9.9289
+1981,125200,4.1506
+1991,125600,8.5539
+1995,126000,20.6911
+2002,126400,11.9305
+2007,126800,16.2045
+2016,127200,6.5353
+2021,127600,17.3839
+2027,128000,13.1571
+2035,128400,11.3393
+2039,128800,22.3028
+2047,129200,12.8433
+2055,129600,12.7583
+2066,130000,6.9837
+2071,130400,18.0303
+2076,130800,18.2896
+2084,131200,8.7988
+2093,131600,8.7713
+2098,132000,11.0877
+2109,132400,7.8183
+2115,132800,12.4679
+2124,133200,10.5047
+2137,133600,6.17
+2142,134000,20.3565
+2148,134400,10.4538
+2154,134800,12.7331
+2161,135200,12.5367
+2171,135600,5.7754
+2178,136000,13.3435
+2182,136400,13.3376
+2195,136800,5.0278
+2203,137200,8.039
+2215,137600,5.0622
+2225,138000,7.6281
+2232,138400,12.4199
+2243,138800,5.7324
+2249,139200,14.5818
+2255,139600,14.0929
+2262,140000,13.6329
+2267,140400,18.3515
+2272,140800,18.0695
+2280,141200,12.0349
+2287,141600,13.6652
+2296,142000,9.2929
+2305,142400,10.1985
+2312,142800,12.7522
+2323,143200,7.2459
+2331,143600,8.9751
+2338,144000,11.4881
+2344,144400,15.2227
+2351,144800,12.8927
+2358,145200,10.6543
+2362,145600,22.496
+2368,146000,13.9616
+2373,146400,18.1932
+2378,146800,16.1787
+2382,147200,21.2142
+2386,147600,22.1002
+2396,148000,8.9528
+2401,148400,15.8869
+2408,148800,13.7149
+2413,149200,13.7033
+2419,149600,17.2193
+2425,150000,11.3894
+2432,150400,13.8544
+2437,150800,17.5939
+2444,151200,12.8075
+2449,151600,12.1515
+2457,152000,10.4033
+2465,152400,11.4859
+2470,152800,14.4762
+2477,153200,12.3627
+2483,153600,14.8347
+2488,154000,18.2382
+2497,154400,9.2311
+2501,154800,19.7235
+2509,155200,13.3697
+2515,155600,11.9598
+2525,156000,7.2526
+2534,156400,9.3025
+2545,156800,8.9835
+2551,157200,12.1765
+2558,157600,14.0303
+2564,158000,13.4739
+2573,158400,9.8322
+2578,158800,19.6338
+2584,159200,15.7125
+2588,159600,17.0086
+2594,160000,14.7127
+2598,160400,23.1588
+2607,160800,11.0373
+2615,161200,9.348
+2619,161600,21.6514
+2624,162000,12.9316
+2631,162400,12.1088
+2636,162800,20.0918
+2640,163200,18.6887
+2644,163600,19.3577
+2653,164000,8.5057
+2662,164400,8.0083
+2668,164800,15.0007
+2676,165200,8.8861
+2682,165600,15.3621
+2689,166000,13.6995
+2696,166400,10.5381
+2701,166800,19.4263
+2708,167200,12.0695
+2713,167600,11.9025
+2719,168000,10.0897
+2725,168400,15.0383
+2731,168800,14.8992
+2735,169200,14.9242
+2739,169600,20.4302
+2745,170000,15.2987
+2750,170400,16.7812
+2754,170800,16.0345
+2759,171200,16.4285
+2768,171600,9.1208
+2780,172000,5.583
+2787,172400,8.2014
+2793,172800,10.1961
+2798,173200,17.1725
+2806,173600,9.115
+2814,174000,9.2754
+2821,174400,10.8946
+2827,174800,11.5879
+2834,175200,10.3869
+2840,175600,13.8918
+2845,176000,12.5769
+2854,176400,10.322
+2863,176800,6.8967
+2869,177200,17.4846
+2874,177600,19.6151
+2881,178000,12.9361
+2886,178400,18.2368
+2892,178800,12.8876
+2898,179200,12.3181
+2903,179600,17.6907
+2908,180000,15.7174
+2915,180400,11.7662
+2920,180800,17.438
+2925,181200,14.2649
+2931,181600,12.5882
+2936,182000,17.2888
+2942,182400,15.7864
+2946,182800,19.7236
+2952,183200,14.7757
+2957,183600,13.2554
+2962,184000,16.9161
+2966,184400,19.4477
+2971,184800,14.806
+2976,185200,14.7174
+2981,185600,14.6584
+2985,186000,13.1555
+2993,186400,8.2998
+2999,186800,10.4079
+3004,187200,14.5865
+3011,187600,9.0036
+3015,188000,13.7298
+3022,188400,8.8899
+3026,188800,15.7034
+3032,189200,11.7676
+3036,189600,17.0897
+3044,190000,10.1182
+3049,190400,13.9028
+3054,190800,16.7113
+3059,191200,16.4022
+3064,191600,18.3592
+3069,192000,17.6439
+3074,192400,15.5535
+3079,192800,15.6137
+3085,193200,14.7975
+3091,193600,11.9988
+3099,194000,10.8644
+3106,194400,14.6474
+3113,194800,12.833
+3117,195200,22.6677
+3121,195600,21.1913
+3126,196000,17.2308
+3131,196400,14.7737
+3140,196800,6.5658
+3147,197200,8.1251
+3155,197600,8.3248
+3162,198000,10.0096
+3168,198400,9.34
+3175,198800,9.7054
+3181,199200,9.9748
+3186,199600,11.6185
+3191,200000,10.6864

code/Lake application/logs/results_2/PPO_frozen_lake_log_4.csv

@@ -0,0 +1,501 @@
+episode,timestep,reward
+30,400,0.8897
+47,800,1.6985
+70,1200,1.5344
+81,1600,2.8837
+93,2000,2.7853
+99,2400,4.9228
+105,2800,4.8879
+109,3200,6.7205
+116,3600,5.1037
+121,4000,6.8188
+126,4400,5.7358
+130,4800,7.5711
+136,5200,5.2419
+141,5600,5.8543
+147,6000,5.2331
+152,6400,4.9293
+161,6800,4.0985
+166,7200,5.2082
+172,7600,5.3103
+177,8000,6.0557
+181,8400,6.9039
+188,8800,4.5342
+192,9200,9.1478
+199,9600,5.3605
+206,10000,5.1273
+211,10400,7.8887
+216,10800,6.2693
+220,11200,7.3964
+224,11600,7.1054
+230,12000,5.4118
+234,12400,7.4287
+238,12800,7.4478
+245,13200,4.4117
+249,13600,7.2105
+253,14000,6.7015
+257,14400,7.4679
+262,14800,6.5292
+267,15200,6.025
+271,15600,7.6356
+275,16000,7.6501
+280,16400,7.0316
+284,16800,8.6843
+289,17200,6.7495
+294,17600,6.023
+299,18000,6.415
+305,18400,5.4163
+311,18800,5.4882
+316,19200,6.2097
+320,19600,8.0329
+324,20000,7.5882
+329,20400,7.3095
+333,20800,7.5687
+339,21200,6.6201
+343,21600,9.5695
+349,22000,6.358
+354,22400,7.729
+362,22800,3.8919
+367,23200,6.2263
+372,23600,8.9296
+377,24000,8.9107
+383,24400,6.8109
+391,24800,4.3733
+397,25200,7.8942
+402,25600,8.1473
+408,26000,6.342
+414,26400,6.6643
+419,26800,8.5145
+423,27200,9.8487
+427,27600,8.3884
+432,28000,8.1417
+438,28400,6.5363
+442,28800,8.3084
+447,29200,7.5203
+451,29600,8.0109
+455,30000,7.1375
+459,30400,6.972
+464,30800,7.2792
+468,31200,8.3772
+472,31600,8.6912
+476,32000,7.6424
+480,32400,8.1047
+484,32800,6.6454
+489,33200,7.4736
+493,33600,8.1904
+497,34000,7.1456
+503,34400,6.2541
+507,34800,7.7885
+511,35200,7.3507
+515,35600,8.0471
+520,36000,8.5436
+524,36400,6.8725
+529,36800,8.4028
+535,37200,5.2433
+542,37600,4.7139
+546,38000,7.3213
+555,38400,3.8831
+561,38800,5.6601
+568,39200,4.7948
+576,39600,4.6981
+584,40000,4.3181
+589,40400,7.5472
+593,40800,9.3392
+602,41200,3.9924
+609,41600,7.1339
+615,42000,6.7132
+620,42400,6.7015
+628,42800,5.4925
+636,43200,3.7468
+644,43600,4.3569
+651,44000,5.8671
+655,44400,8.3115
+660,44800,9.1009
+665,45200,7.2625
+672,45600,5.378
+678,46000,5.686
+684,46400,5.8378
+688,46800,7.674
+693,47200,7.7574
+697,47600,9.4904
+706,48000,4.0155
+712,48400,9.193
+718,48800,6.0672
+723,49200,8.703
+729,49600,6.4219
+737,50000,5.2146
+742,50400,7.6968
+747,50800,9.559
+753,51200,6.7111
+758,51600,9.7201
+764,52000,7.5954
+770,52400,8.0675
+775,52800,7.1163
+782,53200,5.3886
+786,53600,10.9581
+791,54000,9.5825
+800,54400,4.9313
+808,54800,3.2748
+813,55200,9.4975
+819,55600,8.5919
+828,56000,4.0659
+834,56400,6.4677
+839,56800,8.6157
+847,57200,7.6231
+854,57600,6.1867
+864,58000,5.138
+875,58400,4.1107
+884,58800,4.6541
+890,59200,8.6775
+898,59600,4.5193
+903,60000,10.8015
+909,60400,7.2792
+916,60800,6.9898
+920,61200,9.2429
+926,61600,7.8279
+930,62000,9.559
+938,62400,6.2201
+942,62800,12.4695
+949,63200,6.0011
+955,63600,7.5678
+960,64000,8.5841
+965,64400,8.8059
+969,64800,9.559
+974,65200,8.137
+979,65600,6.2258
+985,66000,6.0418
+990,66400,7.6972
+994,66800,10.6031
+999,67200,6.4527
+1004,67600,7.6003
+1009,68000,8.036
+1014,68400,11.559
+1018,68800,9.9028
+1024,69200,8.8209
+1030,69600,6.8682
+1034,70000,9.1513
+1039,70400,8.8808
+1045,70800,6.2892
+1052,71200,6.6137
+1056,71600,9.1258
+1061,72000,8.2712
+1069,72400,5.1346
+1073,72800,9.3301
+1079,73200,8.6006
+1083,73600,9.7199
+1088,74000,8.5393
+1093,74400,9.4136
+1098,74800,9.2309
+1104,75200,9.9483
+1111,75600,6.868
+1117,76000,7.3642
+1123,76400,8.7512
+1128,76800,7.7363
+1133,77200,11.2048
+1138,77600,8.7672
+1142,78000,11.2022
+1148,78400,8.6267
+1156,78800,5.7085
+1165,79200,5.2502
+1171,79600,9.1847
+1181,80000,3.8257
+1187,80400,9.0944
+1193,80800,7.8396
+1204,81200,4.6099
+1214,81600,4.2423
+1223,82000,4.2463
+1231,82400,6.0866
+1236,82800,9.3631
+1244,83200,6.0139
+1250,83600,9.3664
+1258,84000,5.5445
+1263,84400,9.2419
+1270,84800,6.5126
+1277,85200,5.3284
+1284,85600,4.7442
+1289,86000,6.6238
+1295,86400,5.9787
+1300,86800,6.273
+1304,87200,7.7552
+1310,87600,6.0625
+1315,88000,6.2442
+1320,88400,7.2193
+1326,88800,5.8607
+1333,89200,5.8177
+1338,89600,5.6992
+1344,90000,6.309
+1350,90400,7.4904
+1357,90800,5.3341
+1362,91200,9.1276
+1370,91600,5.6335
+1377,92000,5.5404
+1382,92400,10.4014
+1387,92800,8.972
+1393,93200,7.6199
+1400,93600,7.0028
+1408,94000,6.7953
+1417,94400,4.607
+1425,94800,6.7686
+1431,95200,6.3672
+1437,95600,7.3133
+1442,96000,5.5286
+1449,96400,8.1326
+1454,96800,6.6459
+1459,97200,9.6138
+1465,97600,8.3167
+1470,98000,11.7781
+1474,98400,17.2643
+1481,98800,9.2478
+1490,99200,8.6222
+1497,99600,10.0905
+1504,100000,7.0319
+1510,100400,11.7434
+1515,100800,12.7016
+1519,101200,14.5775
+1525,101600,7.4347
+1531,102000,17.0148
+1535,102400,21.5497
+1541,102800,13.3595
+1546,103200,20.2214
+1553,103600,12.2627
+1562,104000,10.1528
+1572,104400,7.0324
+1577,104800,13.988
+1587,105200,9.7205
+1591,105600,22.2021
+1599,106000,12.8697
+1611,106400,6.7423
+1618,106800,9.1311
+1624,107200,13.5325
+1630,107600,15.3574
+1637,108000,11.4605
+1647,108400,8.1593
+1657,108800,7.3692
+1664,109200,12.4204
+1671,109600,16.0635
+1677,110000,11.0747
+1687,110400,10.0776
+1695,110800,9.713
+1704,111200,6.6402
+1709,111600,16.4947
+1714,112000,12.0573
+1720,112400,12.4928
+1726,112800,16.9818
+1731,113200,16.4082
+1735,113600,19.4684
+1741,114000,17.2942
+1746,114400,17.3803
+1752,114800,14.3429
+1759,115200,15.4686
+1764,115600,18.3797
+1771,116000,10.6607
+1778,116400,12.9278
+1783,116800,21.2477
+1789,117200,12.1737
+1795,117600,15.1248
+1801,118000,11.4594
+1808,118400,11.8572
+1816,118800,8.6953
+1822,119200,12.9991
+1830,119600,8.346
+1835,120000,16.5775
+1840,120400,18.7012
+1845,120800,18.5211
+1852,121200,12.3151
+1860,121600,11.191
+1868,122000,10.6305
+1875,122400,12.1361
+1883,122800,12.0561
+1887,123200,21.1206
+1895,123600,10.3102
+1901,124000,15.5468
+1905,124400,21.1214
+1913,124800,12.598
+1919,125200,8.6702
+1923,125600,19.5976
+1928,126000,17.347
+1936,126400,12.0519
+1944,126800,6.2953
+1949,127200,13.6435
+1956,127600,9.3424
+1960,128000,22.6692
+1966,128400,12.2863
+1973,128800,15.4013
+1978,129200,17.9858
+1988,129600,7.2154
+1996,130000,10.964
+2004,130400,10.9658
+2009,130800,16.1921
+2015,131200,19.8994
+2020,131600,12.5598
+2026,132000,18.5603
+2034,132400,8.9442
+2039,132800,15.7247
+2044,133200,19.6043
+2048,133600,22.708
+2055,134000,12.1769
+2059,134400,30.2886
+2064,134800,19.3976
+2069,135200,24.011
+2075,135600,22.3232
+2079,136000,22.4054
+2087,136400,14.8207
+2095,136800,14.1154
+2102,137200,13.3378
+2106,137600,22.9892
+2112,138000,19.1975
+2119,138400,16.2562
+2125,138800,16.5325
+2134,139200,9.7804
+2143,139600,12.9261
+2149,140000,15.1729
+2157,140400,11.4505
+2163,140800,16.225
+2168,141200,15.0464
+2175,141600,12.2286
+2181,142000,14.5324
+2187,142400,17.9193
+2192,142800,21.9792
+2202,143200,7.5693
+2214,143600,7.0395
+2219,144000,20.2988
+2230,144400,8.1503
+2237,144800,12.8959
+2246,145200,11.8272
+2254,145600,15.8534
+2259,146000,20.5079
+2266,146400,10.7379
+2271,146800,21.599
+2279,147200,12.679
+2284,147600,14.8514
+2291,148000,8.6118
+2297,148400,7.5502
+2306,148800,5.1645
+2313,149200,10.6152
+2319,149600,15.1497
+2324,150000,13.6594
+2331,150400,10.1251
+2337,150800,10.9294
+2345,151200,5.0712
+2350,151600,13.3293
+2359,152000,6.394
+2370,152400,4.9969
+2379,152800,7.9595
+2391,153200,4.0272
+2398,153600,7.3762
+2407,154000,7.3333
+2413,154400,10.8586
+2423,154800,9.7345
+2432,155200,7.9822
+2439,155600,10.3486
+2450,156000,6.3284
+2458,156400,9.4372
+2473,156800,3.132
+2481,157200,10.3754
+2487,157600,10.447
+2493,158000,15.9101
+2503,158400,8.3842
+2510,158800,15.4866
+2518,159200,11.682
+2526,159600,11.0361
+2537,160000,8.7871
+2545,160400,11.1971
+2555,160800,4.4022
+2562,161200,13.1779
+2568,161600,12.9045
+2579,162000,8.0949
+2587,162400,11.4998
+2597,162800,7.2315
+2604,163200,14.4484
+2609,163600,15.2563
+2622,164000,7.1052
+2631,164400,8.0708
+2636,164800,15.3412
+2646,165200,7.0698
+2654,165600,10.9479
+2661,166000,12.6783
+2666,166400,16.1794
+2671,166800,18.6718
+2676,167200,13.5239
+2681,167600,12.3507
+2691,168000,7.2502
+2700,168400,6.2099
+2707,168800,13.9091
+2712,169200,16.7988
+2717,169600,22.2866
+2728,170000,8.0224
+2738,170400,11.7132
+2743,170800,22.9338
+2750,171200,15.1354
+2762,171600,8.9432
+2768,172000,14.02
+2773,172400,19.0923
+2780,172800,14.6205
+2787,173200,12.9528
+2796,173600,11.7497
+2803,174000,14.0874
+2808,174400,21.8773
+2813,174800,16.507
+2819,175200,19.8032
+2827,175600,13.8242
+2833,176000,19.0166
+2838,176400,27.172
+2846,176800,11.3364
+2851,177200,19.2692
+2857,177600,21.4003
+2865,178000,13.1762
+2871,178400,20.7159
+2878,178800,12.615
+2882,179200,17.292
+2886,179600,22.37
+2893,180000,16.9824
+2903,180400,7.4276
+2912,180800,12.7024
+2918,181200,14.4444
+2926,181600,14.2302
+2932,182000,18.124
+2938,182400,13.5542
+2943,182800,33.0073
+2950,183200,12.0173
+2954,183600,22.766
+2959,184000,16.2093
+2965,184400,15.9019
+2972,184800,13.7782
+2976,185200,26.2554
+2985,185600,10.2687
+2991,186000,14.767
+2998,186400,12.8508
+3002,186800,19.9929
+3008,187200,16.2693
+3016,187600,14.6283
+3026,188000,9.5896
+3034,188400,11.7475
+3041,188800,12.2039
+3045,189200,23.8742
+3056,189600,8.3229
+3063,190000,11.436
+3070,190400,10.1774
+3077,190800,10.9202
+3082,191200,16.139
+3087,191600,17.0197
+3096,192000,6.3408
+3103,192400,10.6383
+3108,192800,14.7467
+3113,193200,16.4579
+3121,193600,9.5428
+3125,194000,22.9954
+3129,194400,11.4853
+3136,194800,8.377
+3141,195200,13.0133
+3145,195600,14.8357
+3150,196000,20.832
+3159,196400,8.4116
+3166,196800,16.5597
+3178,197200,6.8153
+3186,197600,11.2551
+3197,198000,9.8681
+3202,198400,16.0912
+3210,198800,13.4439
+3216,199200,11.4534
+3222,199600,16.3593
+3229,200000,11.4777

code/Lake application/logs/results_2/PPO_frozen_lake_log_5.csv

@@ -0,0 +1,501 @@
+episode,timestep,reward
+25,400,1.1394
+48,800,1.3002
+55,1200,4.2174
+65,1600,3.1185
+72,2000,4.6995
+77,2400,5.4912
+83,2800,5.3572
+87,3200,6.4659
+93,3600,5.7893
+97,4000,7.5009
+101,4400,7.0671
+105,4800,6.9336
+109,5200,7.0397
+114,5600,6.2202
+120,6000,4.966
+126,6400,5.4569
+131,6800,5.7776
+135,7200,7.3573
+139,7600,6.3645
+144,8000,6.1568
+148,8400,7.3496
+152,8800,6.4834
+157,9200,6.0306
+162,9600,6.5141
+167,10000,6.2576
+172,10400,5.8399
+176,10800,5.7246
+182,11200,5.6336
+186,11600,6.3594
+192,12000,5.9243
+196,12400,7.3959
+200,12800,7.3087
+204,13200,7.4288
+208,13600,6.0092
+212,14000,7.4051
+219,14400,5.0751
+223,14800,6.1337
+228,15200,6.0328
+233,15600,6.2323
+237,16000,7.2947
+242,16400,6.21
+247,16800,5.2467
+251,17200,7.301
+256,17600,6.7575
+260,18000,7.3051
+264,18400,7.3659
+270,18800,4.9364
+275,19200,4.7764
+280,19600,6.6525
+286,20000,5.1666
+290,20400,7.305
+295,20800,5.3332
+301,21200,5.8009
+306,21600,6.3444
+311,22000,5.38
+316,22400,5.8886
+321,22800,5.9808
+326,23200,6.1518
+331,23600,6.2596
+335,24000,7.4699
+340,24400,5.0459
+344,24800,7.1979
+348,25200,7.1906
+354,25600,5.5289
+358,26000,7.4249
+362,26400,7.3209
+366,26800,6.8339
+371,27200,6.2896
+375,27600,5.8839
+382,28000,4.5678
+387,28400,6.4948
+391,28800,7.3334
+395,29200,6.8561
+400,29600,5.7921
+405,30000,5.775
+411,30400,5.2429
+416,30800,6.3733
+420,31200,6.9394
+424,31600,7.4063
+430,32000,5.6989
+434,32400,6.6094
+439,32800,6.4591
+443,33200,7.5158
+448,33600,5.1683
+457,34000,3.2012
+462,34400,6.4385
+466,34800,7.6264
+471,35200,6.8552
+475,35600,6.6808
+479,36000,7.5093
+485,36400,6.0369
+491,36800,5.9012
+497,37200,4.7016
+507,37600,3.1521
+513,38000,5.874
+517,38400,7.713
+522,38800,6.4352
+526,39200,7.6985
+530,39600,7.8557
+540,40000,3.9398
+546,40400,5.5794
+551,40800,6.7952
+555,41200,7.4607
+561,41600,6.2385
+565,42000,6.9421
+571,42400,5.5255
+577,42800,7.0349
+583,43200,7.038
+587,43600,8.1308
+592,44000,6.715
+598,44400,6.6102
+603,44800,5.7423
+609,45200,5.7222
+616,45600,4.6904
+621,46000,6.6802
+627,46400,5.1794
+631,46800,7.8428
+636,47200,6.6569
+640,47600,6.0637
+645,48000,6.6306
+649,48400,7.8827
+654,48800,7.2829
+660,49200,5.3484
+666,49600,4.965
+671,50000,6.2847
+675,50400,7.2723
+681,50800,5.6506
+689,51200,4.2316
+693,51600,6.588
+699,52000,5.8609
+704,52400,6.7229
+710,52800,5.2104
+716,53200,5.5105
+720,53600,6.7428
+728,54000,4.2067
+733,54400,6.1901
+737,54800,6.1446
+743,55200,6.4328
+747,55600,7.5352
+753,56000,4.6297
+758,56400,6.7488
+765,56800,4.4836
+769,57200,7.332
+773,57600,7.3174
+778,58000,7.5865
+784,58400,5.1354
+788,58800,7.5499
+792,59200,7.6182
+802,59600,2.9436
+807,60000,6.7174
+812,60400,6.5712
+820,60800,3.8267
+827,61200,5.0311
+831,61600,7.6775
+835,62000,6.2429
+842,62400,5.1519
+847,62800,5.1346
+858,63200,3.2845
+862,63600,6.7612
+868,64000,5.4555
+874,64400,5.6836
+879,64800,5.4058
+883,65200,7.5245
+887,65600,7.5208
+892,66000,7.063
+897,66400,6.6028
+903,66800,6.9216
+908,67200,7.874
+912,67600,6.5846
+918,68000,6.0015
+923,68400,6.5042
+927,68800,6.4518
+932,69200,7.5216
+937,69600,7.0083
+942,70000,6.8853
+948,70400,5.5392
+954,70800,8.848
+960,71200,11.4058
+967,71600,7.5975
+972,72000,8.9093
+979,72400,9.266
+985,72800,9.023
+991,73200,11.5379
+1000,73600,7.4839
+1006,74000,10.8982
+1011,74400,9.6794
+1016,74800,11.5398
+1022,75200,12.7577
+1032,75600,7.5257
+1040,76000,10.313
+1052,76400,4.9592
+1057,76800,9.929
+1065,77200,9.0269
+1074,77600,10.0283
+1080,78000,9.1994
+1084,78400,17.318
+1090,78800,8.6919
+1099,79200,8.3069
+1104,79600,14.1304
+1110,80000,13.6171
+1114,80400,16.2281
+1119,80800,13.7672
+1127,81200,9.2344
+1133,81600,12.9287
+1139,82000,9.0991
+1146,82400,10.4085
+1154,82800,6.4297
+1161,83200,11.2657
+1170,83600,8.6181
+1176,84000,10.1173
+1182,84400,10.5116
+1189,84800,7.5418
+1197,85200,7.8979
+1204,85600,10.4355
+1214,86000,5.9039
+1228,86400,4.1987
+1238,86800,6.374
+1246,87200,5.9424
+1251,87600,15.9749
+1257,88000,14.0111
+1261,88400,19.8135
+1270,88800,7.7016
+1276,89200,10.2966
+1281,89600,12.6069
+1288,90000,10.6588
+1293,90400,17.1633
+1300,90800,9.8388
+1308,91200,9.1061
+1314,91600,10.2858
+1319,92000,15.5991
+1323,92400,19.9744
+1329,92800,13.7349
+1333,93200,16.2973
+1340,93600,12.1433
+1346,94000,15.6216
+1350,94400,16.604
+1356,94800,17.9473
+1360,95200,22.7261
+1367,95600,9.7798
+1372,96000,18.8177
+1376,96400,23.3835
+1380,96800,22.7676
+1385,97200,17.9196
+1391,97600,11.0045
+1398,98000,12.9032
+1403,98400,19.9428
+1411,98800,10.0649
+1415,99200,23.4105
+1421,99600,14.4659
+1427,100000,12.6086
+1434,100400,11.2016
+1440,100800,16.4195
+1445,101200,14.8528
+1451,101600,13.3423
+1457,102000,15.6651
+1462,102400,19.3589
+1467,102800,19.2814
+1472,103200,17.4048
+1479,103600,10.9156
+1484,104000,19.1347
+1489,104400,16.7974
+1500,104800,7.7077
+1508,105200,8.3873
+1515,105600,11.8502
+1522,106000,12.4617
+1529,106400,10.997
+1534,106800,18.1086
+1538,107200,21.5753
+1542,107600,18.1229
+1548,108000,19.0807
+1553,108400,19.9151
+1557,108800,24.3347
+1565,109200,11.5838
+1571,109600,10.4892
+1576,110000,18.4124
+1583,110400,9.6659
+1589,110800,15.3845
+1594,111200,19.4332
+1603,111600,9.1848
+1608,112000,19.8579
+1614,112400,14.6327
+1620,112800,15.4716
+1628,113200,7.6968
+1633,113600,14.4689
+1637,114000,19.6793
+1642,114400,20.0721
+1647,114800,15.0668
+1652,115200,17.4454
+1657,115600,19.6026
+1663,116000,14.572
+1669,116400,12.3857
+1675,116800,15.0434
+1679,117200,23.0521
+1685,117600,15.9115
+1691,118000,16.4641
+1695,118400,18.5005
+1701,118800,13.3055
+1705,119200,20.5855
+1711,119600,15.2568
+1716,120000,17.1653
+1721,120400,16.2964
+1726,120800,17.3911
+1731,121200,18.9176
+1735,121600,20.2643
+1741,122000,21.9711
+1748,122400,15.1474
+1752,122800,21.0002
+1756,123200,27.017
+1761,123600,21.0847
+1768,124000,16.9835
+1774,124400,16.988
+1780,124800,17.102
+1784,125200,27.0598
+1790,125600,18.8929
+1795,126000,18.4346
+1799,126400,27.4704
+1803,126800,20.451
+1812,127200,12.5548
+1816,127600,24.9355
+1821,128000,14.002
+1826,128400,16.9177
+1834,128800,9.9734
+1847,129200,4.5403
+1853,129600,10.6147
+1865,130000,6.26
+1872,130400,12.1423
+1877,130800,16.8818
+1882,131200,14.1034
+1887,131600,19.5902
+1894,132000,12.8515
+1899,132400,16.2843
+1904,132800,15.5745
+1914,133200,8.3905
+1922,133600,13.0687
+1929,134000,13.9548
+1935,134400,11.7435
+1944,134800,12.2644
+1949,135200,18.9015
+1957,135600,10.5449
+1968,136000,6.614
+1979,136400,7.9006
+1988,136800,8.6919
+1993,137200,19.6558
+1999,137600,13.7705
+2004,138000,19.7431
+2010,138400,16.1015
+2018,138800,7.969
+2024,139200,10.7627
+2033,139600,9.8075
+2038,140000,15.1353
+2044,140400,14.33
+2051,140800,13.0915
+2059,141200,11.0496
+2067,141600,8.5425
+2074,142000,12.6574
+2079,142400,18.6865
+2083,142800,18.9614
+2091,143200,7.6956
+2097,143600,19.3319
+2106,144000,6.1586
+2112,144400,11.7879
+2117,144800,14.6574
+2124,145200,9.78
+2131,145600,8.172
+2138,146000,9.3161
+2145,146400,10.1464
+2151,146800,13.3546
+2158,147200,10.2643
+2162,147600,17.7297
+2167,148000,12.2066
+2174,148400,11.723
+2181,148800,12.61
+2185,149200,20.9512
+2192,149600,9.257
+2200,150000,13.0471
+2206,150400,10.6689
+2212,150800,16.0447
+2219,151200,13.6559
+2225,151600,13.2487
+2235,152000,7.2764
+2242,152400,11.6686
+2248,152800,12.3615
+2255,153200,13.5621
+2263,153600,9.6251
+2269,154000,9.0672
+2276,154400,13.0372
+2281,154800,16.5969
+2286,155200,18.0225
+2292,155600,14.2052
+2298,156000,11.6988
+2304,156400,9.5336
+2312,156800,8.0191
+2324,157200,5.3825
+2330,157600,8.1571
+2337,158000,10.3493
+2344,158400,10.4621
+2350,158800,10.5959
+2356,159200,7.2691
+2364,159600,6.3992
+2372,160000,7.9295
+2377,160400,7.4555
+2384,160800,7.9996
+2389,161200,13.726
+2395,161600,7.7046
+2399,162000,16.8889
+2410,162400,5.528
+2422,162800,4.9575
+2429,163200,8.7608
+2438,163600,7.2575
+2446,164000,7.0835
+2452,164400,10.8246
+2459,164800,7.831
+2467,165200,6.133
+2476,165600,7.8923
+2483,166000,8.0733
+2489,166400,11.0754
+2493,166800,20.1624
+2500,167200,12.4293
+2504,167600,15.0355
+2510,168000,13.0286
+2518,168400,7.4877
+2523,168800,12.2261
+2531,169200,7.3993
+2536,169600,8.9622
+2542,170000,10.8549
+2547,170400,11.6566
+2555,170800,8.8997
+2562,171200,8.861
+2568,171600,13.2091
+2574,172000,10.3659
+2580,172400,11.7853
+2585,172800,14.792
+2592,173200,10.6782
+2602,173600,6.9546
+2609,174000,9.9301
+2614,174400,17.7772
+2623,174800,6.2142
+2630,175200,12.9292
+2637,175600,10.1204
+2645,176000,9.0597
+2651,176400,15.3755
+2657,176800,13.128
+2662,177200,19.3868
+2670,177600,10.7437
+2679,178000,6.904
+2686,178400,9.2907
+2696,178800,6.4837
+2705,179200,8.2248
+2711,179600,12.1069
+2720,180000,8.3973
+2724,180400,22.0167
+2729,180800,15.3768
+2734,181200,15.6707
+2738,181600,21.4503
+2744,182000,14.5199
+2750,182400,16.9138
+2756,182800,12.2078
+2762,183200,15.9948
+2769,183600,12.4933
+2775,184000,14.7625
+2781,184400,16.4597
+2786,184800,12.6036
+2793,185200,11.1748
+2799,185600,13.5976
+2805,186000,13.175
+2811,186400,14.25
+2816,186800,22.0337
+2822,187200,17.4297
+2827,187600,17.1395
+2832,188000,18.1786
+2837,188400,16.0257
+2844,188800,11.8928
+2850,189200,16.6968
+2855,189600,19.1383
+2860,190000,21.8792
+2864,190400,27.2875
+2868,190800,25.2937
+2873,191200,20.9754
+2882,191600,12.5236
+2886,192000,26.9158
+2896,192400,9.8619
+2912,192800,5.0885
+2923,193200,4.6341
+2930,193600,13.8767
+2937,194000,11.1766
+2944,194400,14.2145
+2952,194800,7.6092
+2961,195200,9.0705
+2968,195600,10.5332
+2973,196000,13.6747
+2979,196400,17.6262
+2986,196800,12.0028
+2997,197200,5.1463
+3008,197600,9.2429
+3015,198000,11.3805
+3022,198400,7.9289
+3029,198800,12.3534
+3033,199200,26.1072
+3041,199600,13.3515
+3048,200000,11.5205

code/Lake application/plot_figure.py

@@ -0,0 +1,157 @@
+# -*- coding: utf-8 -*-
+"""
+Created on Mon Mar  6 16:30:32 2023
+
+@author: leona
+"""
+
+import os
+import pandas as pd
+import matplotlib.pyplot as plt
+import seaborn as sns
+
+
+
+
+def save_graph():
+    print("============================================================================================")
+    
+    # experiment_name = '15items_5machines_i100'
+    # experiment_name = '20items_10machines'
+    # experiment_name = '25items_10machines'
+    experiment_name = 'frozen_lake'
+    env_name = experiment_name
+    
+    rolling_window = 10
+
+    BASE_DIR = os.path.dirname(os.path.abspath('__file__'))
+    BASE_DIR = BASE_DIR+'\\Post-Doctorate\\Lot-sizing\\Lake application'  
+
+    # make directory for saving figures
+    figures_dir = BASE_DIR + "\\results"
+    
+    if not os.path.exists(figures_dir):
+        os.makedirs(figures_dir)
+
+    # make environment directory for saving figures
+    figures_dir = figures_dir + '/' + env_name + '_PPO'+'/'
+    if not os.path.exists(figures_dir):
+        os.makedirs(figures_dir)
+
+    #fig_save_path = figures_dir + '/PPO_' + env_name + '_fig_' + str(fig_num) + '.png'
+
+    # get number of log files in directory
+     
+    # Use the logs file in the root path of the main.
+    LOG_DIR = os.path.join(BASE_DIR,'logs')
+    
+    log_dir = LOG_DIR + '/' + env_name + '_PPO' + '/'
+
+    current_num_files = next(os.walk(log_dir))[2]
+    num_runs = len(current_num_files)-1
+
+    all_runs_ppo = []
+    
+    
+    ########################################################################################
+    for run_num in range(num_runs):
+        run_num = run_num + 1
+        log_f_name = log_dir + '/PPO_' + env_name + "_log_" + str(run_num) + ".csv"
+        print("loading data from : " + log_f_name)
+        data = pd.read_csv(log_f_name)
+        data = pd.DataFrame(data)
+
+        print("data shape : ", data.shape)
+
+        all_runs_ppo.append(data)
+        print("--------------------------------------------------------------------------------------------")
+
+    # average all runs
+    df_concat = pd.concat(all_runs_ppo)
+    
+    
+    #Apply rolling mean to reward values
+    df_concat['reward_mean'] = df_concat['reward'].rolling(window=rolling_window, win_type='triang', min_periods=1).mean()
+    
+    # Drop NaN values from beginning of rolling mean
+    df_concat = df_concat.dropna().reset_index(drop=True)
+    
+    # Calculate mean and standard deviation of reward values
+    reward_mean = df_concat.groupby('timestep')['reward_mean'].mean().iloc[rolling_window:]
+    reward_std = df_concat.groupby('timestep')['reward_mean'].std().iloc[rolling_window:]
+    
+    # Set up plot using seaborn
+    sns.set_style("whitegrid")
+    fig, ax = plt.subplots(figsize=(10, 6))
+    
+    sns.set_style("whitegrid")
+    # Plot mean reward with shaded confidence interval
+    sns.lineplot(x=reward_mean.index, y=reward_mean, ax=ax,label='PPO')
+    ax.fill_between(reward_mean.index, reward_mean - reward_std, reward_mean + reward_std, alpha=0.2)
+    # keep only reward_smooth in the legend and rename it
+
+
+    ########################################################################################
+    
+    
+    log_dir = LOG_DIR + '/' + env_name + '_PDPPO' + '/'
+
+    current_num_files = next(os.walk(log_dir))[2]
+    num_runs = len(current_num_files)-1
+
+    all_runs = []
+    
+    for run_num in range(num_runs):
+        run_num = run_num + 1
+        log_f_name = log_dir + 'PDPPO_' + env_name + "_log_" + str(run_num) + ".csv"
+        print("loading data from : " + log_f_name)
+        data = pd.read_csv(log_f_name)
+        data = pd.DataFrame(data)
+
+        print("data shape : ", data.shape)
+
+        all_runs.append(data)
+        print("--------------------------------------------------------------------------------------------")
+
+    # average all runs
+    df_concat = pd.concat(all_runs)
+    
+    #Apply rolling mean to reward values
+    df_concat['reward_mean'] = df_concat['reward'].rolling(window=rolling_window, win_type='triang', min_periods=1).mean()
+    
+    # Drop NaN values from beginning of rolling mean
+    df_concat = df_concat.dropna().reset_index(drop=True)
+    
+    # Calculate mean and standard deviation of reward values
+    reward_mean = df_concat.groupby('timestep')['reward_mean'].mean().iloc[rolling_window:]
+    reward_std = df_concat.groupby('timestep')['reward_mean'].std().iloc[rolling_window:]
+    
+    # Plot mean reward with shaded confidence interval
+    sns.lineplot(x=reward_mean.index, y=reward_mean, ax=ax,label='PDPPO')
+    ax.fill_between(reward_mean.index, reward_mean - reward_std, reward_mean + reward_std, alpha=0.2)
+    #ax.set(xlabel='Timestep', ylabel='Mean Reward', title='Average Reward with Confidence Interval')
+    ax.legend()
+    ########################################################################################
+    
+    # ax.set_yticks(np.arange(0, 1800, 200))
+    # ax.set_xticks(np.arange(0, int(4e6), int(5e5)))
+
+    ax.grid(color='gray', linestyle='-', linewidth=1, alpha=0.2)
+
+    ax.set_xlabel("Timesteps", fontsize=12)
+    ax.set_ylabel("Rewards", fontsize=12)
+
+    fig = plt.gcf()
+
+    fig.set_size_inches(15, 4)
+
+    print("============================================================================================")
+    fig.savefig(os.path.join(figures_dir, f'{experiment_name}.pdf'), dpi=300, bbox_inches='tight')
+    print("figure saved at : ", figures_dir)
+    print("============================================================================================")
+    
+
+if __name__ == '__main__':
+
+    save_graph()
+