forum: HJxPAFgEON

authors:
Jacob Biloki
Chen Liang
Ni Lao

keywords:
Neural Networks
Planning
Reinforcement Learning
Structured Prediction
WikiTableQuestions

TL;DR: A model-based planning component improves RL-based semantic parsing on WikiTableQuestions.

abstract: We consider the problem of weakly supervised structured prediction (SP) with reinforcement learning (RL) – for example, given a database table and a question, perform a sequence of computation actions on the table, which generates a response and  receives  a  binary  success-failure  reward.   This  line  of  research  has  been successful by leveraging RL to directly optimizes the desired metrics of the SP tasks – for example, the accuracy in question answering or BLEU score in machine translation.  However, different from the common RL settings, the environment dynamics is deterministic in SP, which hasn’t been fully utilized by the model-freeRL methods that are usually applied. Since SP models usually have full access to the environment dynamics, we propose to apply model-based RL methods, which rely on planning as a primary model component. We demonstrate the effectiveness of planning-based SP with a Neural Program Planner (NPP), which, given a set of candidate programs from a pretrained search policy, decides which program is the most promising considering all the information generated from executing these programs. We evaluate NPP on weakly supervised program synthesis from natural language(semantic parsing) by stacked learning a planning module based on pretrained search policies. On the WIKITABLEQUESTIONS benchmark, NPP achieves a new state-of-the-art of 47.2% accuracy.

venue: drlStructPred 2019

_bibtex:
@misc{
biloki2019neural,
title={Neural Program Planner for Structured Predictions},
author={Jacob Biloki and Chen Liang and Ni Lao},
year={2019},
url={https://openreview.net/forum?id=HJxPAFgEON}
}


title: LEARNING NEUROSYMBOLIC GENERATIVE MODELS VIA PROGRAM SYNTHESIS

authors:
Halley Young
Osbert Bastani
Mayur Naik

keywords:
structure
deep learning
generative models
structured prediction

TL;DR: Applying program synthesis to the tasks of image completion and generation within a deep learning framework

abstract: Significant strides have been made toward designing better generative models in recent years. Despite this progress, however, state-of-the-art approaches are still largely unable to capture complex global structure in data. For example, images of buildings typically contain spatial patterns such as windows repeating at regular intervals; state-of-the-art generative methods can’t easily reproduce these structures. We propose to address this problem by incorporating programs representing global structure into the generative model—e.g., a 2D for-loop may represent a configuration of windows. Furthermore, we propose a framework for learning these models by leveraging program synthesis to generate training data. On both synthetic and real-world data, we demonstrate that our approach is substantially better than the state-of-the-art at both generating and completing images that contain global structure.

_bibtex:
@misc{
young2019learning,
title={{LEARNING} {NEUROSYMBOLIC} {GENERATIVE} {MODELS} {VIA} {PROGRAM} {SYNTHESIS}},
author={Halley Young and Osbert Bastani and Mayur Naik},
year={2019},
url={https://openreview.net/forum?id=S1gUCFx4dN}
}


title: A Study of State Aliasing in Structured Prediction with RNNs

authors:
Layla El Asri
Adam Trischler

keywords:
deep reinforcement learning
structured prediction
dialogue

abstract: End-to-end reinforcement learning agents learn a state representation and a policy at the same time. Recurrent neural networks (RNNs) have been trained successfully as reinforcement learning agents in settings like dialogue that require structured prediction. In this paper, we investigate the representations learned by RNN-based agents when trained with both policy gradient and value-based methods. We show through extensive experiments and analysis that, when trained with policy gradient, recurrent neural networks often fail to learn a state representation that leads to an optimal policy in settings where the same action should be taken at different states. To explain this failure, we highlight the problem of state aliasing, which entails conflating two or more distinct states in the representation space. We demonstrate that state aliasing occurs when several states share the same optimal action and the agent is trained via policy gradient. We characterize this phenomenon through experiments on a simple maze setting and a more complex text-based game, and make recommendations for training RNNs with reinforcement learning.

_bibtex:
@misc{
asri2019a,
title={A Study of State Aliasing in Structured Prediction with {RNN}s},
author={Layla El Asri and Adam Trischler},
year={2019},
url={https://openreview.net/forum?id=S1eU0KxE_4}
}


authors:
Zafarali Ahmed
Arjun Karuvally
Doina Precup
Simon Gravel

keywords:
variational inference
reinforcement learning
monte carlo methods
stochastic processes

abstract: The problem of inferring unobserved values in a partially observed trajectory from a stochastic process can be considered as a structured prediction problem. Traditionally inference is conducted using heuristic-based Monte Carlo methods. This work considers learning heuristics by leveraging a connection between policy optimization reinforcement learning and approximate inference. In particular, we learn proposal distributions used in importance samplers by casting it as a variational inference problem. We then rewrite the variational lower bound as a policy optimization problem similar to Weber et al. (2015) allowing us to transfer techniques from reinforcement learning. We apply this technique to a simple stochastic process as a proof-of-concept and show that while it is viable, it will require more engineering effort to scale inference for rare observations

_bibtex:
@misc{
ahmed2019learning,
title={Learning proposals for sequential importance samplers using reinforced variational inference},
author={Zafarali Ahmed and Arjun Karuvally and Doina Precup and Simon Gravel},
year={2019},
url={https://openreview.net/forum?id=HJgxTf89vV}
}


title: Buy 4 REINFORCE Samples, Get a Baseline for Free!

authors:
Wouter Kool
Herke van Hoof
Max Welling

keywords:
reinforce
multiple samples
baseline
sequence generation
structured prediction
travelling salesman problem

TL;DR: We show that by drawing multiple samples (predictions) per input (datapoint), we can learn with less data as we freely obtain a REINFORCE baseline.

abstract: REINFORCE can be used to train models in structured prediction settings to directly optimize the test-time objective. However, the common case of sampling one prediction per datapoint (input) is data-inefficient. We show that by drawing multiple samples (predictions) per datapoint, we can learn with significantly less data, as we freely obtain a REINFORCE baseline to reduce variance. Additionally we derive a REINFORCE estimator with baseline, based on sampling without replacement. Combined with a recent technique to sample sequences without replacement using Stochastic Beam Search, this improves the training procedure for a sequence model that predicts the solution to the Travelling Salesman Problem.

_bibtex:
@misc{
kool2019buy,
title={Buy 4 {REINFORCE} Samples, Get a Baseline for Free!},
author={Wouter Kool and Herke van Hoof and Max Welling},
year={2019},
url={https://openreview.net/forum?id=r1lgTGL5DE}
}


authors:
Rodrigo Nogueira
Jannis Bulian
Massimiliano Ciaramita

keywords:
natural language
reinforcement learning
structured prediction
multi-agent learning
deep learning

TL;DR: We use reinforcement learning for query reformulation on two tasks and surprisingly find that when training multiple agents diversity of the reformulations is more important than specialisation.

abstract:
We investigate methods to efficiently learn diverse strategies in reinforcement learning for a generative structured prediction problem: query reformulation. In the proposed framework an agent consists of multiple specialized sub-agents and a meta-agent that learns to aggregate the answers from sub-agents to produce a final answer. Sub-agents are trained on disjoint partitions of the training data, while the meta-agent is trained on the full training set. Our method makes learning faster, because it is highly parallelizable, and has better generalization performance than strong baselines, such as
an ensemble of agents trained on the full data. We evaluate on the tasks of document retrieval and question answering. The
improved performance seems due to the increased diversity of reformulation strategies. This suggests that multi-agent, hierarchical approaches might play an important role in structured prediction tasks of this kind. However, we also find that it is not obvious how to characterize diversity in this context, and a first attempt based on clustering did not produce good results. Furthermore, reinforcement learning for the reformulation task is hard in high-performance regimes. At best, it only marginally improves over the state of the art, which highlights the complexity of training models in this framework for end-to-end language understanding problems.

_bibtex:
@misc{
nogueira2019multiagent,
title={Multi-agent query reformulation: Challenges and the role of diversity},
author={Rodrigo Nogueira and Jannis Bulian and Massimiliano Ciaramita},
year={2019},
url={https://openreview.net/forum?id=BJeypMU5wE}
}


title: Connecting the Dots Between MLE and RL for Sequence Generation

authors:
Bowen Tan*
Zhiting Hu*
Zichao Yang
Ruslan Salakhutdinov
Eric P. Xing

keywords:
sequence generation
maximum likelihood learning
reinforcement learning
policy optimization
text generation
reward augmented maximum likelihood
exposure bias

TL;DR: A unified perspective of various learning algorithms for sequence generation, such as MLE, RL, RAML, data noising, etc.

abstract: Sequence generation models such as recurrent networks can be trained with a diverse set of learning algorithms. For example, maximum likelihood learning is simple and efficient, yet suffers from the exposure bias problem. Reinforcement learning like policy gradient addresses the problem but can have prohibitively poor exploration efficiency. A variety of other algorithms such as RAML, SPG, and data noising, have also been developed in different perspectives. This paper establishes a formal connection between these algorithms. We present a generalized entropy regularized policy optimization formulation, and show that the apparently divergent algorithms can all be reformulated as special instances of the framework, with the only difference being the configurations of reward function and a couple of hyperparameters. The unified interpretation offers a systematic view of the varying properties of exploration and learning efficiency. Besides, based on the framework, we present a new algorithm that dynamically interpolates among the existing algorithms for improved learning. Experiments on machine translation and text summarization demonstrate the superiority of the proposed algorithm.

_bibtex:
@misc{
tan*2019connecting,
title={Connecting the Dots Between {MLE} and {RL} for Sequence Generation},
author={Bowen Tan* and Zhiting Hu* and Zichao Yang and Ruslan Salakhutdinov and Eric P. Xing},
year={2019},
url={https://openreview.net/forum?id=Syl1pGI9wN}
}


authors:
Yesmina Jaafra
Jean Luc Laurent
Aline Deruyver
Mohamed Saber Naceur

keywords:
Neural networks
Deep reinforcement learning
Actor-critic model
Autonomous driving
Carla simulator

TL;DR: An actor-critic reinforcement learning approach with multi-step returns applied to autonomous driving with Carla simulator.

abstract: Autonomous driving is still considered as an “unsolved problem” given its inherent important variability and that many processes associated with its development like vehicle control and scenes recognition remain open issues. Despite reinforcement learning algorithms have achieved notable results in games and some robotic manipulations, this technique has not been widely scaled up to the more challenging real world applications like autonomous driving. In this work, we propose a deep reinforcement learning (RL) algorithm embedding an actor critic architecture with multi-step returns to achieve a better robustness of the agent learning strategies when acting in complex and unstable environments. The experiment is conducted with Carla simulator offering a customizable and realistic urban driving conditions. The developed deep actor RL guided by a policy-evaluator critic distinctly surpasses the performance of a standard deep RL agent.

_bibtex:
@misc{
jaafra2019robust,
title={Robust Reinforcement Learning for Autonomous Driving },
author={Yesmina Jaafra and Jean Luc Laurent and Aline Deruyver and Mohamed Saber Naceur},
year={2019},
url={https://openreview.net/forum?id=Bke03G85DN}
}