The PinBall Domain[ Overview | Features | Documentation | Downloads | Python Version | Papers ]
The PinBall domain is a 4-dimensional continuous test domain for Reinforcement Learning (RL) algorithms. A small blue ball is placed in an arena where it must be manouvered into a red hole. The ball is dynamic (drag coefficient 0.995), so its state is described by four variables: x, y, x velocity and y velocity. Collisions with obstacles are fully elastic and cause the ball to bounce, so rather than merely avoiding obstacles the agent may choose to use them to efficiently reach the hole.
There are five primitive actions: adding or subtracting a small force to x velocity or y velocity (which incurs a reward of -5 per action), or leaving them unchanged (which incurs a reward of -1 per timestep); reaching the goal obtains a reward of 10,000.
The PinBall domain is an interesting test domain for RL algorithms because its dynamic aspects, sharp discontinuities, and extended dynamic control characteristics make it difficult for control and for function approximation -- much more difficult than a simple navigation task, or other common benchmarks tasks (e.g., Mountain Car and Acrobot).
The domain is written in Java (full source code and JavaDoc is available under the GPL) and reads in a configuration file that specifies the size and locations of the ball and target, and descriptions of the obstacles. Users can therefore create their own customized versions of the domain to suit their own needs, for which a GUI configuration editor is provided. Below are two example configurations which are available from the downloads section.
Pierre-Luc Bacon has ported Pinball to Python. This version also has a GPL license and an interface to RL-Glue, and has been incorporated into Will Dabney's PyRL code. It requires the RLGlue codec for Python and the pygame package.
The Pinball Domain was introduced in:
The Pinball Domain has also appeared in: