I am working on a project to combine reinforcement learning with traffic light simulations using the package Pybrain. I have read the tutorial and implemented my own subclasses of Environment and Task. I am using an ActionValueNetwork as controller because I want my state to be a vector with continuous values such that it can contain information about for example the number of cars waiting on each lane, the total waiting times per lane and more.
I set the input dimensions of the ActionValueNetwork to the dimensions of my state vector, which would suggest that it's possible to use vectors as state variables. When I use the Q-learner or the SARSA learner the code runs fine at first but I obtain an error message as soon as the method learn() is called. This function contains the line
state = int(state)
and the error message is
TypeError: only length-1 arrays can be converted to Python scalars
which would suggest that only scalar shaped states can be used.
Does the pybrain reinforcement learning environment support vector shaped states? If so, how can I modify my code such that it will work with their implementations of Q-learning or other methods?
I have found my problem in the meantime. I was using the Q() learner but should have used the NFQ() learner which implements the neural-fitted Q-learning algorithm. Now it works.