Deep Q Learning agent finds solution then diverges again
I am trying to train a DQN Agent to solve AI Gym's Cartpole-v0 environment. I have started with this person's implementation just to get some hands-on experience. What I noticed is that during training, after many episodes the agent finds the solution and is able to keep the pole upright for the maximum amount of timesteps. However, after further training, the policy looks like it becomes more stochastic and it can't keep the pole upright anymore and goes in and out of a good policy. I'm pretty confused by this why wouldn't further training and experience help the agent? At episodes my epsilon for random action becomes very low, so it should be operating on just making the next prediction. So why does it on some training episodes fail to keep the pole upright and on others it succeeds?
Here is a picture of my reward-episode curve during the training process of the above linked implementation.
This actually looks fairly normal to me, in fact I guessed your results were from CartPole before reading the whole question.
I have a few suggestions:
- When you're plotting results, you should plot averages over a few random seeds. Not only is this generally good practice (it shows how sensitive your algo is to seeds), it'll smooth out your graphs and give you a better understanding of the "skill" of your agent. Don't forget, the environment and the policy are stochastic, so it's not completely crazy that your agent exhibits this type of behavior.
- Assuming you're implementing e-greedy exploration, what's your epsilon value? Are you decaying it over time? The issue could also be that your agent is still exploring a lot even after it found a good policy.
- Have you played around with hyperparameters, like learning rate, epsilon, network size, replay buffer size, etc? Those can also be the culprit.
- The “Forward/Backward Passage Size” is too large for the pytorch model (Yolov3)
- How do I use distributed DNN training in TensorFlow?
- Neural network learning to sum two numbers
- Implementation of F1-score, IOU and Dice Score
- use matplotlib_inline and torch、d2l show error :NotImplementedError: Implement enable_gui in a subclass
- how to implement custom metric in keras?
- torchrl: Using SyncDataCollector with a custom pytorch dqn
- Does peft train newly initialized weights?
- Do I have to write custom AutoModel transformers class in case "TypeError: NVEmbedModel.forward() got an unexpected keyword argument 'inputs_embeds'"
- Why RAG is slower than LLM?
- "RuntimeError: Numpy is not available" when using inverse_transform
- Pytorch RuntimeError: "host_softmax" not implemented for 'torch.cuda.LongTensor'
- AMD ROCm with Pytorch on Navi10 (RX 5700 / RX 5700 XT)
- Can we use multiple loss functions in same layer?
- How do I update pixelClassificationLayer() to a custom loss function?
- Neuralnet RMSE is 10x bigger than linear model's RMSE on test data set
- Back Propagation in Convolutional Neural Networks and how to update filters
- Face alignment megaface
- autoencoder.fit() raises 'KeyError: 'Exception encountered when calling Functional.call()'
- When to use numpy.random.randn(...) and when numpy.random.rand(...)?
- What is freezing/unfreezing a layer in neural networks?
- How can I use a pre-trained neural network with grayscale images?
- PyTorch RuntimeError: device >= 0 && device < num_gpus INTERNAL ASSERT FAILED
- How do I initialize weights in PyTorch?
- Does one convolutional filter always have different coefficients for each of the channels of the previous layer?
- Obtain the output of intermediate layer (Functional API) and use it in SubClassed API
- Optuna Hyperband Algorithm Not Following Expected Model Training Scheme
- Broadcasting multiple versions of X_data that pair with the same y_data
- How to make TensorFlow use 100% of GPU?
- Contrastive Loss from Scratch