Policy Iteration vs Value Iteration
In reinforcement learning, I'm trying to understand the difference between policy iteration, and value iteration. There are some general answers to this out there, but I have two specific queries which I cannot find an answer to.
1) I have heard that policy iteration "works forwards", whereas value iteration "works backwards". What does this mean? I thought that both methods just take each state, then look at all the other states it can reach, and compute the value from this -- either by marginalising over the policy's action distribution (policy iteration) or by taking that argmax with regards to the action values (value iteration). So why is there any notion of the "direction" in which each method "moves"?
2) Policy iteration requires an iterative process during policy evaluation, to find the value function -- by However, value iteration just requires one step. Why is this different? Why does value iteration converge in just one step?
Thank you!
The answer provided by @Nick Walker is right and quite complete, however I would like to add a graphical explanation of the difference between Value iteration and Policy iteration, which maybe help to answer the second part of your question.
Both methods, PI and VI, follow the same working principle based in Generalized Policy Iteration. This basically means that they alternate between improving the policy (which requires knowing its value function), and computing the value function of the new, improved, policy.
At the end of this iterative process, both, the value and the policy, converge to the optimal.
However, it was notice that is not necessary to compute exactly the full value function, instead, one step is necessary to allow the convergence. In the following figure, (b) represents the operations performed by Policy Iteration, where the full value function is computed. While (d) shows how Value iteration works.
Obviously this representation of both methods are simplistic, but it highlights the difference between the key idea behind each algorithm.
- Is there a standard file naming convention for key-value pairs in filename?
- how to use ML Models in android application
- How to make a multifactor model in pROC?
- Sagemaker batch transformer with my own pre-trained model
- Can we import a python made ML model (.pkl) in rust?
- How to use OpenCV to do OCR and text detect and recognition
- Realworld parameter optimization
- What do the coefficients on correlated variables mean?
- Handling Class Imbalance in Multi-class Classification with Custom Loss Function
- Struggling to understand complete predictive model process in R
- How to allocate GPUs on AWS Free Tier?
- Open Source Neural Network Library
- How to make FeatureUnion return Dataframe
- What is the role of "Flatten" in Keras?
- Machine learning model predicts training labels themselves as result
- split an audio file into chunks, skip the chunks less than desired time duration, and predict emotion for the entire audio file
- Facing ValueError: Target is multiclass but average='binary'
- Detectron2 - Extract region features at a threshold for object detection
- Detectron2 Checkpoint not found
- Incomprehensible shape error with one of the inputs of my non-sequential keras model
- How to process requests from multiiple users using ML model and FastAPI?
- Alternative to device_map = "auto" in Huggingface Pretrained
- np.where: "ValueError: operands could not be broadcast together with shapes (38658637,) (9456,)"
- How to compute number of weights of CNN?
- How to find the connected instances from a minimum spanning trees model in R
- Can a neural network be trained while it changes in size?
- Keras-rl2 error Compability with Tensorflow
- Separate a ingredients/feature into separate columns that is marked with "0" or "1"
- How to conditionally assign values to tensor [masking for loss function]?
- Uniformity of color and texture in image