stochastic graident for deep learning
I am reading about deep learning concept stochastic gradient. Here in below snap shot I am not understanding what does statement "The general problem with taking a significant step in this direction, however, is that the gradient could be changing under our feet as we move!" We demonstrate this simple fact in following figure. I am not able to how to interpret this figure. Kindly explain
We want to reduce error between the predicted value and the actual value. Consider the actual and predicted values as a point in 2D. You should move the point of predicted value as close as the point of actual value. To move the point you need a direction and SGD provides it.
Look at the image, C, the center of contours is the actual value, and P1 is the first predicted value and the SGD (blue arrow) shows a direction that reduces the distance between P1 and C. If you start from P1 and you take a significant(big) step in the first arrow direction, you will end at P2 which is far from C. However, if you take small steps(the blue dots), and in each step you move based on new SGD direction (the blue arrows in each point) you will get to a point close to C.
Big steps make you fluctuate around the actual value, also too small steps take too long to get to the actual value. Most of the time, we use big steps in start of learning process then make it small and small.
- h2o predict error: Test/Validation dataset has no columns in common with the training set
- Uniformity of color and texture in image
- Multi dimensional inputs in pytorch Linear method?
- What is the role of "Flatten" in Keras?
- Tensorflow keras error AttributeError: 'tuple' object has no attribute 'lower'
- I need to write complex-value neural network in tensorflow but I get an error
- Error in keras - name 'Dense' is not defined
- Issue setting up SciKeras model
- Keras multioutput custom loss with intermediate layers output
- Facial Expression Recognition Data Preparation for CNN
- PyTorch "Caught IndexError in DataLoader worker process 0", "IndexError: too many indices for array"
- What does "unsqueeze" do in Pytorch?
- How to solve strange cuda error in PyTorch?
- How to concatenate a tensor to a keras layer along batch (without specifying batch size)?
- How do I save a trained model in PyTorch?
- Text Image Binary Classifier without deep learning or tesseract
- Instant accuracy increase while dice increasing slowly (U-Net)
- What is the problem with my implementation of the cross-entropy function?
- `CrossEntropyLoss()` in PyTorch
- How to print the "actual" learning rate in Adadelta in pytorch
- How to slice Torch images as numpy images
- RuntimeError: 0D or 1D target tensor expected, multi-target not supported I was training a deep learning model but I am getting this issue
- Why is ReLU a non-linear activation function?
- What is FLOPS in field of deep learning?
- Does tf.math.reduce_max allows gradient flow like torch.max?
- Keras model not predicting values in the Test set
- Avoiding vanishing gradient in deep neural networks
- How can I run python package using Google Collab?
- What is the best choice for an activation function in case of small sized neural networks
- How to get around in place operation error if index leaf variable for gradient update?