Search code examples
mathmachine-learninglinear-regressionrecursive-descent

What exactly does delta mean in the gradient descent algorithm?

As on the picture:

enter image description here

Could someone help me understand what exactly what delta means in the gradient descent algorithm?

Solution

  • The term is a derivative with respect to the theta 0.

    • Mark theta as coordinate on X-axis (let it be A)
    • Find corresponding coordinate on Y-axis (let it be B) so the point belongs to the function J
    • Draw tangent line to that function at the point (A, B)
    • The derivative is the slope of this tangent line.

    enter image description here

    The derivative is used to control two aspects of the cost function (J function) minimization:

    • direction - sign of the slope tells you in which direction you should move along the X-axis in order to converge J
    • rate - magnitude of the slope tells you how fast you should move