multivariate function minimum approximation using gradient descent

Question

How can we approximate the minimum of a function like y = 2x1+x2x2+4 with gradient descent in the error backpropagation? if we consider initial values for x1 and x2 are zero and step width of 0.5.

Answer 1

In general, you can solve this kind of problem from scratch using the autograd package to compute gradient of y(x1, x2). Here is an example:

import autograd

# define your function to mimimize
def y(x1, x2):
    return 2*x1 + x2*x2 + 4

# get analytical gradients of y w.r.t the variables
dy_dx1 = autograd.grad(y, 0)
dy_dx2 = autograd.grad(y, 1)

# define starting values, step size
x1, x2 = 0.0, 0.0
step_size = 0.5
num_iterations = 100

ys = []
for iteration in range(num_iterations):

    # record value
    y_value = y(x1, x2)
    ys.append(y_value)
    print(f'at iteration {iteration}, y({x1:.1f}, {x2:.1f}) = {y_value}')

    # compute gradients
    der_x1 = dy_dx1(x1, x2)
    der_x2 = dy_dx2(x1, x2)

    # update variables to minimize
    x1 -= step_size * der_x1
    x2 -= step_size * der_x2

Note that in your case, computing the gradient analytically is straightforward a well. Also the minimum of this function will be -∞ as x1 → -∞ so the result of this kind of gradient descent might give unhelpful results.