Vector Calculus: Understanding the Gradient

The gradient is a fancy word for derivative, or the rate of change of a function. It’s a vector (a direction to move) that

Points in the direction of greatest increase of a function
Is zero at a local maximum or local minimum (because there is no single direction of increase)

The term gradient (grad) typically refers to the derivative of vector functions, or functions of more than one variable. Yes, you can say a line has a gradient (its slope), but using the term gradient for single-variable functions is unnecessarily confusing. Keep it simple.

“Gradient” can refer to gradual changes of color, but we’ll stick to the math definition if that’s ok with you. You’ll see the meanings are related.

Properties of the Gradient

Now that we know the gradient is the derivative of a multi-variable function, let’s derive some properties.

The regular, plain-old derivative gives us the rate of change of a single variable, usually x. For example, dF/dx tells us how much the function F changes for a change in x. But if a function takes multiple variables, such as x and y, it will have multiple derivatives: the value of the function will change when we “wiggle” x (dF/dx) and when we wiggle y (dF/dy).

We can represent these multiple rates of change in a vector, with one component for each derivative. Thus, a function that takes 3 variables will have a gradient with 3 components:

$\displaystyle{F(x)}$ has one variable and a single derivative: $\displaystyle{\frac{dF}{dx}}$

$\displaystyle{F(x,y,z)}$ has three variables and three derivatives: $\displaystyle{(\frac{dF}{dx},\frac{dF}{dy},\frac{dF}{dz})}$

The gradient of a multi-variable function has a component for each direction.

And just like the regular derivative, the gradient points in the direction of greatest increase (trust me on this, I’ll create a derivation a bit later ). However, now that we have multiple directions to consider (x, y and z), the direction of greatest increase is no longer simply “forward” or “backward” along the x-axis, like it is with functions of a single variable.

If we have two variables, then our 2-component gradient can specify any direction on a plane. Likewise, with 3 variables, the gradient can specify and direction in 3D space to move to increase our function.

A Twisted Example

I’m a big fan of examples to help solidify an explanation. Suppose we have a magical oven, with coordinates written on it and a special display screen:

We can type any 3 coordinates (like “3,5,2″) and the display shows us the gradient of the temperature at that point.

The microwave also comes with a convenient clock. Unfortunately, the clock comes at a price — the temperature inside the microwave varies drastically from location to location. But this was well worth it: we really wanted that clock.

With me so far? We type in any coordinate, and the microwave spits out the gradient at that location.

Be careful not to confuse the coordinates and the gradient. The coordinates are the current location, measured on the x-y-z axis. The gradient is a direction to move from our current location, such as move up, down, left or right.

Now suppose we are in need of psychiatric help and put the Pillsbury Dough Boy inside the oven because we think he would taste good. He’s made of cookie dough, right? We place him in a random location inside the oven, and our goal is to cook him as fast as possible. The gradient can help!

The gradient at any location points in the direction of greatest increase of a function. In this case, our function measures temperature. So, the gradient tells us which direction to move the doughboy to get him to a location with a higher temperature, to cook him even faster. Remember that the gradient does not give us the coordinates of where to go; it gives us the direction to move to increase our temperature.

Thus, we would start at a random point like (3,5,2) and check the gradient. In this case, the gradient there is (3,4,5). Now, we wouldn’t actually move an entire 3 units to the right, 4 units back, and 5 units up. The gradient is just a direction, so we’d follow this trajectory for a tiny bit, and then check the gradient again.

We get to a new point, pretty close to our original, which has its own gradient. This new gradient is the new best direction to follow. We’d keep repeating this process: move a bit in the gradient direction, check the gradient, and move a bit in the new gradient direction. Every time we nudged along and follow the gradient, we’d get to a warmer and warmer location.

Eventually, we’d get to the hottest part of the oven and that’s where we’d stay, about to enjoy our fresh cookies.

Don’t eat that cookie!

But before you eat those cookies, let’s make some observations about the gradient. That’s more fun, right?

First, when we reach the hottest point in the oven, what is the gradient there?

Zero. Nada. Zilch. Why? Well, once you are at the maximum location, there is no direction of greatest increase. Any direction you follow will lead to a decrease in temperature. It’s like being at the top of a mountain: any direction you move is downhill. A zero gradient tells you to stay put - you are at the max of the function, and can’t do better.

But what if there are two nearby maximums, like two mountains next to each other? You could be at the top of one mountain, but have a bigger peak next to you. In order to get to the highest point, you have to go downhill first.

Ah, now we are venturing into the not-so-pretty underbelly of the gradient. Finding the maximum in regular (single variable) functions means we find all the places where the derivative is zero: there is no direction of greatest increase. If you recall, the regular derivative will point to local minimums and maximums, and the absolute max/min must be tested from these candidate locations.

The same principle applies to the gradient, a generalization of the derivative. You must find multiple locations where the gradient is zero — you’ll have to test these points to see which one is the global maximum. Again, the top of each hill has a zero gradient — you need to compare the height at each to see which one is higher. Now that we have cleared that up, go enjoy your cookie.

Mathematics

We know the definition of the gradient: a derivative for each variable of a function. The gradient symbol is usually an upside-down delta, and called “del” (this makes a bit of sense - delta indicates change in one variable, and the gradient is the change in for all variables). Taking our group of 3 derivatives above

$\displaystyle{grad F(x,y,z) = \nabla F(x,y,z) = (\frac{dF}{dx},\frac{dF}{dy},\frac{dF}{dz})}$

Notice how the x-component of the gradient is the partial derivative with respect to x (similar for y and z). For a one variable function, there is no y-component at all, so the gradient reduces to the derivative.

Also, notice how the gradient can itself be a function!

$\displaystyle{F(x,y,z) = x + y^2 + z^3 }$

$\displaystyle{\nabla F(x,y,z) = (1, 2y, 3z^2)}$

If we want to find the direction to move to increase our function the fastest, we plug in our current coordinates (such as 3,4,5) into the equation and get:

$\displaystyle{direction = (1, 2(4), 3(5)^2) = (1, 8, 75)}$

So, this new vector (1, 8, 75) would be the direction we’d move in to increase the value of our function. In this case, our x-component doesn’t add much to the value of the function: the partial derivative is always 1.

Obvious applications of the gradient are finding the max/min of multivariable functions. Another less obvious but related application is finding the maximum of a constrained function: a function whose x and y values have to lie in a certain domain, i.e. find the maximum of all points constrained to lie along a circle. Solving this calls for my boy Lagrange, but all in due time, all in due time: enjoy the gradient for now.

The key insight is to recognize the gradient as the generalization of the derivative. The gradient points to the maximum of the function; follow the gradient, and you will reach the local maximum.

Posted February 17, 2007, under Vector Calculus
Tags: derivative, function, gradient, Math, vector calculus
Related Posts:

Vector Calculus: Understanding Circulation and Curl

Explanations for everyone.

Rescaling the Pythagorean Theorem

A Gentle Introduction To Learning Calculus

Vector Calculus: Understanding Flux

Like it? Share on:

RSS or Email Subscription |

Printable Version

Tools of the trade:

Don't Make Me Think
(Web Design)

On Writing Well
(Writing Style)

MS Office 2007
(Charts & Diagrams)

27 Comments »

Comments

i like it… well explained.

me — February 22, 2007 @ 10:43 am
Super!!!

Jane — March 2, 2007 @ 10:00 am
You are the man! Nice work!

Chris — March 21, 2007 @ 8:01 pm
Thanks, glad it was helpful for you.

Kalid — April 1, 2007 @ 1:07 pm
i was always looking for conceptual and practical examples and yes i finally got.

gaurav — June 9, 2007 @ 2:18 am
Awsome!

Harry — June 10, 2007 @ 4:04 pm
well you made a good explanation, that even a not-so-smart guy gets it, but i think you missed the obvious -> WHY does gradient show the direction of the greatest increase.

I think that the principle of the gradient is quite easy, but understanding why does it work the way it does is a bit tricky and you should have focued on it more.

It would be interesting if you would somehow add it to this good article. Inspiration http://mathforum.org/library/drmath/view/68326.html

good luck !

Palo — August 15, 2007 @ 5:05 pm
Hi Palo, that’s a great point! I’ve been feeling a bit guilty, if you can imagine it, because I’ve lacked that explanation

I’m probably going to do a separate article on the reason *why* the gradient points in the direction of greatest increase — I have another explanation that it works well with. Thanks for the link and feedback!

Kalid — August 15, 2007 @ 5:51 pm
Your introduction is not quite correct:

You claim: “Points in the direction of greatest increase of a function”.

Why? It can also point in the direction of greatest decrease of a function.

A gradient is one or more directional derivatives. These derivatives are considered in a particular direction. In the case of single variable calculus, we generally talk about a directional derivative when we consider multiples of the x unit vector, i.e. k*(1,0). To consider the y unit vector, we deal with the partial derivatives with respect to y in a given direction. In three dimensions, the 3 partial derivatives form what we now call a ‘gradient’.

So in fact it is incorrect to call this a slope or anything else except to say that it describes the partial derivatives of a point in the direction of a given vector in space.

Does this make sense? Please visit my blog for some more interesting reading.

http://mathphile.blogspot.com/

John Gabriel — September 16, 2007 @ 6:47 am
Hi John, thanks for writing. You’re right, the formal definition of a gradient is a set of directional derivatives.

But when thinking about the intuitive meaning, I think it’s ok to consider the gradient as a vector that “points” in the direction of greatest increase (i.e. if you follow that direction your function will tend towards a local maximum).

Unless I’m mistaken, the gradient vector always points in the direction of greatest increase (greatest decrease would be in the opposite direction).

Kalid — September 19, 2007 @ 11:49 am
What I was saying is that it points either one way or the other, it is not restricted to the direction of greatest increase. As a simple example, consider what happens when you differentiate a parabola: You set the derivative equal to 0 and then you determine that it has either a maximum or a minimum at its turning point. It is not always a maximum just as it is not always a minimum. Think I have explained this correctly now.

John Gabriel — September 27, 2007 @ 7:21 pm
good john you have done a great job.

sqib — October 5, 2007 @ 3:41 pm
Hi John, thanks for the clarification. I’d still politely disagree and say that in general, the gradient points in the direction of greatest increase .

In the case of 2 dimensions, the gradient/slope only gives a forward or backward direction. A positive slope means travel “forward” and a negative slope means travel “backwards”.

Consider f(x) = x^2, a regular parobola. The gradient is zero at the minimum (x=0), and there is no *single* direction to go. At x = -1, the slope is negative, which means travel “backwards” (to x = -2) to increase your value. Similarly, at x = 1, you travel forward (to x = 2) to increase your value.

But, as you mention, strange things can happen when the derivative = 0. It can mean you are at a local maximum (no way to improve), or at a local minimum (no single direction to improve your position — forward or back will help). I consider the corner case of zero an exception to the general rule / intuition that the gradient is “the direction to follow” if you want to improve your function.

Kalid — October 5, 2007 @ 10:23 pm
Wonderful explanation!

Vidhya — October 10, 2007 @ 9:09 am
Thanks Vidhya, glad you liked it.

Kalid — October 10, 2007 @ 10:17 am
hi john keep it up you done a great job

bihazo — October 21, 2007 @ 6:21 am
Thanks a bunch! I didn’t think it could be this simple to find the maximum increase at a point, so I thought I’d look it up. Thanks to your great explaination, it turn out it was as easy as it seemed it should be. Great job! Thanks!

Travis

Travis — December 2, 2007 @ 5:53 pm
Awesome, glad it worked for you

Kalid — December 2, 2007 @ 7:27 pm
thanks!!!!

caitlyn — December 5, 2007 @ 11:09 pm
Hi Caitlyn, you’re welcome.

Kalid — December 6, 2007 @ 1:03 am
Thanks! The sadistic microwave example helped a lot.

Derek — December 10, 2007 @ 12:27 pm
Awesome, glad it was useful .

Kalid — December 10, 2007 @ 4:54 pm
Hello Kalid,
Did not read your reply for some
time. Am sorry you do not agree.

Let me give you an example:
Suppose we are dealing with pressure
and height in a certain ‘cubic’
area. Suppose that the middle of the
cube height is 0 meters. Also suppose
that we have a whirlpool generated in the
cube such that the pressure rate increases
as we go below the middle of the cube.
Anything below is negative height and anything above
is positive height. Now, as one rises
higher in the cube, the pressure decreases.
If we find the gradient, then according to
your definition (and many others’), then
the gradient vector for the rate of greatest
increase will point below the middle of the
cube, not above. But above the middle we
find the greatest ‘decrease’ in rate of pressure.

In this example, greatest increase points
downwards and greatest decrease upwards.

It would probably be better to define
gradient as a vector that points in a
direction of greatest increase or decrease.
It’s additive inverse will point in the
diretion of greatest decrease or increase
respectively. For most physical phenomena,
your definition would generally be true.
But what happens when you have an anomaly?
Make sense?

John Gabriel — December 11, 2007 @ 7:32 am
I do not believe I have the best answer to this question but like yourself, I am a believer in trying to find the best possible explanation. Once again, I like your website. Keep up the good work Kalid!

John Gabriel — December 11, 2007 @ 7:34 am
Okay, I think I have the best answer. If f is a real-valued function, then del(f) or gradient of f points to the greatest increase, whereas -del(f) points t0 the greatest decrease.

For once planet math has some decent information on this since I last checked:

http://planetmath.org/encyclopedia/Gradient.html

I do not endorse everything Planet Math publishes but this particular information appears to be correct. In any event, it clears up the previous confusion I think.

John Gabriel — January 16, 2008 @ 1:06 pm
Hi John, thanks for the comment! Yes, that’s an important distinction to make: the positive gradient is the greatest increase, and the negative gradient is the greatest decrease. Thanks for helping clarify .

Kalid — January 16, 2008 @ 7:11 pm
Thank you!

Jared — March 24, 2008 @ 6:42 pm

RSS feed for comments on this post. TrackBack URI

Have a question? Know an explanation that caused your own a-ha moment? Write about it here.