Normal multiplication combines growth rates: “3 x 4″ can mean “Take your 3x growth and make it 4x larger (i.e., 12x)”. Complex multiplication lets us combine rotations. Integrals let us do piece-by-piece multiplication.

A vector is “growth in a direction”. The dot product lets us apply the directional growth of one vector to another: the result is how much we went along the original path (positive progress, negative, or zero).

Today let’s build our intuition for how the dot product works.

Getting the Formula Out of the Way

You’ve seen the dot product equation everywhere:

And also the justification: “Well Billy, the Law of Cosines (you remember that, don’t you?) says the following calculations are the same, so they are.” Not good enough — it doesn’t click! Beyond the computation, what does it mean?

The goal is to apply one vector to another. Each computation examines this from a rectangular perspective (x- and y-coordinates) or a polar one (magnitudes and angles). The “blah = foo” equation above really means “Here’s two equivalent ways to ‘directionally multiply’ vectors”.

(Similarly, we can show that Euler’s formula (e^ix = cos(x) + i*sin(x)) is true because the Taylor series is the same on both sides. Accurate but unsatisfying! Instead, see how both sides can describe the same motion.)

Seeing Numbers as vectors

Let’s start simple, and see 3 x 4 as a dot product:

The number 3 is “directional growth” in a single dimension (x-axis, let’s say), and 4 is “directional growth” in that same direction. 3 x 4 = 12 means 12x growth in that single dimension. Ok.

Now, suppose each number refers to a different dimension? Suppose 3 means “triple your bananas” (sigh… or “x-axis”) and 4 means “quadruple your oranges” (y-axis). They’re not the same “type” of number: what happens when we apply growth (take the dot product) in the (bananas, oranges) universe?

• (3,0) is “Triple your bananas, destroy your oranges”
• (0,4) is “Destroy your bananas, quadruple your oranges”

Applying (0,4) to (3,0) means “Destroy your banana growth, quadruple your orange growth”. But (3, 0) had no orange growth to begin with, so the net result is 0 (“Destroy all your fruit, buddy”).

See how we’re “applying” and not adding. With addition, we sort of smush the items together: (3,0) + (0, 4) = (3, 4) [a vector which triples your oranges and quadruples your bananas].

“Application” is different. We’re mutating the original vector according to the rules in the second. And the rules are “Destroy your banana growth rate, and triple your orange growth rate“. And, sadly, this leaves us with nothing.

The final result of this process can be:

• zero: we don’t have any growth in the original direction
• positive number: we have some growth in the original direction
• negative number: we have negative (reverse) growth in the original direction

Understanding the Calculation

“Applying vectors” is still a bit abstract. I think “How much energy/push is one vector giving to the other?”. Here’s how I visualize it:

Rectangular Coordinates: Component-by-component overlap

Like multiplying complex numbers, see how each x- and y-component interacts:

We list out all four combinations (x-x, y-x, x-y, y-y). Since the x- and y-coordinates don’t affect each other (like holding a bucket sideways under a waterfall — nothing falls in), the total energy absorbtion is absorbtion(x) + absorbtion(y):

Polar coordinates: Projection

The word “projection” is so sterile: I prefer “along the path”. How much energy is actually going in our original direction?

Here’s one way to see it:

Take two vectors, a and b. Rotate our coordinates so b is horizontal: it becomes (|b|, 0), and everything is on this new x-axis. What’s the dot product now? (It shouldn’t change just because we tilted our head).

Well, vector a has new coordinates (a1, a2), and we get:

a1 is really “What is the x-coordinate of a, assuming b is the x-axis?”. That is |a|cos(θ), aka the “projection”:

Analogies for the Dot Product

The common interpretation is “geometric projection”, but it’s so sterile. Here’s some analogies that click for me:

Energy Absorbtion

One vector are solar rays, the other is where the solar panel is pointing (yes, yes, the normal vector). Larger numbers mean stronger rays or a larger panel. How much energy is absorbed?

• Energy = Overlap in direction * Strength of rays * Size of panel
• 

If you hold your panel sideways to the sun, no rays hit (cos(θ) = 0).

Photo credit

But… but… solar rays are leaving the sun, and the panel is facing the sun, and the dot product is negative when vectors are opposed! Take a deep breath, and remember the goal is to embrace the analogy (besides, physicists lose track of negative signs all the time).

Mario-Kart Speed Boost

In Mario Kart, there are “boost pads” on the ground that increase your speed (Never played? I’m sorry.)

Photo source

Imagine the red vector is your speed (x and y direction), and the blue vector is the orientation of the boost pad (x and y direction). Larger numbers are more power.

How much boost will you get? For the analogy, imagine the pad multiplies your speed:

• If you come in going 0, you’ll get nothing
• If you cross the pad perpendicularly, you’ll get 0 [just like the banana obliteration, it will give you 0x boost in the perpendicular direction]

But, if we have some overlap, our x-speed will get an x-boost, and our y-speed gets a y-boost:

Neat, eh? Another way to see it: your incoming speed is |a|, and the max boost is |b|. The amount of boost you actually get (for being lined up with it) is cos(θ), for the total |a||b|cos(θ).

Physics Physics Physics

The dot product appears all over physics: some field (electric, gravitational) is pulling on some particle. We’d love to multiply, and we could if everything were lined up. But that’s never the case, so we take the dot product to account for potential differences in direction.

It’s all a useful generalization: Integrals are “multiplication, taking changes into account” and the dot product is “multiplication, taking direction into account”.

And what if your direction is changing? Why, take the integral of the dot product, of course!

Onward and Upward

Don’t settle for “Dot product is the geometric projection, justified by the law of cosines”. Find the analogies that click for you! Happy math.

Starting with

means “A 13-inch pizza equals a 13-inch pizza”. Sure. But we can trade an inch and get:

Huh? A 12-inch pizza and a 5-inch pizza equal a 13-inch pizza?

The math works (144 + 25 = 169) but, but… we gave up an inch and got a five-inch pizza!

Let’s understand why the tradeoff happens, and how to use it.

Explanation 1: Shaving the Square

A key insight: Bigger numbers are harder to square.

Imagine laying tiles on a porch — as your porch grows, the outer layer needs more tiles. Trimming a 13×13 porch to 12×12 frees up 25 tiles, which is enough to make a new 5×5 porch!

I call this “shaving the square”. Trimming 1 unit from the outside of a large square has more “shavings” which can contribute to a smaller one (trimming an inch from a giant fro can make a sweater for an infant). As we continue to trim, the benefit diminishes because our starting point is smaller and smaller.

Explanation 2: Sliding the Chopstick

A second insight: Slide a little, pivot a lot.

Imagine a chopstick wedged in a corner: the length is fixed, and the ends of the chopstick must touch a wall. What’re the options?

Well, laying on a single wall means 100% for one side (like saying 13² + 0² = 13²). Not that interesting.

By sliding the chopstick (from 13 to 12) we can swing it out by 5 on the other wall!

You need to try it — a small slide gives a giant pivot. As we keep sliding, the tradeoff (How much pivot do we get?) changes.

So What’s the Tradeoff?

Time to see how the a/b tradeoff works. First, let’s use grid coordinates: x & y (horizontal and vertical). Given a fixed distance (13 units, let’s say), our options lay on the circle where x² + y² = 13²:

A few points:

• Each possibility is the same distance, but has a different ratio of x to y (100% x, 100% y, or a mix like (12,5))
• We can only move to neighboring points on the circle (options at the same distance)
• The tradeoff we face is how much “x” we get for “y” when moving to a neighbor. If we’re at (0, 13) we could move to (5, 12). This trades 1 y for 5 x’s.

This is the “chunky” tradeoff where we’re using an entire unit at a time. What about .5 units? .01?

Enter the tangent! The tangent line shows the trajectory of our current path, the direction to our neighbor. We follow the tangent for a tiny, microscopic amount to get our next neighbor. The tangent is an approximation — it’s not pointing exactly at our nearest neighbor, but it’s pretty close.

The tangent shows the tradeoff you are about to make.

What’s the actual amount? Any point (x,y) has a slope of y/x, and a tangent line with slope -x/y, so the tradeoff is…getting confused yet?

Less mindless algebra, more intuition:

• Circles have a tangent line perpendicular to the current point
• If you’re at (5,12) then tangent slope is some ratio of 5 and 12
• Remember “shaving the square”: you get a better deal in the direction of the smaller coordinate (increasing a large square is tough).
• So, at (5, 12) you’re “heavy on the y” and the trade will favor improving your x: it should be “trade 5 y’s for 12 x’s”. And why not the other way? It doesn’t make sense that the more y you have, the easier it is to get y! That’d spiral off into exponential growth, not a circle.
• Lastly, we can’t trade an entire chunk of 5 y’s! The tangent is about our nearest neighbor. We have a trade of 12/5 or 2.4 to 1. Our next, tiny movement will be at this ratio (and then we’ll be at a new point, with a new tangent).

General principle: Our neighbors are on a circle, which encourages balance. You get a better deal in the direction of the smaller coordinate: at (x,y) the tradeoff is y:x.

Optimizing The Tradeoff

Now we know the tradeoff for any point (x,y) — let’s optimize!

In a boring scenario, we get paid based on pure distance, so every point (or direction to move) is the same.

The exciting scenario: our (x,y) position is an input into some other function which gives us a return! Now we want to maximize that function.

Here’s a scenario: Popeye throws cars for cash. He lines up spectators on fences running North and East. The spectators must look straight ahead (they’re in neck braces, due to earlier events) but will pay Popeye if they see a car pass in front of them.

Maximizing Even Payouts

Suppose each spectator offers $1 if they see the car (Payout (x,y) = x + y). Where to throw?

First, assume Popeye has finite energy — he can throw the car 13 meters. Now let’s start somewhere: throwing the car pure North (0, 13):

P(0,13) = 0 + 13 = $13

Ok. What if he threw it slightly East? To (5, 12) let’s say?

P(5,12) = 5 + 12 = $17

Clearly better. This should make sense: at (0,13) the tradeoff is great to get more East. We can give up 1 North and get a whopping 5 East, a “profit” of $4 if we do the trade. We should keep trading as long as it’s profitable — as long as we’re out of balance, the circle will reward us for boosting the smaller side. Following a 45 degree angle for 13 units is the ideal:

P(13 * 1/sqrt(2), 13 * 1/sqrt(2)) = P(13 * .707, 13 * .707) = 9.2 + 9.2 = $18.4

Neat. A 45-degree throw hits 70.7% of the possible spectators for each side.

Psst. Confused about how we got .707? No problem. Taking sides of 1 and 1 means the hypotenuse is 2:

But we want a trajectory of length 1, so we can scale it up easily (multiply by 13). So we divide both sides by 2:

To get the new “a” and “b” back out, convert 2 into “sqrt(2)^2″:

General Technique: Finding the Best Direction

We stumbled upon the way to find the best return:

• Pick any starting point / direction
• Tweak it: if our return improves, keep the new choice (it’s profitable)
• Keep tweaking until our return is no longer profitable

In math slang, this is “finding the local maximum”. In economics slang, it’s finding the point of “zero marginal returns”. Popeye calls it Squeezing the Spinach.

Maximizing Uneven Returns

Now suppose the Northern spectators offer $2 (Eastern stay at $1), so P(x,y) = x + 2*y. Should we throw it 100% North?

P(0, 13) = 0 + 2*13 = $26

Not bad. But what about 45 degrees again?

P(9.2, 9.2) = 9.2 + 2*9.2 = $27.6

Interesting — 45 degrees is still better! But… I think we went too far! Shouldn’t we favor North since it pays more?

Yep. Let’s remember how to Squeeze the Spinach (maximize our returns): start with North and change until it’s not profitable:

• The payout function means 1 North = 2 Easts (North pays $2, so 1 unit North = 2 units East)
• Trades are profitable if we can beat 1 North for 2 Easts (1 North for 3 Easts, for example, would profit $1)

So… where are trades better than 1 North for 2 Easts? In the Northern section, where the circle rewards us by throwing Easts at us (“Please, please go East… I’ll give you a bunch if you give up a little North”).

Remember how circles are about x/y, x & y, x:y, etc.? Well, we have the numbers 1 and 2. (2,1) is in the East section. We want (1,2). Why? At (1,2) we have reached the perfect 1 North = 2 East tradeoff.

Following the direction (1,2) for 13 units is:

P(13 * 1/sqrt(5), 13 * 2/sqrt(5)) = P(5.81, 11.62) = 5.81 + 2*11.62 = $29.05

Tada! Over 29 smackeroos because we maximized our return.

The Gradient Principle

We can supercharge this result:

To maximize return, go in each direction proportional to its payoff.

If North pays 2:1 compared to East, your trajectory should favor North by 2:1. In mathier terms:

• Payoff(x,y) = ax + by
• Best trajectory = (a, b) [in our case, (East, North) => (1, 2)]

And this works in multiple dimensions! Given 3 dimensions, go in a direction (Payoff(x), Payoff(y), Payoff(z)). Vector calculus fans, this is why the gradient is in the direction of greatest increase.

The gradient for F(x,y,z) is

And each partial derivative (dF/dx) is the payoff for moving in that direction.

But does it all balance? Suppose x pays 3, y pays 4, and z pays 5 (at the current position). The 2-dimensional tradeoff trajectories are:

Now for the magic: the combined trajectory

satisfies all 3 requirements! On the x-z plane, x doesn’t care about y — as long as the ratio to z is (3 , ?, 5) you’re getting the best tradeoff from the x-z perspective. The pairs are:

• (3, ?, 5)
• (?, 4, 5)
• (3, 4, ?)

You don’t need a sudoku master to see (3, 4, 5) satisfies all those proportions.

Still not convinced? Imagine the payoff for y was zero. We don’t want to waste energy in our trajectory (3, ?, 5) in a useless direction. But that can’t happen, because the y-z tradeoff will be (?, 0, 5) and the x-y tradeoff will be (3, 0, ?). The x-z tradeoff lets y-z and x-y “figure out” what y should be, which is 0.

Questions I Had That You Might Have Too

Q: I still don’t get why this works at all. Somehow 50% in x and 50% in y leads to .7 + .7 = 1.4?

It’s a deep question about why space behaves like this. I was going crazy staring at chopsticks on a wall.

Here’s my answer: distance is distance. 13 units is 13 units. But in some situations we are “measuring our coordinates” (what are the values of x & y) and not the distance itself.

Coordinates with perpendicular axes are very inefficient, especially for diagonal motion (i.e., you are measuring the sides of the triangle, not the hypotenuse). When .707^2 + .707^2 = 1, it’s a measure how how “inefficient” our x & y coordinates are being. We used 70% of each coordinate to represent an object that could have been 100% on one (i.e, if we used polar coordinates).

Q: I have an offshore investment with 200% return, and an onshore one with 5% return. I have $1000 to spend — should I split my money?

Heavens, no! Remember, this principle is about distance measurements on a grid with the idea that 50% in x and 50% in y covers “more ground” than 100% in x. In investing 1) money is not on a grid and 2) there’s no distance bonus. Putting half your money in each is plain old 0.5 + 0.5 = 1.0. Giving up $1 of the offshore investment gives you $1 for the onshore one.

Put all your money in the best investment.

Q: So all this stuff is useless?

Heavens, no! Ask yourself: am I measuring distance on a coordinate system?

Many things are measured in terms of x-y coordinates (physical phenomena, etc.) and do have the Pythagorean distance tradeoff.

But not every graph is the same. Graphs that aren’t about distance (like “Money vs. Time”) do not get any boost from the Pythagorean theorem. This confused me for a long time: the Pythagorean Theorem works for coordinate distance!

Final Thoughts

The Pythagorean Theorem is so versatile — it’s not about triangles, it covers the nature of distance. I seem to find some new realization when I study it. Really grokking it will help you everywhere, from geometry to vector calculus.

Happy math.

A vector field is usually the source of the circulation. If you had a paper boat in a whirlpool, the circulation would be the amount of force that pushed it along as it went in a circle. The more circulation, the more pushing force you have.

Curl is simply the circulation per unit area, circulation density, or rate of rotation (amount of twisting at a single point). Imagine shrinking your whirlpool down smaller and smaller while keeping the force the same: you'll have a lot of power in a small area, so will have a large curl. If you widen the whirlpool while keeping the force the same as before, then you'll have a smaller curl. And of course, zero circulation means zero curl.

Intuition

Circulation is the amount of "pushing" force along a path. Curl is the amount of pushing, twisting, or turning force when you shrink the path down to a single point. Let's use water as an example.

Suppose we have a flow of water and we want to determine if it has curl or not: is there any twisting or pushing force? To test this, we put a paddle wheel into the water and notice if it turns (the paddle is vertical, sticking out of the water like a revolving door -- not like a paddlewheel boat). If the paddle does turn, it means this field has curl at that point. If it doesn't turn, then there's no curl.

What does it really mean if the paddle turns? Well, it means the water is pushing harder on one side than the other, making it twist. The larger the difference, the more forceful the twist and the bigger the curl. Also, a turning paddle wheel indicates that the field is "uneven" and not symmetric; if the field were even, then it would push on all sides equally and the paddle wouldn't turn at all.

The fact that there is a "twist" means the field is not conservative (this has nothing to do with its political views).

A conservative field is "fair" in the sense that work needed to move from point A to point B, along any path, is the same. For example, consider a river: its field is conservative. Sure, you can get a free ride downstream, but then you have to do work to get back to your starting point. Or, you can do work to move upstream, and get a free ride back. Either way, the amount of work you "put in" is the same as what you get back.

However, in a field with curl (like a whirlpool), you can get a free ride by moving in the direction of the twist. In a whirlpool, you can get a free trip by moving with the current in a circle. If you fight the current and go the wrong way, you have to use energy with no free ride at all.

Conservative fields have zero curl: there are no free twists to push you along. Alternatively, if a field has curl, it is not conservative.

Gravity is another example of a conservative field. Technically, if you lift a rock and then let it fall, the energy you get from falling is the same as what you put in to lift the rock. Theoretically speaking, no energy was gained or lost in this transaction.

Additional Details

To be technical, curl is a vector, which means it has a both a magnitude and a direction. The magnitude is simply the amount of twisting force at a point.

The direction is a little more tricky: it's the orientation of the axis of your paddlewheel in order to get maximum rotation. In other words, it is the direction which will give you the most "free work" from the field. Imagine putting your paddlewheel sideways in the whirlpool - it wouldn't turn at all. If you put it in the proper direction, it begins turning.

But wait a minute -- aren't there two directions to get a twisting motion? Couldn't you just turn the paddlewheel "upside down" and get the maximum curl as well?

Yep, you're right. By convention alone, if the paddle wheel is rotating counterclockwise, its curl vector points out of the page. This is a type of right-hand rule: make a fist with your right hand and stick out your thumb. If the circulation/pushing force follows the twisting of your fingers (counterclockwise), then the curl vector will be in the direction of your thumb.

Mathematics

Circulation is the integral of a vector field along a path - you are adding how much the field "pushes" you along a path.

How do we find this? Well, we should expect some type of dot product, because we want to know the amount that one vector (the force) is pushing in the direction of another (the path). So, the two vectors we need are (1) the path vector and (2) the field vector at every point along the path.

If we have a function that defines the position at any time, \( F(t) \), we can take the time derivative to get the velocity at that position.

The velocity vector is always in the direction of movement -- if you are moving from A to B, the velocity vector will be an arrow from A to B, i.e. your change in position or your direction of movement. So, we can use the velocity to get our direction.

It's important to understand why we aren't using the position vector itself -- it tells us where we are, but not where we're going. We need to know our direction to see how much "push" we are getting: Knowing your position in a river isn't important -- are you going upstream or downstream, and at what angle?

The force vector (2) is defined by the field we are in. No derivatives or other changes are necessary -- every point in the field has some force acting on it.

So, our formula for circulation is:

Force at position \(r = F(r) \)
Direction at position \(r = dr \)
Total pushing force = \(Circulation = \int F(r) \cdot dr \)

Remember, velocity is simply the derivative of position \(r\), so \(dr\) is a vector giving us our direction. We integrate along the entire path and use the dot product to see how much pushing force is applied. We then sum up these "pushes" to get the total circulation.

Since curl is the circulation per unit area, we can take the circulation for a small area (letting the area shrink to 0). However, since curl is a vector, we need to give it a direction -- the direction is normal (perpendicular) to the surface with the vector field. The magnitude is the same as before: circulation/area.

Recall that by convention (a bunch of people agreeing), counterclockwise circulation will give a curl pointing out of the page. Using these facts, we can create the formula for curl:

Where \(S\) is the surface we are considering; the direction of the curl is the normal to the surface.

You'll see fancier equations for curl where the surface shrinks to zero (such as in wikipedia), but recognize the basic intuition -- curl is the circulation per unit area.

Parting Thoughts

You'll often see curl of a field \(F\) written like this:

which is a cross-product of the gradient and the field \(F\). This has to do with how curl is actually computed, which will be material for another article (and probably in your textbook already -- see wikipedia for details).

If I have been successful, you should understand intuitively what circulation and curl mean, and how we got the formulae above. They spring up naturally from our definition of circulation as "pushing force along a path" and curl as "pushing force/area".

Math should be a tool for clearly stating what we already know. Understand the intuition and then tackle the complicated formulas. Happy math.

PS. Have some fun and check out this video of a famous whirlpool. Imagine the circulation on this (go on, imagine):

• Points in the direction of greatest increase of a function (intuition on why)
• Is zero at a local maximum or local minimum (because there is no single direction of increase)

The term gradient (grad) typically refers to the derivative of vector functions, or functions of more than one variable. Yes, you can say a line has a gradient (its slope), but using the term gradient for single-variable functions is unnecessarily confusing. Keep it simple.

“Gradient” can refer to gradual changes of color, but we’ll stick to the math definition if that’s ok with you. You’ll see the meanings are related.

Properties of the Gradient

Now that we know the gradient is the derivative of a multi-variable function, let’s derive some properties.

The regular, plain-old derivative gives us the rate of change of a single variable, usually x. For example, dF/dx tells us how much the function F changes for a change in x. But if a function takes multiple variables, such as x and y, it will have multiple derivatives: the value of the function will change when we “wiggle” x (dF/dx) and when we wiggle y (dF/dy).

We can represent these multiple rates of change in a vector, with one component for each derivative. Thus, a function that takes 3 variables will have a gradient with 3 components:

has one variable and a single derivative:

has three variables and three derivatives:

The gradient of a multi-variable function has a component for each direction.

And just like the regular derivative, the gradient points in the direction of greatest increase. However, now that we have multiple directions to consider (x, y and z), the direction of greatest increase is no longer simply “forward” or “backward” along the x-axis, like it is with functions of a single variable.

If we have two variables, then our 2-component gradient can specify any direction on a plane. Likewise, with 3 variables, the gradient can specify and direction in 3D space to move to increase our function.

A Twisted Example

I’m a big fan of examples to help solidify an explanation. Suppose we have a magical oven, with coordinates written on it and a special display screen:

We can type any 3 coordinates (like “3,5,2″) and the display shows us the gradient of the temperature at that point.

The microwave also comes with a convenient clock. Unfortunately, the clock comes at a price — the temperature inside the microwave varies drastically from location to location. But this was well worth it: we really wanted that clock.

With me so far? We type in any coordinate, and the microwave spits out the gradient at that location.

Be careful not to confuse the coordinates and the gradient. The coordinates are the current location, measured on the x-y-z axis. The gradient is a direction to move from our current location, such as move up, down, left or right.

Now suppose we are in need of psychiatric help and put the Pillsbury Dough Boy inside the oven because we think he would taste good. He’s made of cookie dough, right? We place him in a random location inside the oven, and our goal is to cook him as fast as possible. The gradient can help!

The gradient at any location points in the direction of greatest increase of a function. In this case, our function measures temperature. So, the gradient tells us which direction to move the doughboy to get him to a location with a higher temperature, to cook him even faster. Remember that the gradient does not give us the coordinates of where to go; it gives us the direction to move to increase our temperature.

Thus, we would start at a random point like (3,5,2) and check the gradient. In this case, the gradient there is (3,4,5). Now, we wouldn’t actually move an entire 3 units to the right, 4 units back, and 5 units up. The gradient is just a direction, so we’d follow this trajectory for a tiny bit, and then check the gradient again.

We get to a new point, pretty close to our original, which has its own gradient. This new gradient is the new best direction to follow. We’d keep repeating this process: move a bit in the gradient direction, check the gradient, and move a bit in the new gradient direction. Every time we nudged along and follow the gradient, we’d get to a warmer and warmer location.

Eventually, we’d get to the hottest part of the oven and that’s where we’d stay, about to enjoy our fresh cookies.

Don’t eat that cookie!

But before you eat those cookies, let’s make some observations about the gradient. That’s more fun, right?

First, when we reach the hottest point in the oven, what is the gradient there?

Zero. Nada. Zilch. Why? Well, once you are at the maximum location, there is no direction of greatest increase. Any direction you follow will lead to a decrease in temperature. It’s like being at the top of a mountain: any direction you move is downhill. A zero gradient tells you to stay put – you are at the max of the function, and can’t do better.

But what if there are two nearby maximums, like two mountains next to each other? You could be at the top of one mountain, but have a bigger peak next to you. In order to get to the highest point, you have to go downhill first.

Ah, now we are venturing into the not-so-pretty underbelly of the gradient. Finding the maximum in regular (single variable) functions means we find all the places where the derivative is zero: there is no direction of greatest increase. If you recall, the regular derivative will point to local minimums and maximums, and the absolute max/min must be tested from these candidate locations.

The same principle applies to the gradient, a generalization of the derivative. You must find multiple locations where the gradient is zero — you’ll have to test these points to see which one is the global maximum. Again, the top of each hill has a zero gradient — you need to compare the height at each to see which one is higher. Now that we have cleared that up, go enjoy your cookie.

Mathematics

We know the definition of the gradient: a derivative for each variable of a function. The gradient symbol is usually an upside-down delta, and called “del” (this makes a bit of sense – delta indicates change in one variable, and the gradient is the change in for all variables). Taking our group of 3 derivatives above

Notice how the x-component of the gradient is the partial derivative with respect to x (similar for y and z). For a one variable function, there is no y-component at all, so the gradient reduces to the derivative.

Also, notice how the gradient can itself be a function!

If we want to find the direction to move to increase our function the fastest, we plug in our current coordinates (such as 3,4,5) into the equation and get:

So, this new vector (1, 8, 75) would be the direction we’d move in to increase the value of our function. In this case, our x-component doesn’t add much to the value of the function: the partial derivative is always 1.

Obvious applications of the gradient are finding the max/min of multivariable functions. Another less obvious but related application is finding the maximum of a constrained function: a function whose x and y values have to lie in a certain domain, i.e. find the maximum of all points constrained to lie along a circle. Solving this calls for my boy Lagrange, but all in due time, all in due time: enjoy the gradient for now.

The key insight is to recognize the gradient as the generalization of the derivative. The gradient points to the maximum of the function; follow the gradient, and you will reach the local maximum.

Questions

Why is the gradient perpendicular to lines of equal potential?

Lines of equal potential (“equipotential”) are the points with the same energy (or value for f(x,y,z)). In the simplest case, a circle represents all items the same distance from the center.

The gradient represents the direction of greatest change. If it had any component along the line of equipotential, then that energy would be wasted (as it’s moving closer to a point at the same energy). When the gradient is perpendicular to the equipotential points, it is moving as far from them as possible (this article explains why the gradient is the direction of greatest increase — it’s the direction that maximizes the varying tradeoffs inside a circle).