Calculus – BetterExplained https://betterexplained.com Math lessons that click Sat, 20 Aug 2022 03:09:17 +0000 en-US hourly 1 Intuitive Guide to Convolution https://betterexplained.com/articles/intuitive-convolution/ https://betterexplained.com/articles/intuitive-convolution/#disqus_thread Mon, 23 Nov 2020 19:08:46 +0000 https://betterexplained.com/?p=12383 Like making engineering students squirm? Have them explain convolution and (if you're barbarous) the convolution theorem. They'll mutter something about sliding windows as they try to escape through one.

Convolution is usually introduced with its formal definition:

\displaystyle{ (f * g )(t) = \int_{-\infty}^\infty f(\tau) g(t - \tau) d\tau }

Yikes. Let's start without calculus: Convolution is fancy multiplication.

Part 1: Hospital Analogy

Imagine you manage a hospital treating patients with a single disease. You have:

  • A treatment plan: [3] Every patient gets 3 units of the cure on their first day.
  • A list of patients: [1 2 3 4 5] Your patient count for the week (1 person Monday, 2 people on Tuesday, etc.).

Question: How much medicine do you use each day? Well, that's just a quick multiplication:

Plan  *  Patients       = Daily Usage
[3]   *  [1 2 3 4 5]    = [3 6 9 12 15]

Multiplying the plan by the patient list gives the usage for the upcoming days: [3 6 9 12 15]. Everyday multiplication (3 x 4) means using the plan with a single day of patients: [3] * [4] = [12].

Intuition For Convolution

Let's say the disease mutates and requires a multi-day treatment. You create a new plan: Plan: [3 2 1]

That means 3 units of the cure on the first day, 2 on the second, and 1 on the third. Ok. Given the same patient schedule [1 2 3 4 5], what's our medicine usage each day?

Uh... shoot. It's not a quick multiplication:

  • On Monday, 1 patient comes in. It's her first day, so she gets 3 units.
  • On Tuesday, the Monday gal gets 2 units (her second day), but two new patients arrive, who get 3 each (2 * 3 = 6). The total is 2 + (2 * 3) = 8 units.
  • On Wednesday, it's trickier: The Monday gal finishes (1 unit, her last day), the Tuesday people get 2 units (2 * 2), and there are 3 new Wednesday people... argh.

The patients are overlapping and it's hard to track. How can we organize this calculation?

An idea: imagine flipping the patient list, so the first patient is on the right:

           Start of line
 5 4 3 2 1

Next, imagine we have 3 separate rooms where we apply the proper dose:

 Rooms     3 2 1

On your first day, you walk into the first room and get 3 units of medicine. The next day, you walk into room #2 and get 2 units. On the last day, you walk into room #3 and get 1 unit. There's no rooms afterwards, and your treatment is done.

To calculate the total medicine usage, line up the patients and walk them through the rooms:

 Monday
 ----------------------------
 Rooms                  3 2 1                                      
 Patients       5 4 3 2 1

 Usage                  3

On Monday (our first day), we have a single patient in the first room. She gets 3 units, for a total usage of 3. Makes sense, right?

On Tuesday, everyone takes a step forward:

 Tuesday
 ----------------------------
 Rooms                  3 2 1              
 Patients ->      5 4 3 2 1

 Usage                  6 2      = 8

The first patient is now in the second room, and there's 2 new patients in the first room. We multiply each room's dose by the patient count, then combine.

Every day we just walk the list forward:

 Wednesday
 ----------------------------
 Rooms                  3 2 1              
 Patients ->        5 4 3 2 1

 Usage                  9 4 1    = 14


 Thursday
 -----------------------------
 Rooms                  3 2 1              
 Patients ->          5 4 3 2 1

 Usage                 12 6 2    = 20


 Friday
 -----------------------------
 Rooms                  3 2 1              
 Patients ->            5 4 3 2 1

 Usage                 15 8 3    = 26

Whoa! It's intricate, but we figured it out, right? We can find the usage for any day by reversing the list, sliding it to the desired day, and combining the doses.

The total day-by-day usage looks like this (don't forget Sat and Sun, since some patients began on Friday):

Plan      *  Patient List   = Total Daily Usage

[3 2 1]   *  [1 2 3 4 5]    = [3 8 14 20 26 14 5]
              M T W T F        M T W  T  F  S  S

This calculation is the convolution of the plan and patient list. It's a fancy multiplication between a list of input numbers and a "program".

Interactive Demo

Here's a live demo. Try changing F (the plan) or G (the patient list). The convolution $c(t)$ matches our manual calculation above.

(We define functions $f(x)$ and $g(x)$ to pad each list with zero, and adjust for the list index starting at 1.)

You can do a quick convolution with Wolfram Alpha:

ListConvolve[{3, 2, 1}, {1, 2, 3, 4, 5}, {1, -1}, 0]
{3, 8, 14, 20, 26, 14, 5}

(The extra {1, -1}, 0 aligns the lists and pads with zero.)

Application: COVID Ventilator Usage

I started this article 5 years ago (intuition takes a while...), but unfortunately the analogy is relevant today.

Let's use convolution to estimate ventilator usage for incoming patients.

  • Set $f(x)$ as the percent of patients needing ventilators. For example, [.05 .03 .01] means 5% of patients need ventilators the first week, 3% the second week, and 1% the third week.
  • Set $g(x)$ as the weekly incoming patients, in thousands.
  • The convolution $c(t) = f * g$, shows how many ventilators are needed each week (in thousands). $c(5)$ is how many ventilators are needed 5 weeks from now.

Let's try it out:

  • F = [.05, .03, .01] is the ventilator use percentage by week
  • G = [10, 20, 30, 20, 10, 10, 10], is the incoming hospitalized patients. It starts at 10k per week, rises to 30k, then decays to 10k.

With these numbers, we expect a max ventilator use of 2.2k in 2 weeks:

Convolution_Demo

The convolution drops to 0 after 9 weeks because the patient list has run out. In this example, we're interested in the peak value the convolution hits, not the long-term total.

Other plans to convolve may be drug doses, vaccine appointments (one today, another a month from now), reinfections, and other complex interactions.

The hospital analogy is the mental model I wish I had when learning. Now that we've tried it with actual numbers, let's add the Math Juice and convert the analogy to calculus.

Part 2: The Calculus Definition

So, what happened in our example? We had a list of patients and a plan. If the plan were simple (single day [3]), regular multiplication would have worked. Because the plan was complex, we had to "convolve" it.

Time for some Fun Facts™:

  • Convolution is written $f * g$, with an asterisk. Yes, an asterisk usually indicates multiplication, but in advanced calculus class, it indicates a convolution. Regular multiplication is just implied ($fg$).

  • The result of a convolution is a new function that gives the total usage for any day ("What was the total usage on day $t=3$?"). We can graph the convolution over time to see the day-by-day totals.

Now the big aha: Convolution reverses one of the lists! Here's why.

Let's call our treatment plan $f(x)$. In our example, we used [3 2 1].

The list of patients (inputs) is $g(x)$. However, we need to reverse this list as we slide it, so the earliest patient (Monday) enters the hospital first (first in, first out). This means we need to use $g(-x)$, the horizontal reflection of $g(x)$. [1 2 3 4 5] becomes [5 4 3 2 1].

Now that we have the reversed list, pick a day to compute ($t = 0, 1, 2...$). To slide our patient list by this much, we use: $g(-x + t)$. That is, we reverse the list ($-x$) and jump to the correct day ($+t$).

We have our scenario:

  • $f(x)$ is the plan to use
  • $g(-x + t)$ is the list of inputs (flipped and slid to the right day).

To get the total usage on day $t$, we multiply each patient with the plan, and sum the results (an integral). To account for any possible length, we go from -infinity to +infinity.

Now we can describe convolution formally using calculus:

convolution

(Like colorized math? There's more.)

Phew! That's quite few symbols. Some notes:

  • We use a dummy variable $\tau$ (tau) for the intermediate computation. Imagine $\tau$ as knocking on each room ($\tau={0, 1, 2, 3...}$), finding the dosage [$f(\tau)$], the number of patients [$g(t - \tau)$], multiplying them, and totaling things in the integral. Yowza. The so-called "dummy" variable $\tau$ is like i in a for loop: it's temporary, but does the work. (By analogy, $t$ is a global variable has a fixed value during the loop: it's the day we're calculating the usage for, such as t = Day 5).
  • In the official definition, you'll see $g(t - \tau)$ instead of $g(- \tau+ t)$. The second version shows the flip ($-\tau$) and slide ($+t$). Writing $g(t - \tau)$ makes it seem like we're interested in the difference between the variables, which confused me.
  • The treatment plan (program to run) is called the kernel: you convolve a kernel with an input.

Not too bad, right? The equation is a formal description of the analogy.

Part 3: Mathematical Properties of Convolution

We can't discover a new math operation without taking it for a spin. Let's see how it behaves.

Convolution is commutative: f * g = g * f

In our computation, we flipped the patient list and kept the plan the same. Could we have flipped the plan instead?

You bet. Imagine the patients are immobile, and stay in their rooms: [1 2 3 4 5]. To deliver the medicine, we have 3 medical carts that go to each room and deliver the dose. Each day, they slide forward one position.

  Carts ->
  1 2 3

      1 2 3 4 5
      Patients

As before, though our plan is written [3 2 1] (3 units on the first day), we flip the order of the carts to[1 2 3]. That way, a patient gets 3 units on their first day, as we expect. Checking with Wolfram Alpha, the calculation is the same.

ListConvolve[{1, 2, 3, 4, 5}, {3, 2, 1}, {1, -1}, 0]
{3, 8, 14, 20, 26, 14, 5}

Cool! It looks like convolution is commutative:

\displaystyle{f * g = g * f}

and we can decide to flip either $f$ or $g$ when calculating the integral. Surprising, right?

The integral of the convolution

When all treatments are finished, what was the total medicine usage? This is the integral of the convolution. (A few minutes ago, that phrase would have you jumping out of a window.)

But it's a simple calculation. Our plan gives each patient sum([3 2 1]) = 6 units of medicine. And we have sum([1 2 3 4 5]) = 15 patients. The total usage is just 6 x 15 = 90 units.

Wow, that was easy: the usage for the entire convolution is just the product of the subtotals!

\displaystyle{\int (f * g) = \int f \cdot \int g }

I hope this clicks intuitively. Note that this trick works for convolution, but not integrals in general. For example:

\displaystyle{ \int (x \cdot x) \ne \int x \cdot \int x }

If we separate $x \cdot x$ into two integrals we get:

  • $\int (x \cdot x) = \int x^2 = \frac{1}{3} x^3$
  • $\int x \cdot \int x = \frac{1}{2}x^2 \cdot \frac{1}{2}x^2 = \frac{1}{4}x^4$

and those aren't the same. (Calculus would be much easier if we could split integrals like this.) It's strange, but $\int (f * g)$ is probably easier to solve than $\int (fg)$.

Impulse Response

What happens if we sent a single patient through the hospital? The convolution would just be that day's plan.

Plan    * Patients = Convolution
[3 2 1] * [1]      = [3 2 1]

In other words, convolving with [1] gives us the original plan.

In calculus terms, a spike of [1] (and 0 otherwise) is the Dirac Delta Function. In terms of convolutions, this function acts like the number 1 and returns the original function:

\displaystyle{f(t) * \delta(t) = f(t)}

We can delay the delta function by T, which delays the resulting convolution function too. Imagine our single patient shows up a week late ($\delta(t - T)$), so our medicine usage gets delayed for a week too:

\displaystyle{f(t) * \delta(t - T) = f(t - T)}

Part 4: Convolution Theorem & The Fourier Transform

The Fourier Transform (written with a fancy $\mathscr{F}$) converts a function $f(t)$ into a list of cyclical ingredients $F(s)$:

\displaystyle{f(t) \xrightarrow{\mathscr{F}} F(s)}

As an operator, this can be written $\mathscr{F}\lbrace f \rbrace = F$.

In our analogy, we convolved the plan and patient list with a fancy multiplication. Since the Fourier Transform gives us lists of ingredients, could we get the same result by mixing the ingredient lists?

Yep, we can: Fancy multiplication in the regular world is regular multiplication in the fancy world.

In math terms, "Convolution in the time domain is multiplication in the frequency (Fourier) domain."

Mathematically, this is written:

\displaystyle{f * g \xrightarrow{\mathscr{F}} FG}

or

\displaystyle{\mathscr{F}\lbrace f * g \rbrace = FG}

where $f(x)$ and $g(x)$ are functions to convolve, with transforms $F(s)$ and $G(s)$.

We can prove this theorem with advanced calculus, that uses theorems I don't quite understand, but let's think through the meaning.

Because $F(s)$ is the Fourier Transform of $f(t)$, we can ask for a specific frequency ($s = 2\text{Hz}$) and get the combined interaction of every data point with that frequency. Let's suppose:

\displaystyle{F(2) = 3 + i}

That means after every data point has been multiplied against the 2Hz cycle, the result is $3 + i$. But we could have kept each interaction separate:

\displaystyle{F(2) = 3 + i = c_0 + c_1 + c_2 + c_3 ... + c_t}

Where $c_t$ is the contribution to the 2Hz frequency from datapoint $t$. Similarly, we can expand $G(s)$ into a list of interactions with the 2Hz ingredient. Let's suppose $G(2) = 7 - i$:

\displaystyle{G(2) = 7 - i = d_0 + d_1 + d_2 + d_3 ... + d_t}

The Convolution Theorem is really saying:

\displaystyle{f * g \xrightarrow{\mathscr{F}} FG = (c_0 + c_1 + c_2 + ...)(d_0 + d_1 + d_2 + ...)}

Our convolution in the regular domain involves a lot of cross-multiplications. In the fancy frequency domain, we still have a bunch of interactions, but $F(s)$ and $G(s)$ have consolidated them. We can just multiply $F(2)G(2) = (3 + i)(7-i)$ to find the 2Hz ingredient in the convolved result.

By analogy, suppose you want to calculate:

\displaystyle{(1 + 2 + 3 + 4)(5 + 6 + 7 + 8) = ?}

It's a lot of work to cross-multiply every term: $(1 \cdot 5) + (1\cdot 6) + (1\cdot 7) + ...$

It's better to consolidate the groups into $(1 + 2 + 3 + 4) = 10$ and $(5 + 6 + 7 + 8) = 26$, and then multiply to get $10 \cdot 26 = 260$.

This nuance caused me a lot of confusion. It seems like $FG$ is a single multiplication, while $f * g$ involves a bunch of intermediate terms. I forgot that $F$ already did the work of merging a bunch of entries into a single one.

Now, we aren't quite done.

\displaystyle{f * g \xrightarrow{\mathscr{F}} FG \xrightarrow{\mathscr{F^{-1}}} \ ?}

We can convert $f * g$ in the time domain into $FG$ in the frequency domain, but we probably need it back in the time domain for a usable result:

\displaystyle{f * g = \mathscr{F}^{-1} \lbrace FG \rbrace}

You have a riddle in English ($f * g$), you translate it to French ($FG$), get your smart French friend to work out that calculation, then convert it back to English ($\mathscr{F}^{-1}$).

And in reverse...

The convolution theorem works this way too:

\displaystyle{fg \xrightarrow{\mathscr{F}} F * G}

Regular multiplication in the regular world is fancy multiplication in the fancy world.

Cool, eh? Instead of multiplying two functions like some cave dweller, put on your monocle, convolve the Fourier Transforms, and convert to the time domain:

\displaystyle{fg = \mathscr{F}^{-1} \lbrace F*G \rbrace}

I'm not saying this is fun, just that it's possible. If your French friend has a gnarly calculation they're struggling with, it might look like arithmetic to you.

Mini proof

Remember how we said the integral of a convolution was a multiplication of the individual integrals?

\displaystyle{\int (f * g) = \int f \cdot \int g }

Well, the Fourier Transform is just a very specific integral, right?

\displaystyle{\int \rightarrow \mathscr{F}}

So (handwaving), it seems we could swap the general-purpose integral $\int$ for $\mathscr{F}$ and get

\displaystyle{\mathscr{F} (f * g) = \mathscr{F} (f) \cdot \mathscr{F} (g) }

which is the convolution theorem. I need a deeper intuition for the proof, but this helps things click.

Part 5: Applications

The trick with convolution is finding a useful "program" (kernel) to apply to your input. Here's a few examples.

Moving averages

Let's say you want a moving average between neighboring items in a list. That is half of each element, added together:

\displaystyle{\frac{a + b}{2} = \frac{a}{2} + \frac{b}{2}}

This is a "multiplication program" of [0.5 0.5] convolved with our list:

ListConvolve[{1, 4, 9, 16, 25}, {0.5, 0.5}, {1, -1}, 0] 
{0.5, 2.5, 6.5, 12.5, 20.5, 12.5}

We can perform a moving average with a single operation. Neat!

A 3-element moving average would be [.33 .33 .33], a weighted average could be [.5 .25 .25].

Derivatives

The derivative finds the difference between neighboring values. Here's the plan: [1 -1]

ListConvolve[{1, 2, 3, 4, 5}, {1, -1}, {1, -1}, 0] 
{1, 1, 1, 1, 1, -5}                         // -5 since we ran out of entries

ListConvolve[{1, 4, 9, 16, 25}, {1, -1}, {1, -1}, 0] 
{1, 3, 5, 7, 9, -25}                        // discrete derivative is 2x + 1

With a simple kernel, we can find a useful math property on a discrete list. And to get a second derivative, just apply the derivative convolution twice:

F * [1 -1] * [1 -1]

As a shortcut, we can precompute the final convolutions ([1 -1] * [1 -1] ) and get:

ListConvolve[{1, -1}, {1,-1}, {1, -1}, 0] 
{1, -2, 1}

Now we have a single kernel [1, -2, 1] that gets the second derivative of a list:

ListConvolve[{1, 4, 9, 16, 25}, {1, -2, 1}, {1, -1}, 0] 
{1, 2, 2, 2, 2, -34, 25}

Excluding the boundary items, we get the expected second derivative:

\displaystyle{x^2 \xrightarrow{d/dx} 2x \xrightarrow{d/dx} 2}

Blurring / unblurring images

An image blur is essentially a convolution of your image with some "blurring kernel":

\displaystyle{\text{blurred} = \text{image} * \text{blur}}

The blur of our 2D image requires a 2D average:

Kernel__image_processing__-_Wikipedia

Can we undo the blur? Yep! With our friend the Convolution Theorem, we can do:

\displaystyle{\text{blurred} = \text{image} * \text{blur}}

\displaystyle{\mathscr{F} \lbrace \text{blurred} \rbrace = \mathscr{F} \lbrace \text{image} * \text{blur} \rbrace}

\displaystyle{\mathscr{F} \lbrace \text{blurred} \rbrace = \mathscr{F} \lbrace \text{image} \rbrace \mathscr{F} \lbrace \text{blur} \rbrace}

\displaystyle{\frac{ \mathscr{F} \lbrace \text{blurred} \rbrace }{\mathscr{F} \lbrace \text{blur} \rbrace} = \mathscr{F} \lbrace \text{image} \rbrace }

\displaystyle{\mathscr{F}^{-1} \lbrace \frac{ \mathscr{F} \lbrace \text{blurred} \rbrace }{\mathscr{F} \lbrace \text{blur} \rbrace} \rbrace = \text{image}}

Whoa! We can recover the original image by dividing out the blur. Convolution is a simple multiplication in the frequency domain, and deconvolution is a simple division in the frequency domain.

image-blur

A short while back, the concept of "deblurring by dividing Fourier Transforms" was gibberish to me. While it can be daunting mathematically, it's getting simpler conceptually.

More reading:

Algorithm Trick: Multiplication

What is a number? A list of digits:

1234 = 1000 + 200 + 30 + 4 = [1000 200 30 4]
5678 = 5000 + 600 + 70 + 8 = [5000 600 70 8]

And what is regular, grade-school multiplication? A digit-by-digit convolution! We sweep one list of digits by the other, multiplying and adding as we go:

Intuitive Guide to Convolution

Source

We can perform the calculation by convolving the lists of digits (wolfram alpha):

ListConvolve[{1000, 200, 30, 4}, {8, 70, 600, 5000}, {1, -1}, 0]
{8000, 71600, 614240, 5122132, 1018280, 152400, 20000}

sum {8000, 71600, 614240, 5122132, 1018280, 152400, 20000}
7006652

Note that we pre-flip one of the lists (it gets swapped in the convolution later), and the intermediate calculations are a bit different. But, combining the subtotals gives the expected result.

Faster Convolutions

Why convolve instead of doing a regular digit-by-digit multiplication? Well, the convolution theorem lets us substitute convolution with Fourier Transforms:

\displaystyle{f * g = \mathscr{F}^{-1} \lbrace FG \rbrace}

The convolution ($f * g$) has complexity $O(n^2)$. We have $n$ positions to process, with $n$ intermediate multiplications at each position.

The right side involves:

  • Two Fourier Transforms, which are normally $O(n^2)$. However, the Fast Fourier Transform (a divide-and-conquer approach) makes them $O(n\log(n))$.
  • Pointwise multiplication of the final result of the transforms ($\sum a_n \cdot b_n$), which is $O(n)$
  • An inverse transform, which is $O(n\log(n))$

And the total complexity is: $O(n\log(n)) + O(n\log(n)) + O(n) + O(n\log(n)) = O(n\log(n))$

Regular multiplication in the fancy domain is faster than a fancy multiplication in the regular domain. Our French friend is no slouch. (More)

Convolutional Neural Nets (CNN)

Machine learning is about discovering the math functions that transform input data into a desired result (a prediction, classification, etc.).

Starting with an input signal, we could convolve it with a bunch of kernels:

\displaystyle{\text{input} * k_1 * k_2 * k_3 ... = \text{result}}

Given that convolution can do complex math (moving averages, blurs, derivatives...), it seems some combination of kernels should turn our input into something useful, right?

Convolutional Neural Nets (CNNs) process an input with layers of kernels, optimizing their weights (plans) to reach a goal. Imagine tweaking the treatment plan to keep medicine usage below some threshold.

CNNs are often used with image classifiers, but 1D data sets work just fine.

LTI System Behavior

A linear, time-invariant system means:

  • Linear: Scaling and combining inputs scales and combines outputs by the same amount
  • Time invariant: Outputs depend on relative time, not absolute time. You get 3 units on your first day, and it doesn't matter if it's Wednesday or Thursday.

A fancy phrase is "A LTI system is characterized by its impulse response". Translation: If we send a single patient through the hospital [1], we'll discover the treatment plan. Then we can predict the usage for any sequence of patients by convolving it with the plan.

\displaystyle{\text{system response} = \text{impulse response} * \text{inputs}}

If the system isn't LTI, we can't extrapolate based on a single person's experience. Scaling the inputs may not scale the outputs, and the actual calendar day, not relative day, may impact the result (imagine fewer rooms available on weekends).

Engineering Analogies

From David Greenspan: "Suppose you have a special laser pointer that makes a star shape on the wall. You tape together a bunch of these laser pointers in the shape of a square. The pattern on the wall now is the convolution of a star with a square."

Regular multiplication gives you a single scaled copy of an input. Convolution creates multiple overlapping copies that follow a pattern you've specified.

Real-world systems have squishy, not instantaneous, behavior: they ramp up, peak, and drop down. The convolution lets us model systems that echo, reverb and overlap.

Now it's time for the famous sliding window example. Think of a pulse of inputs (red) sliding through a system (blue), and having a combined effect (yellow): the convolution.

Intuitive Guide to Convolution

(Source)

Summary

Convolution has an advanced technical definition, but the basics can be understood with the right analogy.

Quick rant: I study math for fun, yet it took years to find a satisfying intuition for:

  • Why is one function reversed?
  • Why is convolution commutative?
  • Why does the integral of the convolution = product of integrals?
  • Why are the Fourier Transforms multiplied point-by-point, and not overlapped?

Why'd it take so long? Imagine learning multiplication with $f \times g = z$ instead of $3 \times 5 = 15$. Without an example I can explore in my head, I could only memorize results, not intuit them. Hopefully this analogy can save you years of struggle.

Happy math.

]]> https://betterexplained.com/articles/intuitive-convolution/feed/ 33 Integral of Sin(x): Geometric Intuition https://betterexplained.com/articles/integral-sinx/ https://betterexplained.com/articles/integral-sinx/#disqus_thread Wed, 20 Nov 2019 00:01:09 +0000 https://betterexplained.com/?p=12244 You're minding your own business when some punk asks what the integral of $\sin(x)$ means. Your options:

  • Pretend to be asleep (except not in the engineering library again)
  • Canned response: "As with any function, the integral of sine is the area under its curve."
  • Geometric intuition: "The integral of sine is the horizontal distance along a circular path."

Option 1 is tempting, but let's take a look at the others.

Why "Area Under the Curve" is Unsatisfying

Describing an integral as "area under the curve" is like describing a book as a list of words. Technically correct, but misses the message and I suspect you haven't done the assigned reading.

Integral of Sin(x): Geometric Intuition

Unless you're trapped in LegoLand, integrals mean something besides rectangles.

Decoding the Integral

My calculus conundrum was not having an intuition for all the mechanics.

When we see:

$\int \sin(x) dx$

We can call on a few insights:

  • The integral is just fancy multiplication. Multiplication accumulates numbers that don't change (3 + 3 + 3 + 3). Integrals add up numbers that might change, based on a pattern (1 + 2 + 3 + 4). But if we squint our eyes and pretend items are identical we have a multiplication.

  • $\sin(x)$ just a percentage. Yes, it's also fancy curve with nice properties. But at any point (like 45 degrees), it's a single percentage from -100% to +100%. Just regular numbers.

  • $dx$ is a tiny, infinitesimal part of the path we're taking. 0 to $x$ is the full path, so $dx$ is (intuitively) a nanometer wide.

Ok. With those 3 intuitions, our rough (rough!) conversion to Plain English is:

The integral of sin(x) multiplies our intended path length (from 0 to x) by a percentage

We intend to travel a simple path from 0 to x, but we end up with a smaller percentage instead. (Why? Because $\sin(x)$ is usually less than 100%). So we'd expect something like 0.75x.

In fact, if $\sin(x)$ did have a fixed value of 0.75, our integral would be:

$\int \text{fixedsin}(x) \ dx = \int 0.75 \ dx = 0.75 \int dx = 0.75x$

But the real $\sin(x)$, that rascal, changes as we go. Let's see what fraction of our path we really get.

Visualize The Change in Sin(x)

Now let's visualize $\sin(x)$ and its changes:

Integral of Sin(x): Geometric Intuition

Here's the decoder key:

  • $x$ is our current angle in radians. On the unit circle (radius=1), the angle is the distance along the circumference.

  • $dx$ is a tiny change in our angle, which becomes the same change along the circumference (moving 0.01 units in our angle moves 0.01 along the circumference).

  • At our tiny scale, a circle is a a polygon with many sides, so we're moving along a line segment of length $dx$. This puts us at a new position.

With me? With trigonometry, we can find the exact change in height/width as we slide along the circle by $dx$.

By similar triangles, our change just just our original triangle, rotated and scaled.

  • Original triangle (hypotenuse = 1): height = $\sin(x)$, width = $\cos(x)$
  • Change triangle (hypotenuse = dx): height = $\sin(x) dx$, width = $\cos(x) dx$

Now, remember that sine and cosine are functions that return percentages. (A number like 0.75 doesn't have its orientation. It shows up and makes things 75% of their size in whatever direction they're facing.)

So, given how we've drawn our Triangle of Change, $\sin(x) dx$ is our horizontal change. Our plain-English intuition is:

The integral of sin(x) adds up the horizontal change along our path

Visualize The Integral Intuition

Ok. Let's graph this bad boy to see what's happening. With our "$\sin(x) dx$ = tiny horizontal change" insight we have:

Integral of Sin(x): Geometric Intuition

As we circle around, we have a bunch of $dx$ line segments (in red). When sine is small (around x=0) we barely get any horizontal motion. As sine gets larger (top of circle), we are moving up to 100% horizontally.

Ultimately, the various $\sin(x) dx$ segments move us horizontally from one side of the circle to the other.

A more technical description:

$\int_0^x \sin(x) dx = \text{horizontal distance traveled on arc from 0 to x}$

Aha! That's the meaning. Let's eyeball it. When moving from $x=0$ to $x=\pi$ we move exactly 2 units horizontally. It makes complete sense in the diagram.

The Official Calculation

Using the Official Calculus Fact that $\int \sin(x) dx = -\cos(x)$ we would calculate:

$\int_0^\pi \sin(x) dx = -\cos(x) \Big|_0^\pi = -\cos(\pi) - -\cos(0) = -(-1) -(-1) = 1 + 1 = 2$

Yowza. See how awkward it is, those double negations? Why was the visual intuition so much simpler?

Our path along the circle ($x=0$ to $x=\pi$) moves from right-to-left. But the x-axis goes positive from left-to-right. When convert distance along our path into Standard Area™, we have to flip our axes:

Integral of Sin(x): Geometric Intuition

Our excitement to put things in the official format stamped out the intuition of what was happening.

Fundamental Theorem of Calculus

We don't really talk about the Fundamental Theorem of Calculus anymore. (Is it something I did?)

Instead of adding up all the tiny segments, just do: end point - start point.

The intuition was staring us in the face: $\cos(x)$ is the anti-derivative, and tracks the horizontal position, so we're just taking a difference between horizontal positions! (With awkward negatives to swap the axes.)

That's the power of the Fundamental Theorem of Calculus. Skip the intermediate steps and just subtract endpoints.

Onward and Upward

Why did I write this? Because I couldn't instantly figure out:

$\int_0^\pi \sin(x) dx = 2$

This isn't an exotic function with strange parameters. It's like asking someone to figure out $2^3$ without a calculator. If you claim to understand exponents, it should be possible, right?

Now, we can't always visualize things. But for the most common functions we owe ourselves a visual intuition. I certainly can't eyeball the 2 units of area from 0 to $\pi$ under a sine curve.

Happy math.

Appendix: Average Efficiency

As a fun fact, the "average" efficiency of motion around the top of a circle (0 to $\pi$) is: $\frac{2}{\pi} = .6366$

So on average, 63.66% of your path's length is converted to horizontal motion.

Appendix: Height controls width?

It seems weird that height controls the width, and vice-versa, right?

If height controlled height, we'd have runaway exponential growth. But a circle needs to regulate itself.

$e^x$ is the kid who eats candy, grows bigger, and can therefore eat more candy.

$\sin(x)$ is the kid who eats candy, gets sick, waits for an appetite, and eats more candy.

Appendix: Area isn't literal

The "area" in our integral isn't literal area, it's a percentage of our length. We visualized the multiplication as a 2d rectangle in our generic integral, but it can be confusing. If you earn money and are taxed, do you visualize 2d area (income * (1 - tax))? Or just a quantity helplessly shrinking?

Area primarily indicates a multiplication happened. Don't let team Integrals Are Literal Area win every battle!

]]> https://betterexplained.com/articles/integral-sinx/feed/ 21 Intuition for Taylor Series (DNA Analogy) https://betterexplained.com/articles/taylor-series/ https://betterexplained.com/articles/taylor-series/#disqus_thread Thu, 24 Jan 2019 06:14:43 +0000 https://betterexplained.com/?p=11904 Your body has a strange property: you can learn information about the entire organism from a single cell. Pick a cell, dive into the nucleus, and extract the DNA. You can now regrow the entire creature from that tiny sample.

There's a math analogy here. Take a function, pick a specific point, and dive in. You can pull out enough data from a single point to rebuild the entire function. Whoa. It's like remaking a movie from a single frame.

The Taylor Series discovers the "math DNA" behind a function and lets us rebuild it from a single data point. Let's see how it works.

Intuition for Taylor Series (DNA Analogy)

Pulling information from a point

Given a function like $f(x) = x^2$, what can we discover at a single location?

Normally we'd expect to calculate a single value, like $f(4) = 16$. But there's much more beneath the surface:

  • $f(x)$ = Value of function at point $x$
  • $f'(x)$ = First derivative, or how fast the function is changing (the velocity)
  • $f''(x)$ = Second derivative, or how fast the changes are changing (the acceleration)
  • $f'''(x)$ = Third derivative, or how fast the changes in the changes are changing (acceleration of the acceleration)
  • And so on

Investigating a single point reveals multiple, possibly infinite, bits of information about the behavior. (Some functions have an endless amount of data (derivatives) at a single point).

So, given all this information, what should we do? Regrow the organism from a single cell, of course! (Maniacal cackle here.)

Growing a Function from a point

Our plan is to grow a function from a single starting point. But how can we describe any function in a generic way?

The big aha moment: imagine any function, at its core, is a polynomial (with possibly infinite terms):

\displaystyle{f(x) = c_0 + c_1 x + c_2 x^2 + c_3x^3 + \cdots}

To rebuild our function, we start at a fixed point ($c_0$) and add in a bunch of other terms based on the value we feed it (like $c_1x$). The "DNA" is the values $c_0, c_1, c_2, c_3$ that describe our function exactly.

Ok, we have a generic "function format". But how do we find the coefficients for a specific function like sin(x) (height of angle x on the unit circle)? How do we pull out its DNA?

Time for the magic of 0.

Let's start by plugging in the function value at $x=0$. Doing this, we get:

\displaystyle{f(0) = c_0 + 0 + 0 + 0 + \cdots = c_0}

Every term vanishes except $c_0$, which makes sense: the starting point of our blueprint should be $f(0)$. For $f(x) = \sin(x)$, we can work out $c_0 = \sin(0) = 0$. We have our first bit of DNA!

Getting More DNA

Now that we know $c_0$, how do we isolate $c_1$ in this equation?

\displaystyle{f(x) = c_0 + c_1 x + c_2 x^2 + c_3x^3 + \cdots}

Hrm. A few ideas:

  • Can we set $x = 1$? That gives $f(1) = c_0 + c_1(1) + c_2(1^2) + c_3(1^3) + \cdots$ . Although we know $c_0$, the other constants are summed together. We can't pull out $c_1$ by itself.

  • What if we divide by $x$? This gives:

\displaystyle{\frac{f(x)}{x} = \frac{c_0}{x} + c_1 + c_2 x + c_3x^2 + \cdots}

Then we can set $x=0$ to make the other terms disappear... right? It's a nice idea, except we're now dividing by zero.

Hrm. This approach is really close. How can we almost divide by zero? Using the derivative!

If we take the derivative of the blueprint of $f(x)$, we get:

\displaystyle{f'(x) = (c_0)' + (c_1 x)' + (c_2 x^2)' + (c_3x^3)' + \cdots}

\displaystyle{f'(x) = 0 + c_1 + (2\cdot c_2 x) + (3\cdot c_3x^2) + \cdots}

Every power gets reduced by 1 and the $c_0$, a constant value, becomes zero. It's almost too convenient.

Now we can isolate $c_1$ using our $x=0$ trick:

\displaystyle{f'(0) = 0 + c_1 + (0) + (0) + \cdots = c_1}

In our example, $\sin'(x) = \cos(x)$ so we compute: $f'(0) = \sin'(0) = \cos(0) = 1 = c_1$

Yay, one more bit of DNA! This is the magic of the Taylor series: by repeatedly applying the derivative and setting $x = 0$, we can pull out the polynomial DNA.

Let's try another round:

\displaystyle{f''(x) = 0 + 0 + (2\cdot c_2) + (3\cdot 2 c_3x^1) + \cdots}

After taking the second derivative, the powers are reduced again. The first two terms ($c_0$ and $c_1x$) disappear, and we can again isolate $c_2$ by setting $x=0$:

\displaystyle{f''(0) = 0 + 0 + 2\cdot c_2 + 0 + \cdots}

For our sine example, $\sin'' = -\sin$, so:

\displaystyle{f''(0) = \sin''(0) = -\sin(0) = 0 = 2\cdot c_2}

or $c_2 = 0$.

As we keep taking derivatives, we're performing more multiplications and growing a factorial in front of each term (1!, 2!, 3!).

The Taylor Series for a function around point x=0 is:

\displaystyle{f(x) = f(0) + f'(0) x + \frac{f''(0)}{2!}x^2 + \frac{f'''(0)}{3!}x^3 + \cdots}

(Formally, the Taylor series around the point $x=0$ is called the MacLaurin series.)

The generalized Taylor series, extracted from any point a is:

\displaystyle{f(x) = f(a)+{\frac {f'(a)}{1!}}(x-a)+{\frac {f''(a)}{2!}}(x-a)^{2}+{\frac {f'''(a)}{3!}}(x-a)^{3}+\cdots }

The idea is the same. Instead of our regular blueprint, we use:

\displaystyle{f(x) = c_0 + c_1 (x - a) + c_2 (x-a)^2 + c_3(x-a)^3 + \cdots}

Since we're growing from $f(a)$, we can see that $f(a) = c_0 + 0 + 0 + \dots = c_0$. The other coefficients can be extracted by taking derivatives and setting $x = a$ (instead of $x =0$).

Example: Taylor Series of sin(x)

Plugging in derivatives into the formula above, here's the Taylor series of $\sin(x)$ around $x = 0$:

\displaystyle{\sin(x) = x - \frac{x^3}{3!} + \frac{x^5}{5!} - \frac{x^7}{7!} + \dots}

And here's what that looks like:

Intuition for Taylor Series (DNA Analogy)

A few notes:

1) Sine has infinite terms

Sine is an infinite wave, and as you can guess, needs an infinite number of terms to keep it going. Simpler functions (like $f(x) = x^2 + 3$) are already in their "polynomial format" and don't have infinite derivatives to keep the DNA going.

2) Sine is missing every other term

If we repeatedly take the derivative of sine at x = 0 we get:

\displaystyle{\sin(0) \xrightarrow{\text{derive}} \cos(0) \xrightarrow{\text{derive}} -\sin(0) \xrightarrow{\text{derive}} -\cos(0) \xrightarrow{\text{derive}} \sin(0) \dots}

with values:

\displaystyle{0, 1, 0, -1, \dots}

Ignoring the division by the factorial, we get the pattern:

\displaystyle{(0) + (1)x^1 + (0)x^2 + (-1)x^3 + (0)x^4 + (-1)x^5+ \dots }

So the DNA of sine is something like [0, 1, 0, -1] repeating.

3) Different starting positions have different DNA

For fun, here's the Taylor series of $\sin(x)$ starting at $x =\pi$ (link):

taylor series sin x at x=pi - Wolfram|Alpha 2019-01-18 16-45-04

A few notes:

  • The DNA is now something like [0, -1, 0, 1]. The cycle is similar, but the starting value has changed since we're starting at $x=\pi$.

  • Written as calculated numbers, the denominators 1, 6, 120, 5040 look strange. But they're just every other factorial: 1! = 1, 3! = 6, 5! =120, 7! = 5040. In general, the Taylor series can have gnarly denominators.

  • The $O(x^{12})$ term means there are other components of order (power) $x^{12}$ and higher. Because $\sin(x)$ has infinite derivatives, we have infinite terms and the computer has to cut us off somewhere. (You've had enough Tayloring for today, buddy.)

Application: Function Approximations

A popular use of Taylor series is getting a quick approximation for a function. If you want a tadpole, do you need the DNA for the entire frog?

Intuition for Taylor Series (DNA Analogy)

The Taylor series has a bunch of terms, typically ordered by importance:

\displaystyle{f(x) = f(0) + f'(0) x + \frac{f''(0)}{2!}x^2 + \frac{f'''(0)}{3!}x^3 + \cdots}

  • $c_0 = f(0)$, the constant term, is the exact value at the point
  • $c_1 = f'(0)x$, the linear term, tells us what speed to move from our point
  • $c_2= \frac{f''(0)}{2!}x^2$, the quadratic term, tells us how much to accelerate away from our point
  • and so on

If we only need a prediction for a few instants around our point, the initial position & velocity may be good enough:

\displaystyle{\text{Linear model} = \text{initial point} + \text{velocity effect} = f(0) + f'(0)x}

If we're tracking for longer, then acceleration becomes important:

\displaystyle{\text{Quadratic model} = \text{initial point} + \text{velocity effect} + \text{acceleration effect}} \displaystyle{ = f(0) + f'(0)x + \frac{1}{2}f''(0)x^2}

As we get further from our starting point, we need more terms to keep our prediction accurate. For example, the linear model $\sin(x) = x$ is a good prediction around $x=0$. As we get further out, we need to account for more terms.

Similarly, $e^x \sim 1 + x$ works well for small interest rates: 1% discrete interest is 1.01 after one time period, 1% continuous interest is a tad higher than 1.01. As time goes on, the linear model falls behind because it ignores the compounding effects.

Application: Comparing Functions

What's a common application of DNA? Paternity tests.

If we have a few functions, we can compare their Taylor series to see if they're related.

Here's the expansions of $\sin(x)$, $\cos(x)$, and $e^x$:

\displaystyle{ \sin x = x - \frac{x^3}{3!} + \frac{x^5}{5!} - \dots \xrightarrow{DNA} [0, 1, 0 -1, \dots] }

\displaystyle{ \cos x = 1 - \frac{x^2}{2!} + \frac{x^4}{4!} - \dots \xrightarrow{DNA} [1, 0, -1, 0, \dots] }

\displaystyle{ e^x = 1 + x + \frac{x^2}{2!} + \frac{x^3}{3!} + \dots \xrightarrow{DNA} [1, 1, 1, 1, \dots] }

There's a family resemblance in the sequences, right? Clean powers of $x$ divided by a factorial?

One problem is the sequence for $e^x$ has positive terms, while sine and cosine alternate signs. How can we link these together?

Euler's great insight was realizing an imaginary number could swap the sign from positive to negative:

{\begin{aligned}e^{ix}&=1+ix+{\frac {(ix)^{2}}{2!}}+{\frac {(ix)^{3}}{3!}}+{\frac {(ix)^{4}}{4!}}+{\frac {(ix)^{5}}{5!}}+{\frac {(ix)^{6}}{6!}}+{\frac {(ix)^{7}}{7!}}+{\frac {(ix)^{8}}{8!}}+\cdots \\[8pt]&=1+ix-{\frac {x^{2}}{2!}}-{\frac {ix^{3}}{3!}}+{\frac {x^{4}}{4!}}+{\frac {ix^{5}}{5!}}-{\frac {x^{6}}{6!}}-{\frac {ix^{7}}{7!}}+{\frac {x^{8}}{8!}}+\cdots \\[8pt]&=\left(1-{\frac {x^{2}}{2!}}+{\frac {x^{4}}{4!}}-{\frac {x^{6}}{6!}}+{\frac {x^{8}}{8!}}-\cdots \right)+i\left(x-{\frac {x^{3}}{3!}}+{\frac {x^{5}}{5!}}-{\frac {x^{7}}{7!}}+\cdots \right)\\[8pt]&=\cos x+i\sin x.\end{aligned}}

Whoa. Using an imaginary exponent and separating into odd/even powers reveals that sine and cosine are hiding inside the exponential function. Amazing.

Although this proof of Euler's Formula doesn't show why the imaginary number makes sense, it reveals the baby daddy hiding backstage.

Appendix: Assorted Aha! Moments

Relationship to Fourier Series

The Taylor Series extracts the "polynomial DNA" and the Fourier Series/Transform extracts the "circular DNA" of a function. Both see functions as built from smaller parts (polynomials or exponential paths).

Does the Taylor Series always work?

This gets into mathematical analysis beyond my depth, but certain functions aren't easily (or ever) approximated with polynomials.

Notice that powers like $x^2, x^3$ explode as $x$ grows. In order to have a slow, gradual curve, you need an army of polynomial terms fighting it out, with one winner barely emerging. If you stop the train too early, the approximation explodes again.

For example, here's the Taylor Series for $\ln(1 + x)$. The black line is the curve we want, and adding more terms, even dozens, barely gets us accuracy beyond $x=1.0$. It's just too hard to maintain a gentle slope with terms that want to run hog wild.

Intuition for Taylor Series (DNA Analogy)

source

In this case, we only have a radius of convergence where the approximation stays accurate (such as around $|x| < 1$).

Turning geometric to algebraic definitions

Sine is often defined geometrically: the height of a line on a circular figure.

Intuition for Taylor Series (DNA Analogy)

Turning this into an equation seems really hard. The Taylor Series gives us a process: If we know a single value and how it changes (the derivative), we can reverse-engineer the DNA.

Similarly, the description of $e^x$ as "the function with its derivative equal to the current value" yields the DNA [1, 1, 1, 1], and polynomial $f(x) = 1 + \frac{1}{1!}x + \frac{1}{2!}x^2 + \frac{1}{3!}x^3 + \dots$. We went from a verbal description to an equation.

Phew! A few items to ponder.

Happy math.

]]> https://betterexplained.com/articles/taylor-series/feed/ 17 How to Add 1 through 100 using Calculus https://betterexplained.com/articles/how-to-add-1-to-100-using-calculus/ https://betterexplained.com/articles/how-to-add-1-to-100-using-calculus/#disqus_thread Fri, 17 Feb 2017 20:52:31 +0000 https://betterexplained.com/?p=10127 Earlier we saw a few ways to add a set of numbers (1 to 10).

adding 1 to 100 strategies

And the formula we found was:

\displaystyle{\text{Sum from 1 to n} = \frac{n(n+1)}{2}}

\displaystyle{\text{Sum from 1 to 100} = \frac{100(100+1)}{2} = (50)(101) = 5050}

It seems that regular arithmetic, algebra, geometry, or even statistics could help work out the equation.

But how about Calculus? Is this bringing a nuclear missile to a gun fight?

Let's find out.

The sequence to add (1 2 3 4 5 6 7 8 9 10...) looks a lot like $f(x) = x$. At every position on the x-axis, we put in a number and get the same one out.

Intuitively, the integral is "repeatedly adding a bunch of stuff" -- it seems like we could put it to work. From the rules of Calculus (or using Wolfram Alpha) we get this:

\displaystyle{ \int x = \frac{1}{2} x^2 }

Intuitively: Add up things following the $f(x) = x$ pattern and you end up with $\frac{1}{2} x^2$.

Well, let's see: the actual sum from 1 to 100 is 5050. But using the Calculus equation we get:

\displaystyle{ \frac{1}{2} x^2 = \frac{1}{2}100^2 = \frac{10,000}{2} = 5000 }

Uh oh: there's a difference. What's going on?

Calculus works with continuous patterns, and we used a discrete one.

Here's what's happening:

calculus add area under x

Calculus was built to measure smoothly changing functions, like a line, parabola, circle, etc. The pattern we have is a jumpy staircase (going from 1 to 2 without ever passing through 1.5, or 1.1, or 1.0001). In math class, books harp on analyzing whether a function is "continuous", aka changes smoothly enough for Calculus to work.

So when a pattern changes smoothly, Calculus works great. If a pattern changes suddenly, Calculus can only give an approximate answer. So what's the plan?

Use Calculus where possible, on the smooth part, and adjust for errors in the jumpy part.

calculus integral of x discrete

The area under the line is the integral. We a bunch of triangles above the line we need to include.

  • How many of them? 1 for each item (x)
  • How big are they? They're half a of a 1x1 square, so they have area 1/2.
  • What's the total area to add back in? $x \frac{1}{2} = \frac{x}{2}$

So our final formula should be

\displaystyle{\text{Integral} + \text{Adjustment} = \frac{1}{2} x^2 + \frac{x}{2}}

Aha! Learning Calculus doesn't mean we hunt around for Official Calculus Problems.

Nope. Take your scenario (adding 1 to 100) and realize what Calculus brings to the table: finding patterns in smoothly changing functions. Use Calculus on the smooth parts and adjust (or ignore) the other parts.

(Ironically, Calculus works by making jumpy approximations for smooth functions, and is in fact "jumpy" under the hood. If you are planning on working with jumpy patterns, use Discrete Calculus.)

Example: Adding the first n squares

Let's take this further: what's your guess for the sum of the first 100 square numbers?

\displaystyle{1^2 + 2^2 + 3^2 + 4^2 + ... + 100^2}

Hrm. Getting the exact formula is tricky. But maybe we don't need the exact count, just an estimate.

With Calculus, we'd say: The pattern isn't continuous, but it looks like $f(x) = x^2$. Let's integrate x^2 from 0 to 100.

\displaystyle{ \int x^2 = \frac{1}{3} x^3 }

The indefinite integral is $\frac{1}{3} x^3$, the running total for how much we have. From 0 to 100 it would be

\displaystyle{ \frac{1}{3}100^3 = \frac{1,000,000}{3} \sim 333,333 }

That's our guess, without a calculator. And the actual answer? 338350.

calculus area under x squared

How close were we? 99.9%. Not bad for something we worked out by hand in a minute!

Truly internalizing Calculus means it helps other elements of your math understanding, even regular addition problems.

Happy math.

PS. To keep building your intuition, check out the Calculus Guide.

]]> https://betterexplained.com/articles/how-to-add-1-to-100-using-calculus/feed/ 12