Training Networks: Backpropagation and Gradient Methods

UPI Study › Courses › Introduction to Artificial Intelligence › Free lessons › Lesson 5.5 Free with a free account

Training Networks: Backpropagation and Gradient Methods

Introduction to Artificial Intelligence · Chapter 5: Neural Networks and Learning ArchitecturesSection 5.5 · lesson 36 of 6912 min · free with a free account

← Forward-Flowing Neural Network StructuresNext lesson: Image-Focused Learning with Convolutional Networks →

Earn 3 college credits for Introduction to Artificial IntelligenceRecommended for credit by NCCRS · Computer Science 373 · self-paced · one payment, lifetime access
Enroll in Introduction to Artificial Intelligence · $250 →Download the syllabus (PDF)

In this lesson

How gradient descent and backpropagation train neural network weights.

  • Explain how loss functions and gradient descent with a learning rate reduce errors
  • Compute a backpropagation step on a small network using the chain rule
  • Compare batch, stochastic, and mini-batch training and optimizers such as momentum and Adam
  • Describe vanishing gradients and regularization methods such as weight decay, dropout, and early stopping

Free study notes

Get this lesson's notes by email

The key points and 4 practice questions with answers, in your inbox. One email, free.

One email. Unsubscribe in one click.

Transcript

Everything said in the video, section by section (1,592 words). Read along, or search it.

Opening Analogy

Millions of Dials Set at Random

A network can hold millions of adjustable numbers. Think of a control panel with millions of dials. At first, the dials are set at random, and the network's answers are poor. Nobody could set them all by hand. Training turns every dial a little at a time, always in the direction that makes the answers better, and repeats that many times. So how does a network measure better? And how does it know which way to turn each dial?

Loss Functions

Scoring Mistakes With a Single Number

First, a way to score mistakes. A loss function turns the network's errors into a single number: the larger it is, the worse the predictions. Squared error suits predicting a number. Predict 8 when the truth is 10, and the loss is the gap squared: 4. Cross-entropy suits choosing a category. It's minus the natural log of the probability given to the right answer. Put 0.9 on the right answer, and the loss is 0.105. Put only 0.1 on it, and the loss is 2.303. Confident wrong answers are punished hard. Training means finding weights that make the loss small.

Gradient Descent

Stepping Against the Slope

Picture the loss as a landscape, with the weights as your position on it. The gradient is the slope under your feet. Written d L by d w, it says how much the loss would change if that weight were nudged up a little. To lower the loss, step the opposite way. That's gradient descent. The new weight is the old weight, minus eta times the gradient. Eta is the learning rate: it sets the size of each step. A positive gradient means raising the weight would raise the loss, so the weight goes down.

Training Loop

Forward, Loss, Backward, Update

Every training method repeats the same four-step loop. Forward pass: feed examples through, to get predictions. Loss: compare them with the right answers. Backward pass: compute the gradient for every weight. Update: move each weight a small step against its gradient. Then go around again.

Batch Size

Batch, Stochastic, and Mini-Batch Training

One full pass through the training data is called an epoch. The big choice is how many examples feed each gradient estimate. Say there are 1,000 examples. Batch training uses all of them for each update: one smooth, exact step per epoch, but each one is slow. Stochastic gradient descent, or SGD, uses one example at a time: 1,000 fast updates per epoch, but a very jittery path. Mini-batches use a small group, say 50: 20 updates per epoch, moderate noise, and efficient on parallel hardware. That's the usual choice.

Backpropagation

The Chain Rule Multiplies Local Rates

Gradient descent needs the gradient for every weight, and big networks have millions. Nudging each weight in turn would need one forward pass per weight: far too slow. Backpropagation finds all of them with one forward pass and one backward pass. It relies on the chain rule. If the loss depends on a weight through a chain of steps, the overall rate of change is the product of each step's rate, like gears that multiply each other's effects. For a weight feeding z, then a, then y-hat, then the loss, the gradient is four local rates, multiplied together.

Two Sweeps

Forward Pass and Backward Pass

Backpropagation organizes this into two sweeps. The forward pass computes every value from the input to the loss, and remembers them all. The backward pass starts at the loss. At each step, it takes the gradient arriving from the right, multiplies it by that step's own local rate, and passes it left. Each weight's gradient is read off where that weight enters the chain.

Worked Example

Forward Pass on a Tiny Network

Let's run one step on a tiny network. One input, x equals 1, goes through weight w1, then a sigmoid, then weight w2, to the output y-hat. The target is 1, the weights start at 0.5 and 0.8, the loss is one half of the squared gap, and the learning rate is 0.5. Forward: z is 0.5 times 1, so 0.5. The sigmoid of 0.5 is 0.622. y-hat is 0.8 times 0.622: 0.498. And the loss is one half of 0.498 minus 1, squared: 0.126.

Worked Example

Backward Pass One Local Rate at a Time

Now backward, multiplying one local rate per step. Start at the loss: its rate with respect to y-hat is y-hat minus y, which is minus 0.502. w2's gradient is that, times a: minus 0.312. Passing left through w2 instead gives the gradient at a: minus 0.502 times 0.8, so minus 0.402. The sigmoid's own rate is a times one minus a: 0.235. So the gradient at z is minus 0.094. And since x is 1, w1's gradient is also minus 0.094.

Worked Example

Updating Weights Lowers the Loss

Both gradients are negative, so both weights rise. w1 becomes 0.5, minus 0.5 times minus 0.094: 0.547. w2 becomes 0.8, minus 0.5 times minus 0.312: 0.956. Run the forward pass again: y-hat is now 0.606, and the loss has dropped from 0.126 to 0.078. Training just repeats this loop. Deep networks use the same bookkeeping, one local rate per step, from the loss back to the first layer.

Quick Check

Which Way Weights Move With Target 0

Quick check. Same tiny network, same starting weights, same input. But now the target is 0, not 1. After one backpropagation step, which way do the two weights move? Pause, and predict. Both fall. The first factor is now y-hat minus 0: plus 0.498. The prediction is too high. Every other factor, a, w2, the sigmoid's rate and x, is positive. So both gradients are positive, and both weights go down. Splitting them is tempting, but w1 reaches the loss through the same positive chain as w2.

Learning Rate

Too Small, About Right, or Too Large

The learning rate is the most important setting in training. Too small, and every step is safe but tiny: training crawls, and may stop long before it's done. About right, and the loss falls quickly and settles near the bottom. Too large, and each step overshoots, landing higher on the other side. The loss bounces, and can grow without limit. A common approach: try values ten times apart, like 0.1, 0.01 and 0.001, and watch the loss curve. Many runs also shrink the rate over time.

Momentum

A Heavy Ball Rolling Downhill

Plain gradient descent only reacts to the slope where it stands. In a long, narrow valley, that slope points mostly across, so the path zigzags between the walls. Momentum keeps a running velocity, like a heavy ball rolling downhill. Each new velocity blends the previous one with the new gradient, typically 0.9 of the old. Zigzags partly cancel, and a steady downhill direction builds speed.

Adam Optimizer

Adaptive Step Sizes for Each Weight

Adam, short for adaptive moment estimation, adds another idea: every weight gets its own step size. For each weight, it keeps two running averages: one of the gradient, like momentum, and one of the squared gradient, a measure of how big the gradients usually are. Weights with large, jumpy gradients take smaller steps. Small, steady ones take larger steps. Both averages start at zero, so early steps are corrected for that. The usual settings are 0.9 and 0.999 for the two averages, with a learning rate of 0.001. Adam is a popular default, but no optimizer wins on every problem.

Vanishing Gradients

When Products Shrink Toward Zero

Deep networks bring a problem. Backpropagation multiplies one local rate per layer. If those factors are usually below 1, the product shrinks exponentially as it travels back. The early layers get almost no learning signal. That's the vanishing gradient problem. The sigmoid is a classic cause: its rate of change is never more than 0.25. Ten layers of 0.25 leave 0.25 to the tenth: about one millionth. The reverse, exploding gradients, happens when the factors are above 1.

Remedies

ReLU, Starting Weights, Shortcuts, and Clipping

Common remedies. Rectified linear units have a rate of 1 for positive inputs, so the signal passes back unchanged. Careful starting weights keep signals and gradients a steady size from layer to layer. Shortcut, or residual, connections give the gradient a direct path around several layers. And gradient clipping caps a gradient's size, which fights exploding gradients.

Regularization

Weight Decay, Dropout, and Early Stopping

Finally, honesty. A network with many weights can memorize its training examples: training loss keeps falling, while performance on new data gets worse. That's overfitting. Regularization reduces it. Weight decay adds a penalty for large weights to the loss, usually lambda times the sum of squared weights. Smaller weights give smoother predictions. Dropout switches off a random share of hidden units, such as half, on every training step. No unit can rely on a partner. At test time all units work, scaled to match. And early stopping watches the loss on held-out validation data. When it stops improving for a set patience, training ends, and the best weights are kept.

Quick Check

Early Stopping With Patience 5

Quick check. A network trains for many epochs. Training loss falls every epoch, but validation loss is lowest at epoch 12, then rises. Early stopping uses a patience of 5 epochs. When does training stop, and which epoch's weights are kept? Pause, and decide. It stops at epoch 17, and keeps epoch 12's weights. It can't know 12 was best until five epochs pass with no improvement. Keeping epoch 17 is tempting, but those weights are five epochs into overfitting. The best weights seen so far are the ones saved.

Key Takeaways

Training Neural Networks Summed Up

To sum up. A loss scores mistakes, and gradient descent moves each weight against its gradient, in steps set by the learning rate. Mini-batches balance speed and noise. Backpropagation applies the chain rule in one backward sweep to get every gradient. Momentum and Adam make steps steadier and better scaled. And vanishing gradients starve early layers, while weight decay, dropout and early stopping fight overfitting.

Reflection

Learning Rate in Free Throws

Something to think about. Practice free throws, or darts. After each throw, you see how far off you were, and adjust a little. Which part of that routine is the learning rate? What happens if you adjust far too much after each miss? And what would overfitting look like?

About Introduction to Artificial Intelligence at UPI Study

Introduction to Artificial Intelligence (Computer Science 373) is a self-paced online course of 69 video lessons in 9 chapters, from intelligent agents and search to game strategy, neural networks, logic and reasoning, machine learning, language, robotics and AI ethics. It carries a NCCRS credit recommendation for 3 semester credits.

Every lesson transcript is free on pages like this one, and the videos are free with a free UPI Study account. Enrolling adds a practice quiz after each lesson, the graded work and final exam that earn the credit, lifetime access, and an official transcript on completion. UPI Study has helped over 130,000 students with college credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Enroll now → Course details Download the syllabus All Computer Science courses Every course with PRO →

Compare: AI vs Current Trends in CS and IT

Why take it here

Why students take Introduction to Artificial Intelligence with UPI Study

How machines search, reason and learnIntroduction to Artificial Intelligence covers how AI systems are built: intelligent agents, search and heuristics, game strategy, logic, neural networks, machine learning, language and perception, and the ethics of AI. It is a common upper-level course in computer science degrees.
Credit at a fraction of the priceThe same 3 credits can cost $750 or more at a US college. Here it is one payment, self-paced, with lifetime access.
Finish your degree soonerTransfer credits reduce the courses you still owe, so you can graduate earlier or lighten a heavy semester.
Strengthen an applicationA completed college-level course on an official transcript shows admissions and scholarship committees you can do the work.
Study on your own scheduleNo start dates, no deadlines. Watch the lessons, practice, and take the final when you are ready.
130,000+students helped with college creditsACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Where the credit goes

Universities that work with NCCRS

Introduction to Artificial Intelligence carries a NCCRS credit recommendation. These universities work with NCCRS; some are UPI Study's signed partners. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Thomas Edison State UniversityThomas Edison State UniversitySigned partnerUniversity of the PeopleUniversity of the PeopleSigned partnerCharter Oak State CollegeCharter Oak State CollegeSigned partnerStrayer UniversityStrayer UniversitySigned partnerColumbia Southern UniversityColumbia Southern UniversitySigned partnerIndiana TechIndiana TechSigned partnerPenn StatePenn StateWorks with NCCRSSouthern New Hampshire UniversitySouthern New Hampshire UniversityWorks with NCCRSSUNYSUNYWorks with NCCRS +many more cooperating universitiesFind my college →

Find my college → Not listed? Ask Athena in the corner of this page, or book a 15-minute call.

FAQ

Questions students ask us

About Training Networks: Backpropagation and Gradient Methods

What is gradient descent in machine learning?
Gradient descent is a training method that lowers a loss by moving each weight a small step opposite to its gradient. The update is the new weight equals the old weight minus the learning rate times the gradient. A learning rate that is too small makes training crawl, while one that is too large overshoots, so the loss bounces or grows.
How does backpropagation work?
Backpropagation computes the gradient of the loss for every weight with one forward pass and one backward pass, using the chain rule. The forward pass computes and stores every value from input to loss. The backward pass starts at the loss and, at each step, multiplies the incoming gradient by that step's local rate of change, passing it toward the input layer.
What is the vanishing gradient problem?
The vanishing gradient problem occurs when backpropagation multiplies many local rates below 1, so the gradient shrinks exponentially and early layers barely learn. The sigmoid's rate is at most 0.25, and ten such layers leave about one millionth of the signal. Remedies include ReLU activations, careful starting weights, and residual shortcut connections. Gradient clipping addresses the opposite problem, exploding gradients.
What is the difference between batch, stochastic, and mini-batch gradient descent?
They differ in how many examples feed each gradient estimate. Batch training uses all examples for one smooth but slow step per epoch. Stochastic gradient descent uses one example at a time, giving many fast but jittery updates. Mini-batch training uses a small group, such as 50 of 1,000 examples for 20 updates per epoch, balancing noise and efficiency, and is the usual choice.

About the Introduction to Artificial Intelligence course

How many college credits is Introduction to Artificial Intelligence?
Introduction to Artificial Intelligence (Computer Science 373) carries a NCCRS credit recommendation for 3 semester credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.
How much does Introduction to Artificial Intelligence cost and what is included?
$250, paid once, with lifetime access: all 69 video lessons, a practice quiz after each, the graded work and final exam, and an official transcript when you finish. Students who want several courses choose UPI Study PRO instead.
Which universities accept UPI Study credits?
UPI Study has signed partner agreements with universities such as Thomas Edison State University, University of the People and Charter Oak State College, and works with universities such as Penn State, SNHU and SUNY. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities. Use "Find my college" on this page to check yours.
How long does Introduction to Artificial Intelligence take?
It is self-paced with no deadlines. The 69 lessons total about 16 hours of video; most students add reading and practice time and finish in a few weeks to a few months, on their own schedule.
How are the exams done?
Quizzes and assignments are online in the course. The final exam is taken online with proctoring; a person on the UPI Study team checks assignments, usually within two days.
How do my credits reach my college?
When the course shows 100% complete, request the official transcript from your dashboard. It is sent to the college you name, and you can send it to more than one. Check your college first with "Find my college" on this page, or ask Athena.

Ready for the full course? Earn 3 credits in Introduction to Artificial Intelligence.

69 video lessons, a quiz after each one, and the graded work that earns the credit. Recommended for credit by NCCRS.

Enroll in Introduction to Artificial Intelligence →Download the syllabusEvery course with PRO →
Introduction to Artificial Intelligence
3 credits · NCCRSEnroll →