Sequence-Aware Learning with Recurrent Networks, Explained

UPI Study › Courses › Introduction to Artificial Intelligence › Free lessons › Lesson 5.7 Free with a free account

Sequence-Aware Learning with Recurrent Networks

Introduction to Artificial Intelligence · Chapter 5: Neural Networks and Learning ArchitecturesSection 5.7 · lesson 38 of 6916 min · free with a free account

← Image-Focused Learning with Convolutional NetworksNext lesson: Analytical Thinking and the Role of Mathematical Logic →

Earn 3 college credits for Introduction to Artificial IntelligenceRecommended for credit by NCCRS · Computer Science 373 · self-paced · one payment, lifetime access
Enroll in Introduction to Artificial Intelligence · $250 →Download the syllabus (PDF)

In this lesson

How recurrent networks, LSTMs and GRUs carry memory through sequences.

  • Explain how a hidden state carries information from one time step to the next
  • Describe backpropagation through time and why gradients vanish or explode
  • Compute an LSTM cell state update from forget and input gate values
  • Match sequence tasks to many-to-one, one-to-many and many-to-many patterns

Free study notes

Get this lesson's notes by email

The key points and 4 practice questions with answers, in your inbox. One email, free.

One email. Unsubscribe in one click.

Transcript

Everything said in the video, section by section (2,146 words). Read along, or search it.

Opening Question

Same Words, Opposite Meanings

The dog chased the cat. The cat chased the dog. Exactly the same words, and opposite meanings. A temperature of 30 degrees means one thing after a week of heat, and another after a week of snow. In data like this, the order of the items, and everything that came before, carries the meaning. So how can a network remember what came before?

Motivation

Giving a Network a Memory

The networks we've met so far take one fixed-size input, and give one answer, with no memory of earlier inputs. They're poorly suited to data that arrives as a sequence. Recurrent neural networks, or RNNs, solve this by giving the network a memory, passed from one step to the next.

Definition

Why Sequences Challenge Ordinary Networks

A sequence is an ordered list of items, where position matters: words in a sentence, notes in a song, daily prices, heartbeats, letters of DNA. Sequences are awkward for ordinary networks, in three ways. Their length varies: one sentence has 5 words, the next has 40. Order carries meaning: shuffle the items, and the message changes or breaks. And distant items depend on each other. In, I grew up in France, so I speak fluent, blank, the best guess depends on France, five words earlier.

Comparison

Recurrent Versus Ordinary Networks

Here's how the two kinds of network compare. An ordinary network takes a fixed input size. A recurrent one takes any length: it just keeps taking steps. The ordinary network has no memory of earlier inputs. The recurrent one holds them in a hidden state. It reads items one at a time, in order, so order is built in. And it reuses the same weights at every step, instead of separate weights for each position.

Core Mechanism

The Hidden State Update with Tanh

At each step, a recurrent network combines two things: the new input, x of t, and a summary of everything so far, called the hidden state. It squashes a weighted sum through tanh: h of t equals tanh of, Wx times x of t, plus Wh times the previous h, plus b. When an output is needed, it's computed from h. The hidden state starts at zero, and every step uses the same Wx, Wh, and b. Think of it as the notes you keep while reading a long story. The notes are all you carry forward, and they have limited room, so the network must learn what's worth remembering.

Visualization

Unrolling the Loop Through Time

A recurrent network is often drawn as one cell with a loop. The loop is easier to follow when it's unrolled. Copy the cell once for every time step, and pass the hidden state from each copy to the next. Now it looks like a chain of ordinary layers, one per step. But the copies aren't separate networks. They share the same weights. Unrolling is a way of drawing, and later of computing, how one set of weights acts at each step.

Parameter Count

Weight Sharing Keeps the Model Small

Weight sharing keeps the model small. Say each step has 3 input numbers, and the hidden state has 4. Wx is 4 by 3: 12 numbers. Wh is 4 by 4: 16. The bias adds 4. That's 32 for the recurrent layer. An output layer with 2 outputs adds 8 weights and 2 biases, so the whole network has 42. And that count is the same for 5 steps or 5,000 steps. Separate weights for each of 1,000 positions would need 32,000, for the recurrent layer alone.

Worked Example

Watching a Hidden State Fade

Now watch a hidden state fade. One input number, a hidden state of one number, Wx equals 1, b equals 0. The input is 1 at the first step, then 0. Step one gives tanh of 1: 0.762, whatever the recurrent weight. With Wh equal to 0.5, step two is tanh of 0.381: 0.363. Then 0.180, 0.090, and by step six, 0.022. The first input is almost gone. With Wh equal to 1.5, it stays: 0.815, 0.840, and still 0.857 at step six. That one number controls how long information lasts.

Quick Check

Zero Recurrent Weight Means No Memory

Quick check. Same setup, but now the recurrent weight Wh is 0. The input is 1 at step one, then 0. What is the hidden state at step two? Pause, and work it out. Zero. Step two is tanh of, 0 plus 0 times 0.762, which is tanh of 0: zero. With no recurrent weight, there's no memory at all. 0.762 would mean the state just carried over unchanged. But the old state only gets in through Wh, and here Wh blocks it.

Training

Backpropagation Through Time

How is a recurrent network trained? As with any network: compare the output with the correct answer to get a loss, then use backpropagation. The new twist is time. Unroll the network across the whole sequence, treat it as one very deep network whose layers share weights, and run ordinary backpropagation. That's backpropagation through time. The forward pass runs the sequence step by step, computing each hidden state. The backward pass sends the error signal back, step by step, to the first step. And because the weights are shared, each weight's final adjustment is the sum of its contributions from every step.

Worked Numbers

Signals Shrink or Grow Over Steps

Here's the catch. For an input many steps back to affect the loss, the signal must travel back through every step in between. At each step it's multiplied by a factor, which depends on the recurrent weight and the slope of the activation. Go back k steps, and the signal is roughly the factor to the power k. With a factor of 0.5, after 10 steps it's 0.001. With 0.9, it's 0.35 after 10, and 0.005 after 50. A factor of exactly 1 keeps it, but rarely stays exact. With 1.1, it's 117 after 50 steps. And with 1.5, it's about 638 million.

Key Problem

Vanishing and Exploding Gradients

These are two sides of one problem. Vanishing gradients: the factor is below 1, so the signal reaching early steps shrinks toward zero. The network only learns short-range patterns. Exploding gradients: the factor is above 1, so the signal grows enormously. Weight updates become huge, training turns unstable, and the loss can become an unusable value. The slope of tanh is never more than 1, and much smaller for large inputs. So simple recurrent networks tend to vanish. In practice they struggle beyond perhaps 10 to 20 steps.

Remedies

Gradient Clipping and Truncated Training

Each problem has a usual remedy. Exploding gradients make the loss jump wildly. The fix is gradient clipping: if a gradient is larger than a chosen limit, scale it down. Very long sequences make training slow and memory-hungry. Truncated backpropagation through time unrolls only a window of steps, say 50, at a time. That saves effort, but limits how far back learning reaches. And for vanishing gradients, better starting weights and activations help a little. The strongest fix is a gated design.

LSTM

Three Gates Edit the Cell State

The Long Short-Term Memory network, or LSTM, was introduced by Sepp Hochreiter and Jürgen Schmidhuber in 1997, to fix vanishing gradients. It adds a second memory line, the cell state, that runs through time with only small, controlled edits, like a conveyor belt carrying information along. Gates control the edits. A gate is a small layer with a sigmoid activation. Its output, between 0 and 1, acts like a valve: 0 blocks everything, 1 lets everything through. The forget gate asks how much old memory to keep: which old notes to erase. The input gate asks how much new information to write: which new notes to add. And the output gate asks how much of the memory to show, to the rest of the network now: which notes to read aloud. In symbols, the new cell state c is the forget gate times the old cell state, plus the input gate times the new idea. Then h is the output gate, times tanh of c. Suppose the old cell state is 2.0, the forget gate gives 0.9, the input gate 0.5, and the new idea is 0.6. Then c is 0.9 times 2, plus 0.5 times 0.6: 1.8 plus 0.3, which is 2.1. With an output gate of 0.8, h is 0.8 times tanh of 2.1: 0.776. And if the forget gate had been 0, c would be just 0.3. The old memory wiped.

Why It Works

Additive Updates Preserve Memory

Why do gates help? The cell update is additive: the old state, scaled by the forget gate, plus something new. When the forget gate is near 1, the old memory, and the gradient flowing back through it, passes almost unchanged, instead of being squashed at every step. A forget gate of 0.9 is no better than a simple network: 0.005 left after 50 steps. But at 0.99, 0.61 survives. And at 1, all of it. And the network learns to keep the forget gate near 1, for information that must last.

Quick Check

A New Cell State of 0.82

Quick check, one LSTM memory cell. The old cell state is 1.0. The forget gate gives 0.5, the input gate 0.4, and the new idea is 0.8. What's the new cell state? Pause, and work it out. Zero point eight two. The forget gate keeps half the old memory: 0.5 times 1.0 is 0.5. The input gate lets in 0.4 times 0.8, which is 0.32. Together, 0.82. 1.32 forgets to apply the forget gate. And 1.30 lets the new idea in without the input gate.

GRU

Two Gates in a Lighter Cousin

The Gated Recurrent Unit, or GRU, proposed in 2014 by Kyunghyun Cho and colleagues, is a simpler cousin. It merges the cell state and the hidden state into one, and uses two gates. The update gate decides how much of the old hidden state to keep, and how much to replace with new content. The reset gate decides how much of the old state to use, when making that new content. Compare parameters for 3 inputs and 4 hidden units. A simple RNN has 32. An LSTM has four sets of weights: 128. A GRU has three: 96. In practice, the LSTM and the GRU often perform similarly. Neither wins every study, so many teams try both. The GRU is lighter, and faster to train.

Bidirectional Networks

Reading Forward and Backward

A standard recurrent network reads only left to right, so at each step it knows the past, but not the future. Yet the words that follow often explain the ones before. In, he sat on the bank of the, blank, the word bank is clarified by what comes after it. A bidirectional network runs two recurrent layers over the same sequence: one reads forward, and one reads backward. At each step, their hidden states are joined, so the output sees past and future. That works when the whole sequence is available first: tagging a recorded sentence, or filling a gap in a text. It doesn't work when predictions must come as data arrives, like the next word as someone types.

Input-Output Patterns

One-to-Many, Many-to-One, Many-to-Many

Recurrent networks are flexible, because the number of inputs and outputs can differ. Many-to-one: a whole sequence in, one answer at the end, like the sentiment of a review. One-to-many: one input starts a sequence of outputs, like music generated from a starting note, or a sentence describing an image. Many-to-many, same length: an output at every step, like labeling each word's part of speech, or each frame of a video. And many-to-many, different length: read the whole input, then write an output of another length, like translating or summarizing. Language and speech methods build on these, in later lessons.

Current Use

Where Recurrent Networks Stand Today

Where do recurrent networks stand today? Newer designs built on a mechanism called attention have taken over many language tasks, because they handle very long sequences and train more easily in parallel. Recurrent networks are still widely used for time series, small models, and streaming data. And their ideas, a hidden state, gates that control memory, any length, remain basic to sequence learning.

Quick Check

Labeling Every Second of a Call

Quick check. A call center records whole calls. Afterward, a network labels each second of the recording: customer speaking, agent speaking, or silence. Which pattern is that, and can the network read in both directions? Pause, and decide. One label for every second means an output at every step: many-to-many, same length. And yes, it can read both ways, because the whole recording exists before any label is needed. Many-to-one would give a single answer for the whole call. And the no would be right for a live system, labeling as the call happens.

Key Takeaways

Hidden States, Gates and Patterns

To sum up. A recurrent network reads a sequence one step at a time, carrying a hidden state, with the same weights at every step. Backpropagation through time multiplies the error signal at every step back, so it can vanish or explode. Clipping and truncation help. LSTM gates let memory pass almost unchanged. The GRU does it with two gates. Bidirectional networks need the whole sequence. And tasks fall into one-to-many, many-to-one, and many-to-many patterns.

Think About It

Heart Rate Goals and Patterns

Something to think about. A fitness band records heart rate every second. Which pattern fits, and would you read both ways, for each of three goals? Warning the wearer in real time if the rate looks unsafe. Labeling each minute of a finished workout as rest, jogging, or sprinting. And deciding whether a whole night's sleep was restful.

About Introduction to Artificial Intelligence at UPI Study

Introduction to Artificial Intelligence (Computer Science 373) is a self-paced online course of 69 video lessons in 9 chapters, from intelligent agents and search to game strategy, neural networks, logic and reasoning, machine learning, language, robotics and AI ethics. It carries a NCCRS credit recommendation for 3 semester credits.

Every lesson transcript is free on pages like this one, and the videos are free with a free UPI Study account. Enrolling adds a practice quiz after each lesson, the graded work and final exam that earn the credit, lifetime access, and an official transcript on completion. UPI Study has helped over 130,000 students with college credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Enroll now → Course details Download the syllabus All Computer Science courses Every course with PRO →

Compare: AI vs Current Trends in CS and IT

Why take it here

Why students take Introduction to Artificial Intelligence with UPI Study

How machines search, reason and learnIntroduction to Artificial Intelligence covers how AI systems are built: intelligent agents, search and heuristics, game strategy, logic, neural networks, machine learning, language and perception, and the ethics of AI. It is a common upper-level course in computer science degrees.
Credit at a fraction of the priceThe same 3 credits can cost $750 or more at a US college. Here it is one payment, self-paced, with lifetime access.
Finish your degree soonerTransfer credits reduce the courses you still owe, so you can graduate earlier or lighten a heavy semester.
Strengthen an applicationA completed college-level course on an official transcript shows admissions and scholarship committees you can do the work.
Study on your own scheduleNo start dates, no deadlines. Watch the lessons, practice, and take the final when you are ready.
130,000+students helped with college creditsACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Where the credit goes

Universities that work with NCCRS

Introduction to Artificial Intelligence carries a NCCRS credit recommendation. These universities work with NCCRS; some are UPI Study's signed partners. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Thomas Edison State UniversityThomas Edison State UniversitySigned partnerUniversity of the PeopleUniversity of the PeopleSigned partnerCharter Oak State CollegeCharter Oak State CollegeSigned partnerStrayer UniversityStrayer UniversitySigned partnerColumbia Southern UniversityColumbia Southern UniversitySigned partnerIndiana TechIndiana TechSigned partnerPenn StatePenn StateWorks with NCCRSSouthern New Hampshire UniversitySouthern New Hampshire UniversityWorks with NCCRSSUNYSUNYWorks with NCCRS +many more cooperating universitiesFind my college →

Find my college → Not listed? Ask Athena in the corner of this page, or book a 15-minute call.

FAQ

Questions students ask us

About Sequence-Aware Learning with Recurrent Networks

What is a recurrent neural network?
A recurrent neural network, or RNN, is a network that reads a sequence one item at a time and carries a hidden state from step to step. At each step it computes h(t) = tanh(Wx x(t) + Wh h(t-1) + b), reusing the same weights every time. This lets it handle inputs of any length, with the order of items built in.
What causes the vanishing gradient problem in RNNs?
Vanishing gradients happen because the error signal is multiplied by a factor at every step it travels back in time. If that factor is below 1, the signal shrinks roughly as the factor to the power k; with 0.5, it falls to about 0.001 after 10 steps. Since the slope of tanh is at most 1, simple RNNs often struggle beyond about 10 to 20 steps.
How does an LSTM cell update its memory?
An LSTM sets the new cell state to the forget gate times the old cell state, plus the input gate times the new candidate value. With an old state of 2.0, a forget gate of 0.9, an input gate of 0.5 and a new idea of 0.6, the cell state becomes 1.8 + 0.3 = 2.1. The hidden state is then the output gate times tanh of the cell state.
What is the difference between an LSTM and a GRU?
A GRU merges the cell state and hidden state into one and uses two gates, update and reset, while an LSTM keeps a separate cell state and uses three gates. The LSTM was introduced by Hochreiter and Schmidhuber in 1997, and the GRU by Kyunghyun Cho and colleagues in 2014. With 3 inputs and 4 hidden units, an LSTM has 128 parameters and a GRU 96.

About the Introduction to Artificial Intelligence course

How many college credits is Introduction to Artificial Intelligence?
Introduction to Artificial Intelligence (Computer Science 373) carries a NCCRS credit recommendation for 3 semester credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.
How much does Introduction to Artificial Intelligence cost and what is included?
$250, paid once, with lifetime access: all 69 video lessons, a practice quiz after each, the graded work and final exam, and an official transcript when you finish. Students who want several courses choose UPI Study PRO instead.
Which universities accept UPI Study credits?
UPI Study has signed partner agreements with universities such as Thomas Edison State University, University of the People and Charter Oak State College, and works with universities such as Penn State, SNHU and SUNY. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities. Use "Find my college" on this page to check yours.
How long does Introduction to Artificial Intelligence take?
It is self-paced with no deadlines. The 69 lessons total about 16 hours of video; most students add reading and practice time and finish in a few weeks to a few months, on their own schedule.
How are the exams done?
Quizzes and assignments are online in the course. The final exam is taken online with proctoring; a person on the UPI Study team checks assignments, usually within two days.
How do my credits reach my college?
When the course shows 100% complete, request the official transcript from your dashboard. It is sent to the college you name, and you can send it to more than one. Check your college first with "Find my college" on this page, or ask Athena.

Ready for the full course? Earn 3 credits in Introduction to Artificial Intelligence.

69 video lessons, a quiz after each one, and the graded work that earns the credit. Recommended for credit by NCCRS.

Enroll in Introduction to Artificial Intelligence →Download the syllabusEvery course with PRO →
Introduction to Artificial Intelligence
3 credits · NCCRSEnroll →