How Machines Understand Spoken Language

UPI Study › Courses › Introduction to Artificial Intelligence › Free lessons › Lesson 8.1 Free with a free account

How Machines Understand Spoken Language

Introduction to Artificial Intelligence · Chapter 8: Perception, Robotics, and Hands-On AISection 8.1 · lesson 57 of 6915 min · free with a free account

← Modeling Change Over Time: Dynamic Probabilistic SystemsNext lesson: Teaching Machines to Recognize and Sort Images →

Earn 3 college credits for Introduction to Artificial IntelligenceRecommended for credit by NCCRS · Computer Science 373 · self-paced · one payment, lifetime access
Enroll in Introduction to Artificial Intelligence · $250 →Download the syllabus (PDF)

In this lesson

How speech recognition turns sound into text, and text into speech.

  • Describe the speech recognition pipeline from sampled signal to features, acoustic model, language model and decoder
  • Explain how MFCC features and a language model help a recognizer choose between similar-sounding phrases
  • Calculate word error rate from substitutions, deletions and insertions
  • Compare classical HMM-based and end-to-end recognizers, and outline how text-to-speech works

Free study notes

Get this lesson's notes by email

The key points and 4 practice questions with answers, in your inbox. One email, free.

One email. Unsubscribe in one click.

Transcript

Everything said in the video, section by section (2,087 words). Read along, or search it.

Opening Question

From a Spoken Request to a Timer

Set a timer for ten minutes. Within a second, a small speaker has turned a wave of air into words, understood you, and started a countdown. Reading text is easy for a computer. Listening is not. Speech has no gaps between words. It sounds different from every speaker, and it arrives mixed with noise. So how does a machine get from a wave in the air, to a countdown? That journey is called automatic speech recognition: turning audio into text.

The Signal

Sound Becomes a List of Samples

Start with the sound itself. Sound is a rapidly changing pressure in the air. A microphone turns it into a changing voltage, and a converter measures that voltage thousands of times a second. Each measurement is a sample. The number of samples per second is the sampling rate. So the computer never hears a sound. It receives a long list of numbers.

Sampling Rate

Half the Rate Sets the Frequency Limit

The rate sets a limit. A signal sampled at rate r can only represent frequencies up to half of r. That's why a higher rate keeps more of the high-pitched detail. Old telephone lines use 8,000 samples a second, so they capture sounds up to 4,000 hertz. Speech recognizers typically use 16,000 a second, which reaches 8,000 hertz.

Phonemes

Smallest Sounds That Change a Word

At 16,000 samples a second, two bytes each, a second of speech is 32,000 bytes. Most of that is detail words don't need, like the exact timing of each wave. What matters are phonemes: the smallest units of sound that tell one word from another. English uses about forty. Beach has three phonemes: the B sound, the long E, and the CH. Change only the first one, and you get a different word: peach. A pronunciation lexicon lists the phonemes of every word. That's how a recognizer links sounds to words.

Recognition Pipeline

Stages From Signal to Intent

A recognizer breaks the problem into stages, and each stage hands the next something more useful. First, the signal: the wave captured as numbers. Then features, which summarize which frequencies are present, about a hundred times a second. Then the acoustic model, which scores how likely each speech sound is, moment by moment. The decoder searches for the word sequence that best explains the audio: the transcript. Then intent understanding works out what the speaker wants. The decoder also draws on knowledge that isn't in the audio at all: a language model, and a pronunciation lexicon. Both feed the decoder's search.

How Speech Is Made

Formants and Short Overlapping Frames

Features start from how speech is made. The vocal cords vibrate, and the throat, mouth and nose shape that sound. That shape acts like a filter. It strengthens certain frequencies, called formants. Different vowels have their formants in different places. That's mostly what makes ee sound unlike ah. Speech changes quickly, but it's almost steady over about 25 milliseconds. So the signal is cut into short, overlapping frames, and analyzed one frame at a time.

Spectrogram

Watching Formants Move Between Vowels

A spectrogram makes those formants visible. It shows how the energy at each frequency changes over time. Here's a sketch for an artificial sound: ah, then ee, then oo, each lasting less than half a second. For ah, the first two formants sit close together, in the middle. For ee, they spread far apart. For oo, both drop low. The bands move from vowel to vowel, exactly as the formant values predict. A raw waveform shows none of this clearly.

MFCC Features

Six Steps to 13 Coefficients

Recognizers usually compress this further, into MFCCs: mel-frequency cepstral coefficients. They take six steps. First, boost the high frequencies a little. Second, cut the signal into 25-millisecond frames, one every 10 milliseconds. Third, find the energy at each frequency. Fourth, add it up in 26 bands on the mel scale, which, like hearing, is fine at low pitches and coarse at high ones. Fifth, take the log of each band, to match how loudness is perceived. Sixth, keep only 13 numbers that describe the smooth outline of the spectrum, and drop fine detail such as the pitch of the voice.

Why MFCCs Help

Fewer Numbers That Describe Sound Better

A second of speech gives 98 frames of 13 numbers: 1,274 numbers instead of 16,000, about 12.6 times fewer. And they describe the sound better. In a small test with artificial vowels, a bigger distance between MFCCs means a bigger difference. Doubling the pitch of the same ah moved the MFCCs by about 3. Changing ah to ee moved them by about 18. So MFCCs capture what is being said, and mostly ignore how high the voice is. That's exactly what a recognizer needs.

Acoustic Model

Recognize Speech or Wreck a Nice Beach

Now the acoustic model. It looks at the feature vectors and estimates how likely each phoneme is at each moment. It's trained on many hours of recorded speech with transcripts. But on its own, it can't tell apart phrases that sound alike. Take two famous look-alikes: recognize speech, and wreck a nice beach. Written as phonemes, the two sequences are only three edits apart. To the acoustic model, they're nearly a tie.

Language Model

Bigram Scores Break the Tie

The language model breaks the tie. It judges how likely a word sequence is as language. A simple version is a bigram model: the chance of each word, given the word before it, counted from a collection of text. In a tiny collection of 20 invented sentences, recognize speech appears six times, nice beach once, and wreck a never. The decoder adds the two: the log of the acoustic score, plus a weight L times the log of the language score. Here, L is one.

Worked Example

Combined Scores Pick Recognize Speech

Here are four candidates. The acoustic scores are illustrative: nearly tied, with a slight edge to the wrong answer. On sound alone, wreck a nice beach is best, at minus 41.7. Add the language scores, and recognize speech wins clearly, at minus 48.74. Wreck a nice beach falls to minus 56.36. The language model supplies what the audio can't: knowledge of how people actually talk.

Quick Check

Buy Ice Cream or Buy I Scream

Quick check. A shopper speaks to a voice assistant. Two candidates come out of the acoustic model. Buy I scream scores minus 20.3 on sound, and minus 9.8 on language. Buy ice cream: minus 20.5, and minus 4.1. With a weight of one, which transcript does the decoder pick? Pause, and decide. The decoder picks buy ice cream. Add each row: minus 20.5 plus minus 4.1 is minus 24.6. Buy I scream totals minus 30.1. Its tiny acoustic edge, two tenths, is swamped by a language gap of almost six. Judging on sound alone is the tempting mistake. When the sounds nearly tie, the language model decides.

Decoding

Beam Search Keeps the Best Paths

The decoder can't score every possible transcript. With a vocabulary of 50,000 words, there are about ten to the forty-seventh possible ten-word sequences. So it uses beam search. Moving through the audio, it keeps only the best few partial transcripts, say the best eight, and throws the rest away. In this small example, each step has two choices. Weak paths are cut early, and only the strongest survives to the end.

Word Error Rate

Substitutions, Deletions and Insertions

How good is a recognizer? The standard score is the word error rate. Line the transcript up with the correct text, and count substitutions, deletions and insertions. Then divide by the number of words in the correct text. Turn on the kitchen lights, heard as turn on a kitchen light: two substitutions in five words, forty percent. Set an alarm for seven, heard as set alarm for seven: one deletion, twenty percent. Lower is better. And with many insertions, the rate can even go past one hundred percent.

Quick Check

Scoring Call Mom on Her Cell

Quick check. Someone says: call mom on her cell. The recognizer writes: call Tom on the cell phone. What's the word error rate? Pause, and work it out. It's sixty percent. Mom became Tom, and her became the: two substitutions. Phone is an extra word: one insertion. Three errors, divided by the five words that were actually said. Forty percent forgets the insertion. Fifty percent divides by the six words heard, but the rate always divides by the correct text.

Classical Design

Hidden Markov Model Pipeline

Two broad designs have dominated speech recognition. The classical one is the pipeline we've drawn, built on hidden Markov models. A hidden Markov model treats each phoneme as a short chain of hidden states, which the sound passes through. The acoustic model scores how well each frame fits each state, and the decoder chains states into phonemes, and phonemes into words.

End-to-End Design

CTC Blanks and Collapsing Rules

An end-to-end system replaces most of that with one neural network, from the features, or even the raw signal, straight to letters or words. It's trained as a whole, on pairs of audio and transcript. But nobody marks which frame belongs to which letter. One widely used answer is CTC. For every frame, the network outputs a letter, or a special blank. Then a simple rule collapses the result: merge repeated letters, then delete the blanks. So this first string becomes hello. Without a blank between the two l's, they would merge, and you'd get h e l o.

Comparison

Pipeline Versus One Neural Network

How do the two designs compare? The classical pipeline needs a pronunciation lexicon. An end-to-end network can learn to spell directly. In the pipeline, you can fix one part, say by adding a word to the lexicon. In one big network, that's harder. End-to-end systems need many hours of transcribed audio. With enough of it, they've matched or beaten the classical pipeline on many benchmarks. Hybrids that keep some separate parts, such as a language model, remain common.

Text-to-Speech

Normalization, Pronunciation and Prosody

Now the reverse trip. Text-to-speech turns written text into audio, deciding not just which sounds to make, but how to say them. Text normalization turns numbers and abbreviations into words. Doctor Lee lives at 221 Baker Street. But St. could also mean Saint, and Dr. could mean Drive. Pronunciation turns words into phonemes. Read rhymes with reed in I read daily, and with red in I read it yesterday. Prosody chooses pitch, rhythm and pauses: a rising pitch turns a statement into a question. Finally, the waveform. It can be stitched from recorded snippets, driven by a statistical model, or generated by a neural network: the most natural, and open to misuse imitating real voices.

Intent Understanding

Intents, Slots and Silent Errors

A transcript isn't an action yet. Intent understanding works out the kind of request, the intent, and its details, the slots. Set an alarm for seven a m becomes set alarm, with a time slot. This can use hand-written patterns, or a trained classifier. Recognition errors flow straight into it. Set an alarm for heaven a m matches nothing. That's a visible failure, and the system can simply ask again. Turn off the lights, when you said on, is dangerous: the opposite action. And weather in pairs, for Paris, parses fine with the wrong city: a silent error.

Safeguards

N-Best Lists and Confirmation

The silent errors are the worst, because nothing signals that something went wrong. So practical systems pass along several alternative transcripts, an n-best list, each with a confidence score. And they ask for confirmation when confidence is low, or when the action is risky.

Why Recognizers Fail

Accents, Noise and Speaker Differences

Why do recognizers still fail? They work best when conditions resemble their training data, and real life rarely cooperates. Accents and dialects change vowels, stress and rhythm, and some groups are under-represented in training data. The remedy: train on many accents, and measure each group. Noise and echo mask and smear the speech. Train on noisy recordings, suppress noise, and use several microphones. And speakers differ in pitch, vocal tract, speed, age and health. Train on many speakers, and adapt to each one.

Experiment Results

Diverse Training Recovers Accuracy

A deliberately simple test shows the pattern. A toy recognizer learned to tell five artificial vowels apart, so chance is twenty percent. Trained only on clean, adult-like voices, it was perfect on new clean voices. Noise at ten decibels cut it to seventy-two percent, and at zero decibels to thirty-four. On child-like voices, it fell to twenty percent: chance. A recognizer trained on a diverse mix reached ninety-two, seventy-seven and fifty-two percent in those cases. Variety recovers a lot, though not everything. Real systems show it too: audits found higher error rates for some groups of speakers, which an average can hide. So report the rate group by group.

Key Takeaways

Speech Recognition in Summary

To sum up. Recognition is a pipeline: the signal, features, an acoustic model, a language model, and a decoder that searches with a beam. MFCCs summarize which frequencies are present, and keep what's being said while mostly ignoring pitch. The acoustic model recognizes sounds, but the language model breaks ties between phrases that sound alike. And word error rate counts substitutions, deletions and insertions, over the words actually said.

Think About It

From the Lab to a Moving Car

Here's something to think about. A recognizer works well in the lab, but poorly in a moving car. Name three differences between the two settings, and say how you'd address each one.

About Introduction to Artificial Intelligence at UPI Study

Introduction to Artificial Intelligence (Computer Science 373) is a self-paced online course of 69 video lessons in 9 chapters, from intelligent agents and search to game strategy, neural networks, logic and reasoning, machine learning, language, robotics and AI ethics. It carries a NCCRS credit recommendation for 3 semester credits.

Every lesson transcript is free on pages like this one, and the videos are free with a free UPI Study account. Enrolling adds a practice quiz after each lesson, the graded work and final exam that earn the credit, lifetime access, and an official transcript on completion. UPI Study has helped over 130,000 students with college credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Enroll now → Course details Download the syllabus All Computer Science courses Every course with PRO →

Compare: AI vs Current Trends in CS and IT

Why take it here

Why students take Introduction to Artificial Intelligence with UPI Study

How machines search, reason and learnIntroduction to Artificial Intelligence covers how AI systems are built: intelligent agents, search and heuristics, game strategy, logic, neural networks, machine learning, language and perception, and the ethics of AI. It is a common upper-level course in computer science degrees.
Credit at a fraction of the priceThe same 3 credits can cost $750 or more at a US college. Here it is one payment, self-paced, with lifetime access.
Finish your degree soonerTransfer credits reduce the courses you still owe, so you can graduate earlier or lighten a heavy semester.
Strengthen an applicationA completed college-level course on an official transcript shows admissions and scholarship committees you can do the work.
Study on your own scheduleNo start dates, no deadlines. Watch the lessons, practice, and take the final when you are ready.
130,000+students helped with college creditsACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Where the credit goes

Universities that work with NCCRS

Introduction to Artificial Intelligence carries a NCCRS credit recommendation. These universities work with NCCRS; some are UPI Study's signed partners. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Thomas Edison State UniversityThomas Edison State UniversitySigned partnerUniversity of the PeopleUniversity of the PeopleSigned partnerCharter Oak State CollegeCharter Oak State CollegeSigned partnerStrayer UniversityStrayer UniversitySigned partnerColumbia Southern UniversityColumbia Southern UniversitySigned partnerIndiana TechIndiana TechSigned partnerPenn StatePenn StateWorks with NCCRSSouthern New Hampshire UniversitySouthern New Hampshire UniversityWorks with NCCRSSUNYSUNYWorks with NCCRS +many more cooperating universitiesFind my college →

Find my college → Not listed? Ask Athena in the corner of this page, or book a 15-minute call.

FAQ

Questions students ask us

About How Machines Understand Spoken Language

What are MFCCs in speech recognition?
MFCCs, or mel-frequency cepstral coefficients, are compact features that describe the smooth outline of a sound's spectrum. The signal is cut into 25-millisecond frames every 10 milliseconds, the energy is pooled into bands on the mel scale, the log of each band is taken, and about 13 numbers per frame are kept. They capture what is being said while largely ignoring the pitch of the voice.
How is word error rate calculated?
Word error rate is the number of substitutions, deletions and insertions in the recognizer's transcript, divided by the number of words in the correct text. If turn on the kitchen lights is heard as turn on a kitchen light, there are two substitutions in five words, so the rate is 40 percent. Lower is better, and many insertions can push the rate above 100 percent.
Why does a speech recognizer need a language model?
A language model judges how likely a word sequence is as language, which settles cases where the audio alone nearly ties. Phrases like recognize speech and wreck a nice beach sound almost the same, so acoustic scores barely separate them. The decoder adds the log acoustic score to a weight times the log language score, and the more natural phrase wins. A simple example is a bigram model, which predicts each word from the one before it.
How does CTC work in end-to-end speech recognition?
Connectionist temporal classification, or CTC, lets a neural network learn from audio and transcripts without marking which frame belongs to which letter. For every frame, the network outputs a letter or a special blank symbol. The output is then collapsed by merging repeated letters and deleting the blanks. A blank between the two l's in hello keeps them apart; without one, they would merge and the result would read helo.

About the Introduction to Artificial Intelligence course

How many college credits is Introduction to Artificial Intelligence?
Introduction to Artificial Intelligence (Computer Science 373) carries a NCCRS credit recommendation for 3 semester credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.
How much does Introduction to Artificial Intelligence cost and what is included?
$250, paid once, with lifetime access: all 69 video lessons, a practice quiz after each, the graded work and final exam, and an official transcript when you finish. Students who want several courses choose UPI Study PRO instead.
Which universities accept UPI Study credits?
UPI Study has signed partner agreements with universities such as Thomas Edison State University, University of the People and Charter Oak State College, and works with universities such as Penn State, SNHU and SUNY. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities. Use "Find my college" on this page to check yours.
How long does Introduction to Artificial Intelligence take?
It is self-paced with no deadlines. The 69 lessons total about 16 hours of video; most students add reading and practice time and finish in a few weeks to a few months, on their own schedule.
How are the exams done?
Quizzes and assignments are online in the course. The final exam is taken online with proctoring; a person on the UPI Study team checks assignments, usually within two days.
How do my credits reach my college?
When the course shows 100% complete, request the official transcript from your dashboard. It is sent to the college you name, and you can send it to more than one. Check your college first with "Find my college" on this page, or ask Athena.

Ready for the full course? Earn 3 credits in Introduction to Artificial Intelligence.

69 video lessons, a quiz after each one, and the graded work that earns the credit. Recommended for credit by NCCRS.

Enroll in Introduction to Artificial Intelligence →Download the syllabusEvery course with PRO →
Introduction to Artificial Intelligence
3 credits · NCCRSEnroll →