UPI Study › Courses › Introduction to Artificial Intelligence › Free lessons › Lesson 7.3 Free with a free account
Combining Natural Language Processing with Deep Learning
Introduction to Artificial Intelligence · Chapter 7: Machine Learning, NLP, and PerceptionSection 7.3 · lesson 51 of 6916 min · free with a free account
Free preview
Create a free account to keep watching
It takes a minute and costs nothing. Your free account unlocks all of Chapter 1 and 3 more lessons of Introduction to Artificial Intelligence.
Lesson complete
Earn the 3 college credits for Introduction to Artificial Intelligence
Enrolling adds a quiz after each lesson, the graded work and the official transcript. Recommended for credit by NCCRS.
Next lesson: Information Gain and Entropy in Learning Models →← Discovering Patterns Without Labels: Unsupervised TechniquesNext lesson: Information Gain and Entropy in Learning Models →
In this lesson
How text becomes vectors and how attention and transformers model meaning.
- Describe the traditional NLP pipeline of tokenization, tagging and parsing
- Compute TF-IDF weights and compare documents with cosine similarity
- Explain how attention uses queries, keys and values to weight tokens
- Compare encoder-only, decoder-only and encoder-decoder transformers and how they are pretrained
Free study notes
Get this lesson's notes by email
The key points and 4 practice questions with answers, in your inbox. One email, free.
One email. Unsubscribe in one click.
Transcript
Everything said in the video, section by section (2,156 words). Read along, or search it.
Opening Question
I Saw Her Duck
"I saw her duck." Did you see a bird that belongs to her, or did you watch her lower her head? People settle that from context, without noticing. A computer sees only a string of characters, and every step toward understanding must be built. Natural language processing, or NLP, is the field that builds those steps. How does a machine get from characters to meaning? NLP has passed through three eras: hand-written rules, statistics that count patterns, and deep learning that learns from huge amounts of text.
Traditional Pipeline
Tokenize, Tag and Parse
Before deep learning, NLP systems were a chain of stages, each adding structure to the one before. Tokenization splits the text into tokens, the units a program works with. Tagging labels each token's grammatical role. And parsing finds the sentence's structure.
Tokenization
Spaces, Contractions and Subword Tokens
Splitting on spaces is a start, but it isn't enough. In mat, with a period after it, the period should be its own token. Don't is often split into do, and n't. A name like New York may need to stay together. And Chinese and Japanese have no spaces between words, so the tokenizer must decide where words begin and end. Modern models split text into subword tokens, so unhappiness might become un, happi, and ness. That keeps the vocabulary to tens of thousands of tokens, yet handles words never seen before.
Tagging and Parsing
Parts of Speech and Dependency Parses
A part-of-speech tagger labels each token: noun, verb, determiner like the, or a preposition like on. The tag depends on context. In please book a flight, book is a verb. In she read a book, it's a noun. Parsing finds the grammar. In a dependency parse, every word links to the word it depends on, and one word, usually the main verb, is the root. In the cat chased the mouse, chased is the root. The cat is its subject, and the mouse its object. Swap the nouns, and the meaning reverses. Other stages find names of people and places, and reduce sitting and sat to sit.
Limitations
Errors Cascade Through the Pipeline
The pipeline is easy to inspect: a linguist can see exactly which stage made a mistake. But errors cascade. A wrong tag gives a wrong parse, which gives a wrong answer. And experts had to design each stage's features by hand. Deep learning removed much of that effort, by learning useful representations straight from data. For that, text has to become numbers.
Bag of Words
Counting Words, Ignoring Order
The simplest way to turn text into numbers is the bag of words: count each vocabulary word, and ignore the order, as if the words were tipped out of a bag. Three tiny documents: the cat sat on the mat. The dog sat on the log. And the cat chased the dog. The vocabulary has eight words, so each document becomes a vector of eight counts. It's quick, and works surprisingly well for spam detection and topic labels. But it has three weaknesses. Order is lost: the dog bit the man, and the man bit the dog, give identical counts. Real vectors are huge, and mostly zeros. And words have no relationships: cat and kitten are as unrelated as cat and carburetor.
TF-IDF
Weighting Rare Words Above Common Ones
Raw counts reward common words like the, which tell us nothing. TF-IDF weights each word by two factors. Term frequency is how often the word appears in this document. Inverse document frequency is how rare it is across all the documents. A common formula is the natural log of N over d f: the number of documents, over the number containing the word. A word in every document scores zero. In D1, the appears in all three documents, so its weight is zero. Cat, sat and on appear in two, so 0.405 each. Mat appears only in D1, so it weighs most: 1.099.
Quick Check
Log Gets the Biggest Weight
Quick check. Same three documents. In D2, the dog sat on the log, which word gets the biggest TF-IDF weight? Pause, and decide. Log. It's in only one document, so its IDF is the natural log of three: 1.099, the highest. The appears twice, but it's in every document, so its IDF is zero, and twice zero is zero. Dog is shared with D3, so it gets only 0.405.
Cosine Similarity
Comparing Documents by Angle
To compare documents, measure the angle between their vectors with cosine similarity: the dot product, divided by the product of the lengths. It's one for vectors pointing the same way, and zero for unrelated ones. With raw counts, D1 and D2 score 0.750, and every other pair 0.668. The shared word the makes everything look similar. With TF-IDF, D1 and D2, both about sitting on something, score 0.193, and the other pairs only 0.102. The false closeness is gone.
Word Embeddings
Dense Vectors from Shared Contexts
Counting treats words as unrelated labels. Word embeddings give each word a short, dense vector of real numbers, typically a few hundred long. They're learned so that words used in similar contexts get similar vectors: a word is known by the company it keeps. With made-up three-number vectors, cat and dog have a cosine of 0.996, while cat and car score only 0.293. In a toy space, king, minus man, plus woman, lands exactly on queen. Real embeddings like word2vec, from 2013, and GloVe, from 2014, show this only approximately. An embedding layer is just a lookup table, one row per word. For fifty thousand words and three hundred dimensions, that's fifteen million learned numbers.
Comparison
From Sparse Counts to Contextual Vectors
Lined up, the four representations improve on each other. Bag of words and TF-IDF are long and sparse, and ignore both order and meaning. A static embedding captures similar meaning, but gives bank one vector, whether the sentence is about money or a river. A contextual embedding gives a word a different vector in each sentence. Producing those is exactly what transformers do.
Sequence to Sequence
Encoders, Decoders and the Bottleneck
Many tasks turn one sequence into another: a French sentence into English, an article into a summary, a question into an answer. A sequence-to-sequence model has two parts. An encoder reads the input, and builds a summary of it. A decoder then writes the output one token at a time, each chosen with a softmax over the vocabulary. The first successful ones, around 2014, used recurrent networks. And they had a bottleneck: the whole input was squeezed into one fixed-size summary, so long sentences lost detail.
Attention
Queries, Keys and Values
Attention removes the bottleneck. Instead of one summary, the decoder looks back at every input position, and decides how much each one matters right now. Each token gets three vectors, made from its embedding by learned matrices. A query: what it's looking for. A key: what it offers for matching. And a value: what it passes on. Step one: score every key against the query with a dot product, divided by the square root of the vector length, d. Step two: turn the scores into weights with softmax, so they're positive and add up to one. Step three: output the weighted sum of the values.
Worked Example
What Does It Refer To
A worked example: the animal was tired because it... What does it refer to? The query for it is one, one, zero, zero, and d is four, so scores are divided by two. The keys: The is zero, zero, two, two. Animal is two, two, zero, zero. And tired is one, zero, one, zero. The dot products are 0, 4, and 1. Divided by two, the scores are 0, 2, and 0.5. Softmax turns them into weights of 0.100, 0.736, and 0.164. So the output is mostly animal's value: it means the animal.
Self-Attention
Multiple Heads and Position Information
In self-attention, the queries, keys and values all come from the same sequence, so every token can gather information from every other token. Written for a whole sequence at once, it's softmax of Q times K transposed, over the square root of d, times V: the same three steps, as matrices. A transformer runs several attention heads side by side. One head may track which noun a pronoun means, another the nearby words, and their results are combined. Attention itself has no sense of order, so each token's embedding also gets position information. That's how the model tells dog bites man from man bites dog.
Transformer
Encoder-Only, Decoder-Only and Encoder-Decoder
The transformer was introduced in 2017, in a paper titled Attention Is All You Need. It stacks identical blocks. Each block has multi-head self-attention, then a feed-forward network, with residual connections and layer normalization that keep deep stacks trainable. Blocks come in three arrangements. Encoder-only models, like BERT, see the whole input in both directions, and are trained to fill in hidden words. Decoder-only models, like GPT, see only earlier tokens, and are trained to predict the next one. They write text one token at a time. Encoder-decoder models, like T5, turn input text into output text, for translation and summaries.
Why Transformers Won
Parallelism, Short Paths and Scale
Why did transformers take over? Parallelism: a recurrent network processes tokens in order, but a transformer processes every position of a layer at once, which suits graphics processors. Short paths: any two tokens connect in one attention step, so long-distance links are easy to learn. And scale: performance kept improving as models and data grew. The cost: attention compares every token with every other, so the work grows with the square of the length. A hundred tokens means ten thousand pairs. A thousand tokens, a million pairs. Ten thousand tokens, a hundred million.
Quick Check
Triple the Length, Nine Times the Work
Quick check. A model handles a 2,000-token document. A user sends one three times as long: 6,000 tokens. How much more attention work does one layer do? Pause, and predict. Nine times. The pairs go from 2,000 squared, four million, to 6,000 squared, thirty-six million. Three times the length means three squared, nine times the work. Three times would be true of a recurrent network, whose work grows in step with the length.
Pretraining
Self-Supervised Learning Then Adaptation
Training a large transformer from scratch for every task would need huge amounts of labeled data. Pretraining avoids that. The model first learns general language from vast ordinary text, with no labels, by solving a task the text provides itself. Masked language modeling hides some words and predicts them, as in BERT. Next-token prediction predicts each token from the ones before, as in GPT. That self-supervised learning forces the model to pick up grammar, facts and patterns of reasoning. It's costly, but done once. Then it's adapted cheaply. Fine-tuning continues training on a few hundred or more labeled examples. Prompting gives instructions and a few examples, and changes no weights. Or the model's vectors can feed a simple model, or a search, which is cheap and fast.
Model Sizes
From BERT to GPT-3
Size is one reason pretraining is so costly. BERT-base, from 2018, has about 110 million parameters, and BERT-large about 340 million. GPT-2, in 2019, reached about 1.5 billion. GPT-3, in 2020, about 175 billion. Models released since are larger still.
Applications
Common Tasks and How They Are Scored
Pretrained models handle the common tasks: sorting text into categories, like spam or not; finding named entities, like Marie Curie as a person, and Paris as a place. Answering questions from a passage, translating, summarizing, generating text from a prompt, and semantic search, which finds matching documents even with different words. Categories are scored by accuracy, or the F1 score, which balances catching true cases against false alarms. Translations and summaries are scored by word overlap with a human reference, using measures like BLEU and ROUGE.
Cautions
Hallucinations, Bias, Context and Cost
Four cautions. Fluent but wrong: a generative model picks likely words, not verified facts, so it can state falsehoods with confidence. These are called hallucinations. Inherited bias: it learns from human writing, and can reproduce its stereotypes and gaps. Limited context: it attends to only so many tokens at once. And cost: training and running big models takes real computing power and energy. So match the tool to the job. A small labeled set and a clear task: fine-tune a pretrained model. No labels at all: prompt a large model, and check its answers. A quick keyword labeling job with plenty of data: TF-IDF with a simple model may be enough, cheaper, and easier to explain.
Application
Context Settles the Duck
Back to I saw her duck. A tagger has to pick NOUN or VERB for duck, and the sentence alone can't settle it. Add context: as the ball flew at her. Attention lets duck draw on flew and ball, and its contextual vector shifts toward the action. That's the path from characters to meaning: tokens, then numbers, then vectors shaped by every other word around them.
Key Takeaways
From Characters to Contextual Meaning
To sum up. The traditional pipeline, tokenize, tag and parse, adds structure step by step, but its errors cascade. Bag of words and TF-IDF are simple, but ignore order and meaning. Embeddings give similar words similar vectors. Attention weighs every token against every other, and blends their values. Transformers stack it, train in parallel, and cost grows with length squared. Pretraining on unlabeled text, then fine-tuning or prompting, is the standard recipe, with care about errors and bias.
Think About It
Telling Not Good from Good
Here's something to think about. A bag-of-words model scores the movie was not good, and the movie was good, not bad, almost the same. What in this lesson would help a model tell them apart? Would you reach for a pretrained transformer first, or try something simpler?
About Introduction to Artificial Intelligence at UPI Study
Introduction to Artificial Intelligence (Computer Science 373) is a self-paced online course of 69 video lessons in 9 chapters, from intelligent agents and search to game strategy, neural networks, logic and reasoning, machine learning, language, robotics and AI ethics. It carries a NCCRS credit recommendation for 3 semester credits.
Every lesson transcript is free on pages like this one, and the videos are free with a free UPI Study account. Enrolling adds a practice quiz after each lesson, the graded work and final exam that earn the credit, lifetime access, and an official transcript on completion. UPI Study has helped over 130,000 students with college credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.
Enroll now → Course details Download the syllabus All Computer Science courses Every course with PRO →
Compare: AI vs Current Trends in CS and IT
Why take it here
Why students take Introduction to Artificial Intelligence with UPI Study
Where the credit goes
Universities that work with NCCRS
Introduction to Artificial Intelligence carries a NCCRS credit recommendation. These universities work with NCCRS; some are UPI Study's signed partners. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.
Thomas Edison State UniversitySigned partner
University of the PeopleSigned partner
Charter Oak State CollegeSigned partner
Strayer UniversitySigned partner
Columbia Southern UniversitySigned partner
Indiana TechSigned partner
Penn StateWorks with NCCRS
Southern New Hampshire UniversityWorks with NCCRS
SUNYWorks with NCCRS
+many more cooperating universitiesFind my college →Find my college → Not listed? Ask Athena in the corner of this page, or book a 15-minute call.
FAQ
Questions students ask us
About Combining Natural Language Processing with Deep Learning
What is TF-IDF?
How does attention work in a transformer?
What is the difference between BERT and GPT?
Why does transformer attention cost grow with the square of the input length?
About the Introduction to Artificial Intelligence course
How many college credits is Introduction to Artificial Intelligence?
How much does Introduction to Artificial Intelligence cost and what is included?
Which universities accept UPI Study credits?
How long does Introduction to Artificial Intelligence take?
How are the exams done?
How do my credits reach my college?
Ready for the full course? Earn 3 credits in Introduction to Artificial Intelligence.
69 video lessons, a quiz after each one, and the graded work that earns the credit. Recommended for credit by NCCRS.
3 credits · NCCRSEnroll →