Agents That Improve Through Experience, Explained

UPI Study › Courses › Introduction to Artificial Intelligence › Free lessons › Lesson 2.9 Free with a free account

Agents That Improve Through Experience

Introduction to Artificial Intelligence · Chapter 2: Intelligence, Cognition, and Agent DesignSection 2.9 · lesson 17 of 6912 min · free with a free account

← Maximizing Outcomes: Utility-Driven Agent BehaviorNext lesson: Writing Pseudocode and Designing Step-by-Step Solutions →

Earn 3 college credits for Introduction to Artificial IntelligenceRecommended for credit by NCCRS · Computer Science 373 · self-paced · one payment, lifetime access
Enroll in Introduction to Artificial Intelligence · $250 →Download the syllabus (PDF)

In this lesson

How learning agents use feedback and exploration to improve themselves.

  • Explain why many tasks need agents that learn instead of fixed rules
  • Identify the performance element, critic, learning element, and problem generator in a learning agent
  • Classify feedback as supervised, unsupervised, or reward feedback
  • Describe the learning curve and the trade-off between exploration and exploitation

Free study notes

Get this lesson's notes by email

The key points and 4 practice questions with answers, in your inbox. One email, free.

One email. Unsubscribe in one click.

Transcript

Everything said in the video, section by section (1,745 words). Read along, or search it.

Opening Question

A Music App That Learns Your Taste

On Monday, a music app knows nothing about your taste. It plays the same popular songs it plays for everyone. A month later, it queues tracks you love, including a few you'd never have searched for. Nobody rewrote its code in between. It learned. So what does an agent need, to get better at its job on its own?

Core Idea

Improving the Program From Experience

Earlier lessons described agents that act according to a program, written by their designer. A learning agent adds machinery that improves that program from experience. So it can become better than the version its designer first built.

Why Learn

Three Reasons Fixed Rules Fall Short

Why not just write down the best action for every situation in advance? Three problems make that impossible, for many real tasks. First, the designer can't foresee every situation. A delivery robot meets blocked doors, new furniture, and unexpected visitors. Learning lets it adapt to cases nobody planned for. Second, the world changes. Shoppers' tastes shift with seasons and trends, and a learning agent keeps updating, instead of going stale. Third, the designer may not know how to write the solution at all. Nobody can list exact rules for recognizing a spoken word or a face. The agent works out its own rules, from examples.

Architecture

Four Parts of a Learning Agent

A learning agent has four parts. One part acts, and the other three work together to make the acting part better. The performance element chooses the actions. It's the whole agent from earlier lessons, built with whatever design suits the task. The critic watches what happens, compares it with a fixed performance standard, and reports how well the agent is doing. That report is feedback. The learning element uses that feedback to change the performance element, so it does better next time. And the problem generator suggests new actions to try, so the agent gains informative experiences it wouldn't otherwise have.

Example

All Four Parts in a Self-Driving Taxi

Picture all four in a self-driving taxi. The performance element decides to brake, steer, or change lanes. The critic notes that a turn was abrupt, and uncomfortable for the passenger. The learning element adjusts how early, and how gently, the taxi starts to brake. And the problem generator, in a safe test area, tries a different braking distance, to see what works best.

The Critic

A Performance Standard Fixed From Outside

Why does an agent need a critic at all? Because percepts alone don't say whether it's doing well. A chess program sees a board. But nothing in the board tells it whether the position is good. The critic holds the performance standard, and supplies that judgment. And the standard is fixed from outside the agent. If the learning element could change it, the agent could improve, simply by lowering its standards.

Problem Generator

Trying Actions That Look Second Best

The problem generator can look odd at first. It sometimes suggests an action that probably isn't the agent's best guess. It earns its place, because an agent that only repeats what already works never discovers anything better.

Quick Check

Which Part Shifts the Thermostat Schedule

Quick check, with a new agent. A smart thermostat learns a family's habits. Its goal: nobody should have to touch the dial. Each turn of the dial by hand is logged as a miss. Each night, one part uses the day's misses to shift tomorrow's heating schedule. Which part is that? Pause, and decide. The learning element. It takes the feedback, the misses, and changes how the performance element acts: tomorrow's schedule. The critic is tempting, because the misses are its work. But the critic only scores what happened. It doesn't change the schedule. And nothing new is being tried, so it isn't the problem generator.

Feedback Types

Labels, Structure, and Rewards

The critic's report is called feedback. It comes in three main forms, and the form decides what the agent can learn. Supervised feedback: examples that each come with the correct answer. Emails marked spam or not spam teach a filter to sort new mail. Unsupervised feedback: data with no answers and no scores. A store finds three kinds of shopper, without anyone naming them. Reward feedback: a score, a reward or a penalty, after the agent acts, often delayed. A robot learning to walk is rewarded for staying upright and moving forward. With supervised feedback, a teacher supplies the right answer, usually as a label. It's clear and direct, but someone must produce all those labels, which is slow and expensive. With unsupervised feedback, nothing says whether the agent's groups are right. People must decide whether the structure it found is useful. With reward feedback, nobody says which action was correct. The agent works that out from its scores, and two difficulties come up.

Reward Pitfalls

Delayed Rewards and Careless Reward Design

Delayed reward. In a long game, the win or loss arrives only at the end. The agent must work out which of its many earlier moves deserve the credit, or the blame. Reward design. The agent maximizes the reward it's given, not the result its designer imagined. Pay a vacuum per unit of dirt collected, and it can earn points without doing the real job. The methods, and the math for learning from rewards, come in later lessons.

Learning Curve

Fast Gains, Slower Gains, Then a Plateau

How does performance improve over time? Progress is gradual, and usually follows a recognizable shape: a learning curve, which plots performance against experience. First come fast early gains. The first experiences teach the most obvious lessons. Then gains slow. The easy improvements are used up, so only subtle ones remain. Then a plateau: the agent has learned what its data and its design allow. And along the way there are small ups and downs, because feedback is noisy, and some experiences are unrepresentative.

Trade-Off

Exploit What Works or Explore Something New

A learning agent also faces a constant choice. It can exploit what it knows, repeating the best-known action. Or it can explore, trying something new that might turn out better. Exploiting gives reliable results right now, but it may get stuck with a good-enough answer. Like a music app that always plays more of the same genre. Exploring may discover something much better, but some attempts fail, and short-term results drop. Like occasionally playing an unfamiliar genre. The problem generator is the part that drives exploration.

Comparison

The Experimenting Agent Ends Higher

Compare two music agents, with made-up but typical numbers. The agent that never experiments improves faster at first, then stalls. The agent that occasionally experiments starts more slowly, since some trial songs miss. But it keeps discovering new tastes, and ends far higher. Exploration costs a little early, and pays off later.

Limits

Four Things That Cap Improvement

What limits how far an agent can improve? First, the quality of feedback. Noisy or misleading feedback teaches the wrong lessons, like accidental skips recorded as dislikes. Second, the amount and variety of experience. A listener only ever shown one genre leaves gaps. Third, the flexibility of the performance element. A rule that only looks at the time of day can't express what was learned. And fourth, the cost and safety of experiments. A taxi can't test dangerous maneuvers on public roads.

Good Practice

Measuring on Unseen Situations Over Time

So keep checking. Measure performance against the standard, over a long period, and on situations the agent hasn't already seen. When the world changes, knowledge that was once correct can become outdated. A good learning agent keeps learning, rather than stopping after one training period.

Worked Example

PEAS for a Listening Queue

Let's put it all together, in one agent: a music app that builds a listening queue for one user. First, its task, with the PEAS checklist. The performance measure is a high share of songs the listener enjoys. Finished or saved songs count for it, and early skips count against. The environment is the music catalog, the listener, and the time of day. The actuators choose the next song, and add it to the queue. And the sensors pick up plays, skips, saves, thumbs-up and thumbs-down taps, and the time of day.

Worked Example

The Playlist Agent's Four Parts

Now its four parts. The performance element picks the next song from the current taste profile. At eight in the evening it queues a mellow track, because past evenings favored them. The critic scores each outcome. A skip after six seconds is logged as a miss, and a saved song as a hit. The learning element updates the taste profile. After three skipped rock songs and two saved jazz tracks, it raises jazz, and lowers rock. And the problem generator slips in an unfamiliar song, about one time in ten.

Worked Example

Three Kinds of Feedback in One App

All three kinds of feedback can feed this one agent. Thumbs-up and thumbs-down taps are labeled examples: supervised feedback, to predict which songs a listener will like. Finishing or saving a song earns a reward, and skipping earns a penalty. From that, it learns which choices keep the listener happy over a whole session. And songs that listeners play together, with no labels, let it group similar songs, so one liked song can lead to related ones. That's unsupervised.

Results

From 28 to 79 Percent Enjoyed

Here's how its score grows, with illustrative numbers. In week zero, it plays generic popular songs, and the listener enjoys 28 percent. After week one, the first skips and saves teach the basics: 36 percent. By week two, a taste profile forms: 43. By week four, disliked genres are mostly avoided: 54. By week eight, a genre found by experimenting has become a favorite: 68. And by week sixteen, 79 percent, with small gains. It has learned most of what it can. No programmer ever wrote a rule about this listener.

Quick Check

An Elevator Learning From Wait Times

Quick check. An elevator controller in an office building is never told which floor it should wait at. After each hour, it gets a score: the average wait, and the lower the better. What kind of feedback is it learning from? Pause, and decide. Reward feedback. Nobody gives it a correct floor for each moment, only a score after it acts, and it must work out which choices earned it. Supervised is tempting, because it gets a number. But a label says what the right answer was. A score only says how well it did. And it isn't unsupervised: there is a score.

Key Takeaways

Critic, Learner, Generator, and Feedback

To sum up. A learning agent adds three parts to the performance element: a critic, a learning element, and a problem generator. The critic judges results against a standard that's fixed outside the agent. Feedback comes as labeled answers, structure with no answers, or rewards after acting. Performance rises fast, then levels off. And exploring costs a little now, but can raise the final level.

Reflection

Which Experiments Belong on Public Roads

Something to think about. Choose an app that seems to adapt to you. What are its performance element, critic, learning element, and problem generator? And which kinds of feedback does it use? And a self-driving taxi needs a problem generator too. Which experiments should be allowed on public roads, and which should happen only in simulation?

About Introduction to Artificial Intelligence at UPI Study

Introduction to Artificial Intelligence (Computer Science 373) is a self-paced online course of 69 video lessons in 9 chapters, from intelligent agents and search to game strategy, neural networks, logic and reasoning, machine learning, language, robotics and AI ethics. It carries a NCCRS credit recommendation for 3 semester credits.

Every lesson transcript is free on pages like this one, and the videos are free with a free UPI Study account. Enrolling adds a practice quiz after each lesson, the graded work and final exam that earn the credit, lifetime access, and an official transcript on completion. UPI Study has helped over 130,000 students with college credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Enroll now → Course details Download the syllabus All Computer Science courses Every course with PRO →

Compare: AI vs Current Trends in CS and IT

Why take it here

Why students take Introduction to Artificial Intelligence with UPI Study

How machines search, reason and learnIntroduction to Artificial Intelligence covers how AI systems are built: intelligent agents, search and heuristics, game strategy, logic, neural networks, machine learning, language and perception, and the ethics of AI. It is a common upper-level course in computer science degrees.
Credit at a fraction of the priceThe same 3 credits can cost $750 or more at a US college. Here it is one payment, self-paced, with lifetime access.
Finish your degree soonerTransfer credits reduce the courses you still owe, so you can graduate earlier or lighten a heavy semester.
Strengthen an applicationA completed college-level course on an official transcript shows admissions and scholarship committees you can do the work.
Study on your own scheduleNo start dates, no deadlines. Watch the lessons, practice, and take the final when you are ready.
130,000+students helped with college creditsACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Where the credit goes

Universities that work with NCCRS

Introduction to Artificial Intelligence carries a NCCRS credit recommendation. These universities work with NCCRS; some are UPI Study's signed partners. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Thomas Edison State UniversityThomas Edison State UniversitySigned partnerUniversity of the PeopleUniversity of the PeopleSigned partnerCharter Oak State CollegeCharter Oak State CollegeSigned partnerStrayer UniversityStrayer UniversitySigned partnerColumbia Southern UniversityColumbia Southern UniversitySigned partnerIndiana TechIndiana TechSigned partnerPenn StatePenn StateWorks with NCCRSSouthern New Hampshire UniversitySouthern New Hampshire UniversityWorks with NCCRSSUNYSUNYWorks with NCCRS +many more cooperating universitiesFind my college →

Find my college → Not listed? Ask Athena in the corner of this page, or book a 15-minute call.

FAQ

Questions students ask us

About Agents That Improve Through Experience

What is a learning agent in artificial intelligence?
A learning agent is an agent that improves its own way of acting from experience, so it can end up better than the version its designer first built. It is useful when a designer cannot foresee every situation, when the world keeps changing, or when nobody knows how to write the rules directly, as with recognizing faces or spoken words. A music app that learns a listener's taste is a familiar example.
What are the four components of a learning agent?
The four components are the performance element, the critic, the learning element, and the problem generator. The performance element chooses actions. The critic compares results with a fixed performance standard and reports feedback. The learning element uses that feedback to change the performance element. The problem generator suggests new, informative actions to try. In a self-driving taxi, the critic might flag an abrupt turn, and the learning element would adjust braking.
Why is the critic's performance standard fixed outside the agent?
The standard is fixed outside the agent so the agent cannot appear to improve simply by lowering its own standards. Percepts alone do not say whether the agent is doing well; a chess program sees the board but not whether its position is good. The critic supplies that judgment, and if the learning element could rewrite the standard, it could make poor performance look like success.
What is the exploration versus exploitation trade-off?
Exploitation means repeating the action that currently works best, while exploration means trying something new that might turn out better. Exploiting gives reliable results now but can leave an agent stuck with a good-enough answer. Exploring costs some failed attempts and lower short-term results, yet it can raise the final level of performance. In a learning agent, the problem generator is the part that drives exploration.

About the Introduction to Artificial Intelligence course

How many college credits is Introduction to Artificial Intelligence?
Introduction to Artificial Intelligence (Computer Science 373) carries a NCCRS credit recommendation for 3 semester credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.
How much does Introduction to Artificial Intelligence cost and what is included?
$250, paid once, with lifetime access: all 69 video lessons, a practice quiz after each, the graded work and final exam, and an official transcript when you finish. Students who want several courses choose UPI Study PRO instead.
Which universities accept UPI Study credits?
UPI Study has signed partner agreements with universities such as Thomas Edison State University, University of the People and Charter Oak State College, and works with universities such as Penn State, SNHU and SUNY. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities. Use "Find my college" on this page to check yours.
How long does Introduction to Artificial Intelligence take?
It is self-paced with no deadlines. The 69 lessons total about 16 hours of video; most students add reading and practice time and finish in a few weeks to a few months, on their own schedule.
How are the exams done?
Quizzes and assignments are online in the course. The final exam is taken online with proctoring; a person on the UPI Study team checks assignments, usually within two days.
How do my credits reach my college?
When the course shows 100% complete, request the official transcript from your dashboard. It is sent to the college you name, and you can send it to more than one. Check your college first with "Find my college" on this page, or ask Athena.

Ready for the full course? Earn 3 credits in Introduction to Artificial Intelligence.

69 video lessons, a quiz after each one, and the graded work that earns the credit. Recommended for credit by NCCRS.

Enroll in Introduction to Artificial Intelligence →Download the syllabusEvery course with PRO →
Introduction to Artificial Intelligence
3 credits · NCCRSEnroll →