UPI Study › Courses › Introduction to Artificial Intelligence › Free lessons › Lesson 5.3 Free with a free account
Splitting Data with Decision Tree Models
Introduction to Artificial Intelligence · Chapter 5: Neural Networks and Learning ArchitecturesSection 5.3 · lesson 34 of 6911 min · free with a free account
Free preview
Create a free account to keep watching
It takes a minute and costs nothing. Your free account unlocks all of Chapter 1 and 3 more lessons of Introduction to Artificial Intelligence.
Lesson complete
Earn the 3 college credits for Introduction to Artificial Intelligence
Enrolling adds a quiz after each lesson, the graded work and the official transcript. Recommended for credit by NCCRS.
Next lesson: Forward-Flowing Neural Network Structures →← How Neural Networks Drive Machine LearningNext lesson: Forward-Flowing Neural Network Structures →
In this lesson
Build, read, and prune decision trees for classification and regression.
- Identify the root, decision nodes, branches, and leaves of a decision tree and read its rules
- Choose splits by comparing weighted impurity scores
- Explain overfitting and prevent it with stopping rules, pruning, and validation data
- Distinguish classification trees, regression trees, and random forests
Free study notes
Get this lesson's notes by email
The key points and 4 practice questions with answers, in your inbox. One email, free.
One email. Unsubscribe in one click.
Transcript
Everything said in the video, section by section (1,660 words). Read along, or search it.
Opening Analogy
Twenty Questions as a Diagram
In twenty questions, one player thinks of something, and the other narrows it down with yes-or-no questions. Each answer rules out a big share of the possibilities, so a few well-chosen questions are enough. A decision tree is that same game, drawn as a diagram and learned from data. It asks questions about an item, follows the answers, and arrives at a prediction. Like the rules of an expert system, a tree can be read aloud as IF-THEN statements. The difference is that the rules are found from data, not written by experts.
Running Example
Twelve Subscribers Who Renew or Cancel
One example runs through this whole lesson. A company wants to predict whether each subscriber will renew, or cancel. It knows three facts about each customer: usage, often or rarely; complaints, none or some; and the plan, basic or premium. Here are twelve past customers. Six renewed, and six canceled. So the data starts out as mixed as it can be. The goal is to find questions that separate the two groups.
Tree Anatomy
Root, Decision Nodes, Branches, and Leaves
Here's the tree this data will produce. The root node sits at the top, and holds the first question, applied to all the data: usage. Any node that asks a question about one feature is a decision node. Here, the two plan questions. A branch is one possible answer, leading to the next node: often, or rarely. A leaf asks nothing more. It simply gives the prediction. And the depth of a tree is the number of questions on its longest path. This one has depth two.
Making Predictions
Every Path Is an IF-THEN Rule
To predict, start at the root, answer each question about the new customer, follow the matching branch, and read the leaf where you land. Every path from root to leaf is one rule. The leftmost says: IF usage is often AND the plan is basic, THEN predict renew. That readability is the main reason people like trees. Anyone can follow the reasoning, which is much harder with the neural networks covered later.
Building a Tree
Recursive Splitting From the Top Down
How is a tree like that built? From the top down. Start with all the training data at the root. Find the best question to split the data into groups, where each group is dominated by a single label. Then treat each group as a smaller problem, and repeat. A method that applies itself to the pieces it creates is recursive, so this is called recursive splitting. It continues until a stopping rule says to end.
Impurity
Scoring How Mixed a Group Is
To compare questions, we need a number for how mixed a group is. A group is pure if every item has the same label, and impure if the labels are mixed. An impurity measure scores this, with zero meaning perfectly pure. Here we'll use the simplest one: the share of items that disagree with the group's most common label. Five renew and one cancel scores one sixth, about 0.17. Three and three scores 0.5, the worst possible for two labels. To score a whole split, average its groups' scores, weighted by how many items each group holds. Real tree builders use refined measures called entropy, which gives information gain, and Gini impurity. They play exactly the same role, and we'll work through them in a later topic.
Worked Example
Usage Wins the Root Question
Now score the three possible questions at the root. With no split at all, six and six, the score is 0.5. Ask about usage: often gives five renew and one cancel; rarely gives one and five. Each group scores 0.17, so the split scores 0.17. Ask about complaints: none gives four and two; some gives two and four. That's 0.33. Ask about the plan: basic gives five and five, premium one and one. Still 0.5. Both groups stay exactly as mixed as before. Usage gives the purest groups, so usage becomes the root question.
Worked Example
The Plan Splits Each Branch
Now treat each group as its own problem. Among the six often customers, five renew and one cancels. The best question here is the plan: the five basic customers all renew, and the one premium customer cancels. Both groups are pure, so the score drops to zero. Among the six rarely customers, the plan again gives two pure groups: the basics all cancel, and the one premium renews. Every leaf is now pure, so splitting stops, and each leaf predicts its most common label. Notice: the plan was the worst question at the root, and the best one inside each branch. The best question depends on which items are in the group, so the tree recalculates at every node.
Quick Check
One Pure Group Is Not Enough
Quick check, with new numbers. A node holds ten items: five yes, and five no. Question A splits it into four yes and one no, and one yes and four no. Question B splits it into five yes and three no, and a pure group of two no's. Using the same impurity score, weighted by group size, which split is better? Pause, and work it out. A is better. Each of its groups has one item out of five against the majority: 0.2, so the split scores 0.2. Question B makes a pure group, which is tempting. But its big group, five and three, scores three eighths, and it holds eight of the ten items. Eight tenths of three eighths is 0.3. Worse than A. One pure group doesn't make the best split. The weighted average does.
Overfitting
Perfect on Training, Weak on New Data
Look again at the grown tree. Its two plan splits each rest on a single customer: number six, and number seven. A tree that keeps splitting until every leaf is pure can build rules around oddities. That's overfitting: matching the quirks and noise of the training data so closely that it does worse on new data. To test that, we need data the tree has never seen. Now take six new customers, and two trees. The full tree, and a pruned tree that keeps only the usage question: often predicts renew, rarely predicts cancel. On the twelve training customers, the full tree is perfect, and the pruned tree gets ten. On the six new customers, the full tree gets three right. The pruned tree gets five.
Depth Trade-Off
Overfitting Versus Underfitting
The full tree is perfect on the data it learned from, and only a coin flip on new data. That's the typical pattern. As a tree grows deeper, accuracy on the training data keeps rising. But accuracy on new data rises at first, then falls. And a tree that's too shallow misses real patterns. That's called underfitting. The goal is a depth in between. This sketch is illustrative.
Prevention
Stopping Rules and Pruning
Two ways prevent overfitting. Stopping rules end the growth early. Common rules: a maximum depth; a minimum number of items before a node may split, say five; a minimum improvement in impurity; and a minimum number of items in a leaf. With a minimum leaf size of two, our tree could never have made its single-customer leaves. Pruning works in the opposite order. Grow the tree large first, then cut back branches that don't help on held-out data, replacing each with a leaf. Pruning the two plan splits gives the usage-only tree. Both rely on validation data: examples kept aside to judge how well a tree will do on new cases.
Regression Trees
Leaves That Predict Averages
The renewal tree is a classification tree: it predicts a category. Trees can also predict numbers. A regression tree has the same shape, but each leaf holds a number: the average of the training targets that reached it. Instead of scoring how mixed the labels are, it scores how spread out the numbers are. A good split puts similar numbers together. Take house prices. Under fifteen hundred square feet: an average of a hundred eighty-five thousand dollars. Under twenty-two hundred: three hundred thousand. Larger: four hundred forty-seven thousand. Every house in a leaf gets the same price, so the predictions form steps. Every house between fifteen hundred and twenty-two hundred square feet gets three hundred thousand. Simple and robust, but it can't follow a smooth trend.
Random Forests
Many Trees That Vote
A single tree is unstable: a small change in the training data can produce a very different tree, and a deep tree tends to overfit. A random forest fixes this with a simple idea: build many trees, and let them vote. Each tree trains on a different random sample of the data, and at each split it considers only a random subset of the features. So the trees differ. For classification, the forest takes the majority vote. Here, two of three trees say renew. For regression, it averages the trees' numbers. It works because the trees make different mistakes, and the vote tends to cancel them out, much as a committee often judges better than any one member. Combining models like this is called an ensemble. The price: a forest is much harder to read than one tree, and it takes more computing.
Quick Check
Predicting Delivery Minutes From a Leaf
Quick check. A store predicts how many minutes each delivery will take, from its distance, the hour, and the weather. One leaf of its tree was reached by three training deliveries: twenty, twenty-five, and thirty-six minutes. A new delivery lands in that leaf. What does the tree predict? Pause, and decide. Twenty-seven minutes. This target is a number, so it's a regression tree, and a leaf predicts the average of its training deliveries: eighty-one minutes over three is twenty-seven. Twenty is just the fastest delivery, and thirty-six the slowest. Neither is a fair guess for a typical new one.
Key Takeaways
Decision Trees Summed Up
To recap. A tree asks questions at decision nodes, follows branches, and reads its prediction from a leaf. Every path is an IF-THEN rule. It's built by recursive splitting: pick the purest split, then repeat inside each group. A fully grown tree can overfit. Stopping rules and pruning keep it simpler, judged on validation data. Classification trees predict labels; regression trees predict averages; and a random forest combines many trees.
Reflection
Predicting Who Finishes a Course
Something to think about. You're building a tree to predict whether a student finishes an online course. Which three questions would you let it ask? And which question could tempt it to build rules around a handful of unusual students?
About Introduction to Artificial Intelligence at UPI Study
Introduction to Artificial Intelligence (Computer Science 373) is a self-paced online course of 69 video lessons in 9 chapters, from intelligent agents and search to game strategy, neural networks, logic and reasoning, machine learning, language, robotics and AI ethics. It carries a NCCRS credit recommendation for 3 semester credits.
Every lesson transcript is free on pages like this one, and the videos are free with a free UPI Study account. Enrolling adds a practice quiz after each lesson, the graded work and final exam that earn the credit, lifetime access, and an official transcript on completion. UPI Study has helped over 130,000 students with college credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.
Enroll now → Course details Download the syllabus All Computer Science courses Every course with PRO →
Compare: AI vs Current Trends in CS and IT
Why take it here
Why students take Introduction to Artificial Intelligence with UPI Study
Where the credit goes
Universities that work with NCCRS
Introduction to Artificial Intelligence carries a NCCRS credit recommendation. These universities work with NCCRS; some are UPI Study's signed partners. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.
Thomas Edison State UniversitySigned partner
University of the PeopleSigned partner
Charter Oak State CollegeSigned partner
Strayer UniversitySigned partner
Columbia Southern UniversitySigned partner
Indiana TechSigned partner
Penn StateWorks with NCCRS
Southern New Hampshire UniversityWorks with NCCRS
SUNYWorks with NCCRS
+many more cooperating universitiesFind my college →Find my college → Not listed? Ask Athena in the corner of this page, or book a 15-minute call.
FAQ
Questions students ask us
About Splitting Data with Decision Tree Models
What is a decision tree in machine learning?
How does a decision tree decide which feature to split on?
How do you prevent a decision tree from overfitting?
What is a random forest and why is it better than a single decision tree?
About the Introduction to Artificial Intelligence course
How many college credits is Introduction to Artificial Intelligence?
How much does Introduction to Artificial Intelligence cost and what is included?
Which universities accept UPI Study credits?
How long does Introduction to Artificial Intelligence take?
How are the exams done?
How do my credits reach my college?
Ready for the full course? Earn 3 credits in Introduction to Artificial Intelligence.
69 video lessons, a quiz after each one, and the graded work that earns the credit. Recommended for credit by NCCRS.
3 credits · NCCRSEnroll →