How Neural Networks Drive Machine Learning

UPI Study › Courses › Introduction to Artificial Intelligence › Free lessons › Lesson 5.2 Free with a free account

How Neural Networks Drive Machine Learning

Introduction to Artificial Intelligence · Chapter 5: Neural Networks and Learning ArchitecturesSection 5.2 · lesson 33 of 6916 min · free with a free account

← Probabilistic Graphical Models for LearningNext lesson: Splitting Data with Decision Tree Models →

Earn 3 college credits for Introduction to Artificial IntelligenceRecommended for credit by NCCRS · Computer Science 373 · self-paced · one payment, lifetime access
Enroll in Introduction to Artificial Intelligence · $250 →Download the syllabus (PDF)

In this lesson

How artificial neurons and perceptrons work, and what one neuron cannot do.

  • Compute an artificial neuron's output from inputs, weights, bias, and activation
  • Explain how a perceptron draws a linear decision boundary and computes basic logic gates
  • Prove why XOR is not linearly separable for a single neuron
  • Contrast hand-designed features with features learned by neural networks

Free study notes

Get this lesson's notes by email

The key points and 4 practice questions with answers, in your inbox. One email, free.

One email. Unsubscribe in one click.

Transcript

Everything said in the video, section by section (2,239 words). Read along, or search it.

Opening Question

One Tiny Ingredient Behind Many Apps

Your phone recognizes your face. A photo app finds every picture of your dog. A voice assistant turns your speech into text. Very different jobs. Yet many of them are built from the same tiny ingredient, repeated thousands or millions of times: the artificial neuron. A single neuron is surprisingly simple. It takes some numbers in, weighs them, adds them up, and decides whether to fire. So what can one neuron do on its own? Where does it fail? And how do networks of them find their own patterns in data?

Biological Inspiration

Dendrites, Synapses, and Axons

The idea comes from the brain, which has roughly eighty-six billion nerve cells, called neurons. Each one connects to thousands of others. Branching fibers called dendrites receive signals from other neurons. Each connection point is a synapse, and synapses differ in strength. Some excite the receiving neuron; others hold it back. The cell body combines all the incoming influences. If the total is strong enough, the neuron fires a signal down its axon. If not, it stays quiet. Learning in the brain is thought to involve changing synapse strengths. That idea, knowledge living in the strength of connections, is what artificial neurons borrowed.

Analogy

From Brain Cell to Artificial Neuron

The artificial neuron mirrors those parts. Signals arriving at the dendrites become inputs. Synapse strengths become weights: how much each input matters. The cell body becomes a weighted sum, plus a bias. The firing threshold becomes an activation function, and the signal along the axon becomes the output. But it's a loose inspiration, not a copy. Real neurons send timed pulses, change over time, and use many chemical messengers. Engineers use the idea because it works.

History

McCulloch and Pitts, Rosenblatt, Minsky and Papert

The history is short. In 1943, Warren McCulloch and Walter Pitts proposed a simple mathematical model of a neuron. In 1958, Frank Rosenblatt introduced the perceptron, a neuron whose weights could be set from examples, built as a machine for recognizing simple images. And in 1969, Marvin Minsky and Seymour Papert published Perceptrons, showing what a single neuron cannot do. That limit helped cool interest in neural networks for years.

How a Neuron Computes

Weighted Sum, Bias, and Activation

An artificial neuron computes in two steps. Step one: multiply each input by its weight, add them up, and add the bias. Call the total z. Step two: pass z through an activation function, f, to get the output y. A large positive weight pushes the neuron to fire. A negative weight holds it back. A weight near zero means that input is ignored. Think of it as a weighted vote. Each input votes with the strength of its weight, and the bias sets how many votes are needed for a yes. That's why a bias is often described as a negative threshold. With a bias of minus 2.8, the weighted sum must reach 2.8 before the neuron fires.

Worked Example

A Neuron That Flags Spam

Here's one at work. A neuron flags spam, looking at three yes-or-no clues about an email. Does it contain the word free? Weight two. Is the sender unknown? Weight one and a half. Does it have lots of exclamation marks? Weight one. The bias is minus 2.8, and it uses the step rule: output one, flag it, when z is zero or more. An email arrives containing free, from an unknown sender, with no exclamation marks. z is two, plus one and a half, plus zero, minus 2.8. That's 0.7, which is zero or more, so the neuron fires: flagged.

Worked Example

Which Clue Combinations Trigger It

Try the other combinations. Free alone gives z of minus 0.8: not enough. No single clue is enough on its own. Free with lots of exclamation marks gives 0.2: flagged. All three give 1.7. But an unknown sender with exclamation marks gives minus 0.3. Just short. So free, together with either other clue, passes the bar. That's what weights and bias do: they decide which combinations of evidence trigger the neuron.

Parameters

How Weights and Bias Change Behavior

Change the settings, and the behavior changes. Raise one weight, and that input counts for more. Make a weight negative, and it argues against firing. Raise the bias, say from minus 2.8 to minus 1.8, and the neuron fires more easily, flagging more emails. Lower it to minus 3.8, and it's harder to trigger. The weights and bias are the neuron's parameters. Everything it knows lives in them. Here we set them by hand; finding them from data is training, a later topic.

Quick Check

Does the Sprinkler Neuron Fire

Quick check. A neuron switches on a garden sprinkler. Dry soil has weight two. A sunny day, weight one. Rain in the forecast, weight minus three. The bias is minus one and a half, and it fires when z is zero or more. Today the soil is dry, it's sunny, and rain is forecast. Does the sprinkler neuron fire? Pause, and work it out. It stays quiet. z is two, plus one, minus three, minus one and a half: minus one and a half. That's below zero. The rain forecast's negative weight cancels both reasons to water, and the bias still has to be overcome. Remove the rain forecast, and z becomes plus one and a half, so it fires.

Perceptrons

Computing AND, OR, NAND, and NOR

A neuron with that step activation, output one or zero, is called a perceptron. It answers a yes-or-no question. The spam flagger is one. Even a single perceptron can compute basic logic, with inputs of zero for false and one for true. AND: weights one and one, bias minus 1.5. OR: the same weights, bias minus 0.5. NAND and NOR flip the signs: weights minus one, with bias 1.5 or 0.5. Check AND. Both inputs on: one plus one minus 1.5 is 0.5, so it outputs one. Only one on: minus 0.5, so it outputs zero. The bias demands both.

Decision Boundary

A Straight Line Splits the Inputs

Here's a picture of what a perceptron does. With two inputs, the four possible input pairs sit at the corners of a square. The equation, w1 x1 plus w2 x2 plus b equals zero, is a straight line. Points on one side output one; points on the other side output zero. For AND, the line is x1 plus x2 equals 1.5. Only the top-right corner is above it. For OR, the line is x1 plus x2 equals 0.5, and only the origin falls below. The weights tilt the line, and the bias slides it. With more inputs, the line becomes a flat sheet, but the idea is the same. This line is the decision boundary.

Activation Functions

Sigmoid, Tanh, and ReLU

The step is only one choice of activation. Modern neurons usually use smoother rules. The sigmoid, one over one plus e to the minus z, is a smooth S between zero and one. Its output can be read as a confidence. Tanh is a smooth S between minus one and one, centered on zero. And ReLU, the larger of zero and z, is zero for negative z, then rises in a straight line. It's very widely used. At z of minus two, zero and two, the sigmoid gives 0.12, 0.5 and 0.88. Tanh gives minus 0.96, zero, and 0.96. ReLU gives zero, zero, and two. Why have an activation at all? Without one, a whole network of neurons collapses into one big weighted sum, no more powerful than a single neuron. The non-linearity is the power.

Worked Example

Same z, Four Different Responses

One worked example. Inputs two and one, weights one half and minus one, bias 0.3. z is one half times two, minus one times one, plus 0.3: that's 0.3. A step outputs one. A sigmoid gives 0.574, about fifty-seven percent confident. Tanh gives 0.291, and ReLU passes 0.3 straight through. Same z, four different responses. Smooth activations also make the weights easier to improve by training, a point we'll come back to.

The XOR Problem

No Single Line Separates XOR

Now the famous limit. XOR, exclusive or, is true when exactly one of its two inputs is true. Either one alone, but not both, and not neither. Put its outputs on the square. The two ones sit at opposite corners, and the two zeros sit at the other two. No straight line can put both ones on one side and both zeros on the other. Try it: any line that separates one pair cuts across the other. We can prove it with arithmetic. Suppose some weights and a bias worked. The four rows would demand: b below zero; w2 plus b at least zero; w1 plus b at least zero; and w1 plus w2 plus b below zero. Add the middle two: w1 plus w2 plus b is at least minus b. And since b is negative, minus b is positive. So that sum is positive, but the last row says it's negative. Contradiction. No weights and bias exist.

Linear Separability

The Limit of One Neuron

A problem is linearly separable if a single straight line, or flat sheet, can separate the two answers. A single neuron can solve only those. AND and OR are; XOR isn't. XOR is tiny, but it exposed a hard limit, and many real problems aren't linearly separable, like telling a cat from a dog from raw pixels. Minsky and Papert's 1969 analysis discouraged funding and research for years. But the limit applies to one neuron. XOR can be built from simpler gates that single neurons can compute, so combining neurons removes it. That's where the next topics go.

Big Picture

Neural Networks Within Machine Learning

So where do neural networks fit in machine learning? Machine learning means programs that improve at a task by learning from data, rather than following step-by-step rules. Neural networks are one family of methods, alongside decision trees, nearest-neighbor methods and linear models. What makes them special is how they handle features. A feature is a measurable property of the input that helps with the task. For handwritten digits, features might be the number of loops: a zero has one, an eight has two. Or the number of stroke endpoints, or whether there's a long vertical stroke. The learning method works with those features, so the choice of features strongly affects how well it can do.

Features

Hand-Designed Versus Learned Features

There are two ways to get features. Hand-designed features: experts decide what to measure, write code to compute it, and hand those numbers to a simple model. That takes skill, time and knowledge, and it can miss important patterns. Learned features: a neural network gets raw data, such as pixel values. Its weights adjust so that its neurons come to respond to whatever patterns are useful. The network discovers its own features. That's called representation learning.

Worked Example

A Neuron That Detects Vertical Lines

Even one neuron shows how this works. Its weights act as a template: it responds most when the input matches the pattern of its weights. This neuron looks at a three by three patch of pixels: one for dark, zero for blank. Its weights are plus two down the middle column, and minus one on the sides. Its bias is minus three. A vertical line down the middle puts all three dark pixels on the plus twos. The sum is six, z is three, and it fires. A horizontal line covers one plus two and two minus ones: a sum of zero, so z is minus three. Quiet. A diagonal gives the same. A vertical line at the left edge covers three minus ones: z is minus six. This neuron has become a detector for one feature: a vertical line in the middle. Here we chose the weights by hand. In a real network they're learned, and later neurons combine these detections into richer features.

Quick Check

Testing the Detector on a Plus Sign

Quick check, with the same neuron. This time the patch shows a plus sign: the whole middle column is dark, and so is the whole middle row. That's five dark pixels. Work out the total. Does the neuron fire? Pause, and decide. It fires. The middle column gives three plus twos: six. The two extra dark pixels in the middle row sit on minus ones: minus two. The sum is four, minus the bias of three: z is one, so it fires. The side pixels pull it down, but not enough to cancel the vertical line.

Comparison

Traditional Methods Versus Neural Networks

Compare the two approaches. Traditional machine learning uses features people design. Neural networks learn them from raw data: pixels, sound waves, text. Traditional methods need high effort on features, but often work with modest data, and their features are easy to explain. Neural networks need less feature design, but usually need lots of examples, large computing power, and their learned features can be hard to interpret. They can also learn mistakes or biases present in their data. In return, they cope with patterns too complex for people to describe in rules, and they improve as data grows.

Method

Five Steps to Work Out a Neuron

To work out any neuron by hand, use five steps. List the inputs. List the weights and bias. Multiply and add. Add the bias. Activate. Then sanity-check. Does each weight's sign match whether that input should push toward firing? Does raising the bias make firing easier? And is the output in the activation's range? A sigmoid can never give 1.4.

Key Takeaways

Neurons, Perceptrons, and XOR Summed Up

To recap. An artificial neuron weighs its inputs, adds a bias, and passes the total through an activation function. A perceptron uses a step, splits its inputs with a straight line, and can compute AND, OR, NAND and NOR. XOR isn't linearly separable, so no single neuron can compute it. And neural networks learn their features from raw data, with weights acting as templates.

Reflection

Spotting Ripe Fruit From Photos

Something to think about. Imagine teaching a computer to spot ripe fruit from photos. Which features might a person design by hand, like color, shape or size? What would a neuron looking at raw pixels have to discover for itself? And in which case would you worry more about it learning something misleading?

About Introduction to Artificial Intelligence at UPI Study

Introduction to Artificial Intelligence (Computer Science 373) is a self-paced online course of 69 video lessons in 9 chapters, from intelligent agents and search to game strategy, neural networks, logic and reasoning, machine learning, language, robotics and AI ethics. It carries a NCCRS credit recommendation for 3 semester credits.

Every lesson transcript is free on pages like this one, and the videos are free with a free UPI Study account. Enrolling adds a practice quiz after each lesson, the graded work and final exam that earn the credit, lifetime access, and an official transcript on completion. UPI Study has helped over 130,000 students with college credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Enroll now → Course details Download the syllabus All Computer Science courses Every course with PRO →

Compare: AI vs Current Trends in CS and IT

Why take it here

Why students take Introduction to Artificial Intelligence with UPI Study

How machines search, reason and learnIntroduction to Artificial Intelligence covers how AI systems are built: intelligent agents, search and heuristics, game strategy, logic, neural networks, machine learning, language and perception, and the ethics of AI. It is a common upper-level course in computer science degrees.
Credit at a fraction of the priceThe same 3 credits can cost $750 or more at a US college. Here it is one payment, self-paced, with lifetime access.
Finish your degree soonerTransfer credits reduce the courses you still owe, so you can graduate earlier or lighten a heavy semester.
Strengthen an applicationA completed college-level course on an official transcript shows admissions and scholarship committees you can do the work.
Study on your own scheduleNo start dates, no deadlines. Watch the lessons, practice, and take the final when you are ready.
130,000+students helped with college creditsACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Where the credit goes

Universities that work with NCCRS

Introduction to Artificial Intelligence carries a NCCRS credit recommendation. These universities work with NCCRS; some are UPI Study's signed partners. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Thomas Edison State UniversityThomas Edison State UniversitySigned partnerUniversity of the PeopleUniversity of the PeopleSigned partnerCharter Oak State CollegeCharter Oak State CollegeSigned partnerStrayer UniversityStrayer UniversitySigned partnerColumbia Southern UniversityColumbia Southern UniversitySigned partnerIndiana TechIndiana TechSigned partnerPenn StatePenn StateWorks with NCCRSSouthern New Hampshire UniversitySouthern New Hampshire UniversityWorks with NCCRSSUNYSUNYWorks with NCCRS +many more cooperating universitiesFind my college →

Find my college → Not listed? Ask Athena in the corner of this page, or book a 15-minute call.

FAQ

Questions students ask us

About How Neural Networks Drive Machine Learning

How does an artificial neuron work?
An artificial neuron multiplies each input by its weight, adds the results, adds a bias, and passes that total, z, through an activation function to produce an output. Positive weights push it toward firing, negative weights hold it back, and weights near zero mean an input is ignored. The bias acts like a negative threshold: with a bias of -2.8, the weighted sum must reach 2.8.
What is a perceptron in machine learning?
A perceptron is an artificial neuron with a step activation that outputs 1 or 0, answering a yes-or-no question. Frank Rosenblatt introduced it in 1958. Its decision boundary is a straight line, or a flat sheet with more inputs. With weights 1 and 1, a bias of -1.5 makes it compute AND, and a bias of -0.5 makes it compute OR.
Why can't a single perceptron solve the XOR problem?
A single perceptron cannot solve XOR because XOR is not linearly separable: its two true outputs sit at opposite corners of the input square, so no straight line separates them from the false ones. Minsky and Papert's 1969 book Perceptrons highlighted such limits. Combining several neurons removes the limit, because XOR can be built from gates that single neurons compute.
Why do neural networks need activation functions?
Activation functions add non-linearity; without them, a whole network of neurons collapses into one big weighted sum, no more powerful than a single neuron. Common choices are the sigmoid, which gives a smooth S from 0 to 1; tanh, from -1 to 1; and ReLU, the larger of 0 and z. At z = 2, they give about 0.88, 0.96, and 2.

About the Introduction to Artificial Intelligence course

How many college credits is Introduction to Artificial Intelligence?
Introduction to Artificial Intelligence (Computer Science 373) carries a NCCRS credit recommendation for 3 semester credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.
How much does Introduction to Artificial Intelligence cost and what is included?
$250, paid once, with lifetime access: all 69 video lessons, a practice quiz after each, the graded work and final exam, and an official transcript when you finish. Students who want several courses choose UPI Study PRO instead.
Which universities accept UPI Study credits?
UPI Study has signed partner agreements with universities such as Thomas Edison State University, University of the People and Charter Oak State College, and works with universities such as Penn State, SNHU and SUNY. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities. Use "Find my college" on this page to check yours.
How long does Introduction to Artificial Intelligence take?
It is self-paced with no deadlines. The 69 lessons total about 16 hours of video; most students add reading and practice time and finish in a few weeks to a few months, on their own schedule.
How are the exams done?
Quizzes and assignments are online in the course. The final exam is taken online with proctoring; a person on the UPI Study team checks assignments, usually within two days.
How do my credits reach my college?
When the course shows 100% complete, request the official transcript from your dashboard. It is sent to the college you name, and you can send it to more than one. Check your college first with "Find my college" on this page, or ask Athena.

Ready for the full course? Earn 3 credits in Introduction to Artificial Intelligence.

69 video lessons, a quiz after each one, and the graded work that earns the credit. Recommended for credit by NCCRS.

Enroll in Introduction to Artificial Intelligence →Download the syllabusEvery course with PRO →
Introduction to Artificial Intelligence
3 credits · NCCRSEnroll →