Forward-Flowing Neural Network Structures, Explained

UPI Study › Courses › Introduction to Artificial Intelligence › Free lessons › Lesson 5.4 Free with a free account

Forward-Flowing Neural Network Structures

Introduction to Artificial Intelligence · Chapter 5: Neural Networks and Learning ArchitecturesSection 5.4 · lesson 35 of 6914 min · free with a free account

← Splitting Data with Decision Tree ModelsNext lesson: Training Networks: Backpropagation and Gradient Methods →

Earn 3 college credits for Introduction to Artificial IntelligenceRecommended for credit by NCCRS · Computer Science 373 · self-paced · one payment, lifetime access
Enroll in Introduction to Artificial Intelligence · $250 →Download the syllabus (PDF)

In this lesson

How multi-layer perceptrons compute predictions with a forward pass.

  • Describe the input, hidden, and output layers of a feedforward network
  • Compute a forward pass with weight matrices, biases, ReLU, and softmax
  • Count a network's parameters and explain the trade-off between width and depth
  • Match output activations and loss functions to regression, binary, multi-class, and multi-label tasks

Free study notes

Get this lesson's notes by email

The key points and 4 practice questions with answers, in your inbox. One email, free.

One email. Unsubscribe in one click.

Transcript

Everything said in the video, section by section (1,923 words). Read along, or search it.

Opening Analogy

A Network Like an Assembly Line

Picture a factory assembly line. Raw material enters at one end, each station changes it a little, and a finished product leaves at the other end. Nothing ever travels backward along the line. A feedforward neural network works the same way. Numbers enter, each layer transforms them, and a prediction comes out. The most common design is the multi-layer perceptron, or MLP. It's the foundation for many bigger architectures, and it's small enough to compute by hand. So let's.

Network Structure

Input, Hidden, and Output Layers

Each neuron does three things: multiply each input by a weight, add them up with a bias to get a total called z, and pass z through an activation function. Neurons are arranged in layers. The input layer simply holds the data, one value per feature. It does no calculation. One or more hidden layers sit in between. Their outputs aren't shown to the outside world, which is why they're called hidden. The output layer produces the final prediction. In a fully connected, or dense, layer, every neuron receives the outputs of all the neurons in the layer before. Information moves only forward. Real networks often stack several hidden layers.

Activation Functions

ReLU, Sigmoid, Tanh, and Linear

Four activations do most of the work. ReLU, the larger of zero and the total, is the usual choice for hidden layers. The sigmoid squeezes the total into a value between zero and one, for yes-or-no outputs. Tanh, a stretched S from minus one to one, is sometimes used in hidden layers. And linear means no activation at all: the total passes through unchanged, for outputs that predict a number.

Why Hidden Layers

Two ReLU Neurons Solve XOR

Why hidden layers? A single neuron can only separate things with a straight line, and XOR, true when exactly one input is one, can't be split that way. Two hidden neurons solve it. The first is ReLU of x1 plus x2. The second is ReLU of x1 plus x2, minus one. The output is the first, minus twice the second. Inputs zero, zero: both hidden values are zero, so the output is zero. Zero, one, or one, zero: the first is one, the second zero. Output one. One, one: the first is two, the second is one. Two minus twice one is zero. Exactly XOR, with weights chosen by hand. The hidden layer re-describes the inputs so a straight line is enough.

Key Insight

Without Activations, Layers Collapse

But the activation matters. If every layer used only weights and biases, with no activation, the whole network would collapse into a single layer. Feed one layer into another: W2 times W1 x plus b1, plus b2. Multiply it out, and you get one matrix, W2 W1, times x, plus one bias. Stacking layers would add nothing. The non-linear activation between layers is what lets each layer build on the one before it.

Forward Pass Rule

W Times Values Plus b

The forward pass is the calculation that turns an input into a prediction. Vectors and matrices keep it short, and let a computer run it fast. One neuron computes w1 x1 plus w2 x2 plus b. A layer of three neurons does that three times, with different weights. Stack the three weight rows into a matrix W, and the three biases into a vector b. Now the whole layer is one line: the total is W times the previous layer's values, plus b. Then apply the activation. W has one row per neuron in this layer, and one column per neuron in the layer before. Previous layer m, this layer n: W is n by m, and b has n entries.

Worked Example

Computing the Hidden Layer

Let's run one. Two inputs, x equals two and one. A hidden layer of three ReLU neurons, and an output layer of two. Hidden neuron one has weights one and minus one, and bias zero. One times two, minus one times one, plus zero: one. Neuron two: weights one half and one, bias minus one. One, plus one, minus one: one. Neuron three: weights minus one and one half, bias one half. Minus two, plus one half, plus one half: minus one. Apply ReLU: one stays one, one stays one, and minus one becomes zero. The hidden layer outputs one, one, zero.

Worked Example

Output Totals Through Softmax

The output layer has two neurons. The first has weights one, two, one, and bias zero: one, plus two, plus zero. A total of three. The second has weights minus one, one, three, and bias one half: minus one, plus one, plus zero, plus a half. A total of one half. These two raw totals go into softmax, which turns them into probabilities. e to the three is about 20.09, and e to the one half, about 1.65. Together, 21.73. Divide each by the total: 0.924, and 0.076. The network predicts the first category, with probability 0.924. And notice: the third hidden neuron was zeroed by ReLU, so it played no part in this answer.

Shape Checks

Seventeen Parameters and Batched Inputs

Matrix shapes give a quick error check: the number of columns in W must equal the length of the vector it multiplies. The hidden W is three by two, with three biases: nine parameters. The output W is two by three, with two biases: eight. Seventeen in all. In general, a dense layer with m inputs and n neurons has m times n weights, plus n biases. And computers rarely process one example at a time. Put many examples side by side as the columns of a matrix, and one multiplication handles them all. That's why graphics processors, built for matrix arithmetic, suit neural networks so well.

Quick Check

Parameters in a Five-by-Four Layer

Quick check. A dense layer takes five inputs, and has four neurons. How many parameters does it have: twenty, twenty-four, or twenty-five? Pause, and work it out. Twenty-four. W is four by five: twenty weights, one per connection. Then each of the four neurons has its own bias. Twenty forgets the biases. Twenty-five gives a bias to each input, but biases belong to neurons, not inputs.

Width and Depth

The Universal Approximation Theorem

Two numbers describe an MLP's size: its width, the neurons in a layer, and its depth, the number of layers. A famous result, the universal approximation theorem, says one hidden layer, with a non-linear activation, can approximate any continuous function over a limited range, as closely as you like, given enough neurons. Reassuring. But enough can be astronomically large. For some functions, a deeper network needs far fewer neurons in total than a shallow, very wide one. Depth can build features step by step. For handwritten digits, the first hidden layer might respond to short strokes, the next to curves and corners, the next to loops and crossings, and the output to the digit itself. That's an illustration. What each layer learns is not guaranteed.

Model Size

Counting Parameters for Digit Images

Size has a direct cost: every weight and bias must be stored and computed. Take 28 by 28 images, so 784 inputs, and 10 categories, with hidden layers of 128 and 64. The first layer alone has 100,480 parameters. The second, 8,256. The output, 650. In all, 109,386. Most sit in the first layer, because the input is so large. Now spend the same 96 hidden neurons three ways. One wide layer of 96: 76,330 parameters. Three layers of 32: 27,562. Six layers of 16: just 14,090. Wider layers learn more features side by side, but parameters grow fast. More layers build features from features, often with fewer parameters, but they're harder to train, and each layer adds sequential computation. Either can memorize the training data if it's much bigger than the data supports. The practical rule: start modest, and grow only while results keep improving.

Output Design

Matching Output Layer and Loss to the Task

The final layer must produce numbers that suit the task, and a loss function measures how far the output is from the correct answer: smaller is better. The two are chosen together. To predict a number, like a house price: one neuron, linear, with mean squared error. For a yes-or-no decision, like spam: one sigmoid neuron, with binary cross-entropy. To pick one label from several, like which digit: one neuron per category, softmax, with categorical cross-entropy. And for several labels at once, like photo tags, beach, dog, sunset: one sigmoid per label, with binary cross-entropy for each.

Sigmoid and Softmax

Turning Logits Into Probabilities

A sigmoid squeezes any total into zero to one: one over one plus e to the minus total. A total of zero gives a half. A total of 1.2 gives 0.77: a seventy-seven percent chance of yes. When exactly one of K categories is right, the last layer's raw totals are called logits. Softmax raises e to each logit, then divides each by their sum. Logits two, one, and 0.1 become 7.39, 2.72 and 1.11, summing to 11.21. So the probabilities are 0.659, 0.242 and 0.099. The outputs are all between zero and one, and add up to one. The largest logit gets the largest share, bigger than its logit alone suggests. And only differences matter: logits 3 and 0.5 give the same probabilities as 2.5 and 0. Software subtracts the largest logit first, to avoid overflow. With two categories, softmax of the total and zero is exactly the sigmoid. One sigmoid neuron is the compact two-way output.

Loss Functions

Mean Squared Error and Cross-Entropy

Now the losses. Mean squared error averages the squared gaps. Predict 2.5 and 4 when the truth is 3 and 5: errors of a half and one, squares of a quarter and one, so the loss is 0.625. Cross-entropy suits probabilities. It looks only at the probability p given to the correct answer, and the loss is minus the natural log of p. A confident right answer, 0.99, costs 0.01. A coin flip, 0.5, costs 0.69. A confident wrong answer, 0.01, costs 4.6. Squared error on the same probabilities levels off below one. Cross-entropy keeps growing as the model gets more confidently wrong. In our forward pass, if the right answer is category one, the loss is minus log 0.924: 0.079. If it were category two, minus log 0.076: 2.58. For a yes-or-no output, one formula covers both cases: minus the quantity y log p, plus one minus y, times log of one minus p.

Why Pairing Matters

Prediction Minus Truth as the Signal

Why does the pairing matter? Sigmoid or softmax with cross-entropy works well, because the loss grows steeply when the network is confidently wrong. That's a strong signal to correct it. With those pairs, the error signal works out to simply the prediction minus the truth. Squared error with a sigmoid output tends to learn slowly when the network is confidently wrong, because the penalty levels off. And a linear output with cross-entropy makes no sense: cross-entropy needs probabilities between zero and one.

Quick Check

Forecasting a High Temperature

Quick check. A weather model must forecast tomorrow's high temperature, in degrees. What should its last layer look like, and which loss goes with it? Pause, and decide. One linear neuron with mean squared error. A temperature is a number, and it can be any number, even below zero, so the output needs no squeezing. A sigmoid would trap the answer between zero and one. And softmax would force you to treat each possible temperature as a separate category.

Key Takeaways

Multi-Layer Perceptrons Summed Up

To recap. An MLP is a stack of dense layers, and information flows forward only. Hidden layers with non-linear activations solve problems, like XOR, that no single neuron can. The forward pass repeats one rule: W times the previous values, plus b, then an activation. Shape checks catch mistakes. Match the output to the task: linear for numbers, sigmoid for yes or no, softmax for one of many, each with its loss.

Reflection

Four Conditions in One Scan

Something to think about. A hospital wants a network that reads a scan and reports which of four conditions are present. But some patients have two at once. Should the output layer use softmax or sigmoids? And which loss would you pair with it?

About Introduction to Artificial Intelligence at UPI Study

Introduction to Artificial Intelligence (Computer Science 373) is a self-paced online course of 69 video lessons in 9 chapters, from intelligent agents and search to game strategy, neural networks, logic and reasoning, machine learning, language, robotics and AI ethics. It carries a NCCRS credit recommendation for 3 semester credits.

Every lesson transcript is free on pages like this one, and the videos are free with a free UPI Study account. Enrolling adds a practice quiz after each lesson, the graded work and final exam that earn the credit, lifetime access, and an official transcript on completion. UPI Study has helped over 130,000 students with college credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Enroll now → Course details Download the syllabus All Computer Science courses Every course with PRO →

Compare: AI vs Current Trends in CS and IT

Why take it here

Why students take Introduction to Artificial Intelligence with UPI Study

How machines search, reason and learnIntroduction to Artificial Intelligence covers how AI systems are built: intelligent agents, search and heuristics, game strategy, logic, neural networks, machine learning, language and perception, and the ethics of AI. It is a common upper-level course in computer science degrees.
Credit at a fraction of the priceThe same 3 credits can cost $750 or more at a US college. Here it is one payment, self-paced, with lifetime access.
Finish your degree soonerTransfer credits reduce the courses you still owe, so you can graduate earlier or lighten a heavy semester.
Strengthen an applicationA completed college-level course on an official transcript shows admissions and scholarship committees you can do the work.
Study on your own scheduleNo start dates, no deadlines. Watch the lessons, practice, and take the final when you are ready.
130,000+students helped with college creditsACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Where the credit goes

Universities that work with NCCRS

Introduction to Artificial Intelligence carries a NCCRS credit recommendation. These universities work with NCCRS; some are UPI Study's signed partners. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.

Thomas Edison State UniversityThomas Edison State UniversitySigned partnerUniversity of the PeopleUniversity of the PeopleSigned partnerCharter Oak State CollegeCharter Oak State CollegeSigned partnerStrayer UniversityStrayer UniversitySigned partnerColumbia Southern UniversityColumbia Southern UniversitySigned partnerIndiana TechIndiana TechSigned partnerPenn StatePenn StateWorks with NCCRSSouthern New Hampshire UniversitySouthern New Hampshire UniversityWorks with NCCRSSUNYSUNYWorks with NCCRS +many more cooperating universitiesFind my college →

Find my college → Not listed? Ask Athena in the corner of this page, or book a 15-minute call.

FAQ

Questions students ask us

About Forward-Flowing Neural Network Structures

What is a multilayer perceptron?
A multilayer perceptron, or MLP, is a feedforward neural network made of an input layer, one or more hidden layers, and an output layer, where information flows only forward. In its dense layers, every neuron receives all outputs of the layer before, computes a weighted sum plus a bias, and applies an activation. Non-linear hidden layers let it solve problems, like XOR, that one neuron cannot.
How do you count the parameters in a dense neural network layer?
A dense layer with m inputs and n neurons has m times n weights plus n biases. A layer with five inputs and four neurons therefore has 20 weights and 4 biases, or 24 parameters. A common mistake is giving each input a bias, but biases belong to neurons. In a digit-image MLP with 784 inputs and hidden layers of 128 and 64, the first layer holds 100,480 of its 109,386 parameters, because the input is large.
What does the softmax function do?
Softmax turns a layer's raw totals, called logits, into probabilities that are each between 0 and 1 and add up to 1. It raises e to each logit and divides by the sum. Logits 2, 1, and 0.1 become about 0.659, 0.242, and 0.099. Only differences between logits matter, and with two categories softmax matches the sigmoid.
Which output activation and loss function should a neural network use?
Match the output to the task. To predict a number, such as a temperature, use one linear neuron with mean squared error. For a yes-or-no decision, use one sigmoid neuron with binary cross-entropy. To pick one label from several, use softmax with categorical cross-entropy. For several labels at once, use one sigmoid per label with binary cross-entropy for each.

About the Introduction to Artificial Intelligence course

How many college credits is Introduction to Artificial Intelligence?
Introduction to Artificial Intelligence (Computer Science 373) carries a NCCRS credit recommendation for 3 semester credits. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities.
How much does Introduction to Artificial Intelligence cost and what is included?
$250, paid once, with lifetime access: all 69 video lessons, a practice quiz after each, the graded work and final exam, and an official transcript when you finish. Students who want several courses choose UPI Study PRO instead.
Which universities accept UPI Study credits?
UPI Study has signed partner agreements with universities such as Thomas Edison State University, University of the People and Charter Oak State College, and works with universities such as Penn State, SNHU and SUNY. ACE and NCCRS have over 2,100 cooperating colleges that will consider accepting credits; the final decision is with the universities. Use "Find my college" on this page to check yours.
How long does Introduction to Artificial Intelligence take?
It is self-paced with no deadlines. The 69 lessons total about 16 hours of video; most students add reading and practice time and finish in a few weeks to a few months, on their own schedule.
How are the exams done?
Quizzes and assignments are online in the course. The final exam is taken online with proctoring; a person on the UPI Study team checks assignments, usually within two days.
How do my credits reach my college?
When the course shows 100% complete, request the official transcript from your dashboard. It is sent to the college you name, and you can send it to more than one. Check your college first with "Find my college" on this page, or ask Athena.

Ready for the full course? Earn 3 credits in Introduction to Artificial Intelligence.

69 video lessons, a quiz after each one, and the graded work that earns the credit. Recommended for credit by NCCRS.

Enroll in Introduction to Artificial Intelligence →Download the syllabusEvery course with PRO →
Introduction to Artificial Intelligence
3 credits · NCCRSEnroll →