Neural Networks Explained with a Bit of Math and a Duck

Neural Networks

Neural networks used in machine learning behave in ways very similar to how our brains work. A neuron receives input, often from a collection of other neurons. Based on that input, if a threshold is crossed, the neuron fires and passes information to the next layer of neurons.

In machine learning, we feed the model, data. That data is received by neurons (same concept). A function is then performed and if a certain threshold (same concept) is met , the neuron fires and sends information to the next layer of neurons (again, same concept).

It’s possible to build a simple machine learning model with only one layer containing a few neurons. But DNNs (Deep Neural Networks) can be comprised of thousands of neurons in thousands of layers. If you’re new to ML, it may seem like magic, but here’s a peek into what is happening.

Input Data

X1 and X2 in this diagram represent two inputs being sent to our neuron, the circle on the right. Let’s say you are trying to build a ML model to identify pictures that contain ducks.

X1 could represent that it has feathers. X2 could indicate that it has webbed feet. Are these inputs equal in determining if the picture is of a duck? No. Feathers are found on all birds, but webbed feet are only on a very small subset of birds. Therefore, when trying to identify a duck, our brains give more weight to the input of webbed feet than we give to the input of feathers.

Some Inputs Carry More Weight

That’s what W1 and W2 represent. They are the different weights we give each input. Weight is basically just a multiplier. So maybe, in this example, the importance of webbed feet is multiplied by 1.6, whereas the importance of feathers has .4 for a multiplier.

When expressed mathematically, we’re saying take (Weight1 x Input1) and (Weight2 x Input2). Then add those two values together. When we do that, we get a value. If that value is greater than a set threshold, the neuron will fire.

Greater Than the Threshold

The letter b stands for bias. If you think of bias in a very general sense, you take sides. In machine learning, bias is another way of saying threshold. It’s defining which side we landed on. Fire or Don’t Fire? Duck or Not Duck? Threshold Crossed or Not Crossed. We read the equation like this: “Weight the first input, weight the second input, and add them together. If the total is greater than b, the specified threshold, fire the neuron. If it is less than or equal to the threshold, don’t fire.

Let’s say that the threshold is 70%. If the animal has feathers, maybe there’s a 5% chance it’s a duck. But if it has feathers and webbed feet (both inputs), there’s say an 82% chance it’s a duck. If the threshold is set to 70, and this represented our entire model, the output would indicate that your picture is probably showing a duck.