Perceptrons and Sigmoid

Page 1

Two important types of artificial neuron (the perceptron and the sigmoid neuron)

standard learning algorithm for neural networks, known as stochastic gradient descent

Perceptrons

A perceptron takes several binary inputs, x1,x2,…x1,x2,…, and produces a single binary output

In the example shown the perceptron has three inputs, x1,x2,x3x1,x2,x3. In general it could have more or fewer inputs. Rosenblatt proposed a simple rule to compute the output. He introduced weights, w1,w2,…w1,w2,…, real numbers expressing the importance of the respective inputs to the output. The neuron's output, 00 or 11, is determined by whether the weighted sum ∑jwjxj∑jwjxj is less than or greater than some threshold value. Just like the weights, the threshold is a real number which is a parameter of the neuron. To put it in more precise algebraic terms:

output={01if ∑jwjxj≤ thresholdif ∑jwjxj> threshold(1)(1)output={0if ∑jwjxj≤ threshold1if ∑jwjxj> threshold

That's all there is to how a perceptron works!

Let's simplify the way we describe perceptrons. The condition ∑jwjxj>threshold∑jwjxj>threshold is cumbersome, and we can make two notational changes to simplify it. The first change is to write ∑jwjxj∑jwjxj as a dot product, w⋅x≡∑jwjxjw⋅x≡∑jwjxj, where ww and xx are vectors whose components are the weights and inputs

Using the bias instead of the threshold, the perceptron rule can be rewritten:

output={01if w⋅x+b≤0if w⋅x+b>0(2)(2)output={0if w⋅x+b≤01if w⋅x+b>0

You can think of the bias as a measure of how easy it is to get the perceptron to output a 11. Or to put it in more biological terms, the bias is a measure of how easy it is to get the perceptron to fire. For a perceptron with a really big bias, it's extremely easy for the perceptron to output a 11.

Sigmoid Neurons

Sigmoid neurons are similar to perceptrons, but modified so that small changes in their weights and bias cause only a small change in their output. That's the crucial fact which will allow a network of sigmoid neurons to learn.

Just like a perceptron, the sigmoid neuron has inputs, x1,x2,…x1,x2,…. But instead of being just 00 or 11, these inputs can also take on any values between 00 and 11. So, for instance, 0.638…0.638… is a valid input for a sigmoid neuron. Also just like a perceptron, the sigmoid neuron has weights for each input, w1,w2,…w1,w2,…, and an overall bias, bb. But the output is not 00 or 11. Instead, it's σ(w⋅x+b)σ(w⋅x+b), where σσ is called thesigmoid function**Incidentally, σσ is sometimes called the logistic function, and this new class of neurons called logistic neurons. It's useful to remember this terminology, since these terms are used by many people working with neural nets. However, we'll stick with the sigmoid terminology., and is defined by:

σ(z)≡11+e−z.(3)(3)σ(z)≡11+e−z.

To put it all a little more explicitly, the output of a sigmoid neuron with inputs x1,x2,…x1,x2,…, weights w1,w2,…w1,w2,…, and bias bb is

11+exp(−∑jwjxj−b).(4)(4)11+exp⁡(−∑jwjxj−b).

At first sight, sigmoid neurons appear very different to perceptrons. The algebraic form of the sigmoid function may seem opaque and forbidding if you're not already familiar with it. In fact, there are many similarities between perceptrons and sigmoid neurons, and the algebraic form of the sigmoid function turns out to be more of a technical detail than a true barrier to understanding.

Page 2