Neural Networks vs Genetic Algorithms: Part 1 - Introduction to FeedForward ANN

For the purposes of this blog, I am primarily interested in discussing two artificial models of biological phenomena, namely Artificial Neural Networks and Genetic Algorithms.
Neither of these topics is trivial, however, neither is really as complex as it appears.
Although unrelated, they can be used together to create something more interesting than the sum of its components - let's start with Neural Networks.

They're a model of what we think is happening inside the brains of those living entities of high enough order to actually have a brain, which of course includes humans, but also includes the lowliest invertebrates and even some creatures whose brains are 'unconventional'.
According to the theoretical model, the brain is constructed of 'Neurons', which are effectively simple biological logic gates that, when sufficiently interconnected, are capable of performing more complex calculations, and more abstract tasks such as pattern recognition (image recognition, face tracking, optical character recognition, gesture recognition, voice command recognition and voice to text translation, the list really does go on and on, and I sometimes wonder what problems exist where they 'cannot' be applied.

A theoretical neuron looks a bit like a squid or jellyfish, typically it has lots of 'inputs' that can be visualized as tentacles, with which the neuron receives external stimula, and it has a singular 'head' producing one output - each input has its own 'weight' which could be visualized as 'electrical resistance per input tentacle', and finally, each neuron has one additional input, known as Bias, which is not set up to sense the environment like the other inputs, but instead takes a fixed value, typically a constant value - like the other inputs it has a Weight (or resistance), but unlike the others, it is not connected to anything else.

There's a simple mathematical formula that we can apply to any neuron which will result in a value known as the 'activation sum', and a related function known as a 'Sigmoid Operator' whose purpose is to force Activation Sums back into the Normal number range - simply put, based on the neuron's inputs and weights, the sigmoid-transformed activation sum will indicate whether that neuron 'fires' or not, producing a binary output at its head.
Neurons are typically connected together in layers - the output of each neuron in layer N is connected to one of the inputs owned by EACH neuron in layer N+1 - the network is Directed, and Highly-Connected... Therefore, outputs on each neuron in a layer have a varying effect on neurons in all subsequent layers. Of course, real biological brains don't quite operate this way - but they are very similar to the theoretical model, enough so that this model can and does work very well, even if it's not yet clear how to make any use of this stuff, please be patient, as each post will contribute to a greater understanding of how game developers can take advantage of this technology, and how it's relevant to modern game developers targeting today's consumer devices.

Neurons do not merely resemble logic gates - one neuron can be used to implement any simple logic gate - the kind of logic gates that make your computer work!
You see, both biological neurons, and their artificial counterparts, can be 'trained' - and this is where it gets interesting for game developers, and other kinds of engineers... for any input signal pattern, we can teach a neuron to produce a known output - this holds true not just for simple neurons in isolation, but also for entire networks of them - given pattern X at its inputs, we can teach a neural network to produce pattern Y at its outputs - and given pattern Y, it can produce pattern Z - you can think of a neural network as a black box that can LEARN to map any given input to any given output - rather like a reprogrammable 'PLC' (programmable logic controller) you can find in some industrial electronics, this is able to do something that conventional electronic hardware (like your computer) cannot do at the hardware level - its logic can be reprogrammed to produce ideal logical outcomes for a set of given input scenarios. The number of i/o mappings supported by a neural network is related directly to the complexity of the network (number of neurons, connectivity, etc.) and inversely to the accuracy of the output - that is to say, the less mappings we teach a network, the more reliably it can produce the outputs we taught it, even when the inputs are 'dirty'. The more different outcomes we attempt to map (to a network of constant complexity), the less reliable the network output becomes (since previously-learned mappings can be corrupted), even when the input is nice and clean... and the only two ways to 'repair' such a scenario, are to spend more time and iterations training the network to increase its reliability, or to increase the network's complexity (by adding more neurons and connections) which again requires more training, more time, to achieve a high degree of confidence in the accuracy of the output for arbitrary inputs.

The way that neural networks receive 'training' can vary - the typical 'supervised training' applied to consumer software based on this sort of tech today (voice recognition for example) involves showing a set of inputs to the network, looking at the outputs that produced, comparing them to a 'desirable' set of outputs, measuring the error term, and then propagating the error term 'backwards' through the network (from output layer, back towards the input layer), which results in adjustment to the 'input weights' at each neuron in the network. This takes a lot of iterations (developer time) and does achieve good results in terms of producing a 'smarter network' however there are other training methods based on self-learning which may be more applicable to games, for example allowing the AI to directly learn from player inputs during gameplay, that's a technique that some car racing games have used to teach braindead AI drivers how to race, but it's not the technique that intrigues me the most, because it still relies on humans to help improve the outcome.
My next post or two will look at a machine learning technique that does not rely on human intervention of any kind, it is equally capable of self-improvement, with or without human players being involved.