## neuron function

in which i set out all the operations that a neuron can perform together with haskell code snippetss and graphs

- logit – how a neuron calculates its potential
- output – some smooth non linear function

### logit

represents the load or activation potential of the neuron the logit z is the bias plus the sum of its input line xᵢ multiplied by the weight on that line wᵢⱼ

digraph G { rankdir = BT; a [ style=invis ] b [label="zⱼ=b + Σⱼ (yᵢ × wᵢⱼ)"] c [ style=invis ] d [ style=invis ] b -> a [ label="yⱼ = 1 / ( 1 + e⁻ᶻ)"] c -> b [ label=yᵢ] d -> b [ label=yᵢ₊₁] }

### output

- some smooth non linear function
- approx 0 when z big and negative
- approx 1 when z is big and positive
- changes smoothly inbetween
- gives it nice deriviative

#### derivatives

we need the derivatives of the output of the logistic neuron with respect to the weights for learning we first need to find

- the derivative with respect to the logit

- the derivative of logit wrt weight
the derivative of the logit with respect to wᵢⱼ is xᵢ

δz / δwᵢⱼ = xᵢ

- the derivative of logit with respect to xᵢ
similarly the derivative of the logit with respect to xi is wᵢⱼ although this appears not to be needed

δz / δxᵢ = wᵢⱼ

- the derivative of the output with respect to the logit
yⱼ = 1 / ( 1 + e⁻ᶻ) δy / δz = y × (1 - y)

- first find the derivative of the output with respect to the weight
use chain rule

yⱼ = 1 / ( 1 + e⁻ᶻ) δyⱼ / δwᵢⱼ = (δz / δwᵢⱼ) × (δyⱼ / δzⱼ) = xᵢ × yⱼ ×(1 - yⱼ)

- explain chain rule

- then find the derivative of the error with respect to the weights
so the way that the error changes as we change a weight is the sum over all the training cases ₙ of the value on the input line xᵢⁿ multiplied by and the residual tⁿ – yⁿ multiplied by the slope of the logistic function yⱼⁿ × (1 – yⁿ)

note that ⁿ represents values for a particular training cast t represents the target output for a neuron ᵢ represents an input layer ⱼ represents an output layer wᵢⱼ represents a weight connecting an output from layer ᵢ to a neuron in layer ⱼ

yⱼ = 1 / ( 1 + e⁻ᶻ) δEⱼ / δwᵢⱼ = Σₙ (δyⁿ / δwᵢ) × (δE / δyⁿ) = - Σₙ xᵢⁿ × yⱼⁿ × (1 - yⁿ) × (tⁿ - yⁿ)

import Data.Graph --http://web.engr.oregonstate.edu/~erwig/papers/abstracts.html#JFP01 type VertexId = String type ParentVertexIds = [VertexId] type VertexDef a = (a, VertexId, [VertexId]) data LayerDef = LayerDef {getLayerId::String, getLayerSize::Integer} data NetworkDef = NetworkDef {getLayerDef::LayerDef, getSubLayerDefs::[NetworkDef]} data TrainingCase = TrainingCase { trainingInputs::[Double] , trainingExpectedOutputs::[Double]} data Neuron = Neuron { neuronBias::Double} deriving Show buildVertexDef :: ParentVertexIds -> String -> Integer -> VertexDef Neuron buildVertexDef parentlayerkeys lid neuronidx = (Neuron 0 0 0,"layer"++lid ++ show neuronidx, parentlayerkeys) buildLayerDef :: ParentVertexIds -> LayerDef -> [VertexDef Neuron] buildLayerDef pkeys ldef = map (buildVertexDef pkeys lid) [1..n] where lid = getLayerId ldef n = getLayerSize ldef buildNetworkDef :: ParentVertexIds ->NetworkDef -> [VertexDef Neuron] buildNetworkDef pkeys (NetworkDef ldef []) = buildLayerDef pkeys ldef buildNetworkDef pkeys (NetworkDef ldef subs) = concat $ thislayer : subdefs where thislayer = buildLayerDef pkeys ldef thislayerkeys = map (\(_, k, _) -> k) thislayer subdefs = map (buildNetworkDef thislayerkeys) subs myNetWorkSchema::NetworkDef myNetWorkSchema = NetworkDef (LayerDef "i" 5) [layerj] where layerj = NetworkDef (LayerDef "j" 5) [] myNetWorkDef::[VertexDef Neuron] myNetWorkDef = buildNetworkDef [] myNetWorkSchema myNetwork :: (Graph,Vertex -> VertexDef Neuron,VertexId -> Maybe Vertex) myNetwork = graphFromEdges myNetWorkDef neuronWeights :: Neuron -> [Double] neuronWeights n = undefined neuronInputs :: Neuron -> [Double] neuronInputs n = undefined neuronLogit :: Neuron -> Double neuronLogit n = b + $ sum ws where zs = zipWith (*) is ws ws = neuronWeights n is = neuronInputs n b = neuronBias n neuronOutput:: Neuron -> Double neuronOutput = 1 / (1 + exp (-z)) where z =logit n ewrtws :: TrainingCase -> Neuron -> [Double] ewrtws tc n = undefined