Neuron function

neuron function

in which i set out all the operations that a neuron can perform together with haskell code snippetss and graphs

  • logit – how a neuron calculates its potential
  • output – some smooth non linear function

logit

represents the load or activation potential of the neuron the logit z is the bias plus the sum of its input line xᵢ multiplied by the weight on that line wᵢⱼ

digraph G {
rankdir = BT;
  a [ style=invis ]
  b [label="zⱼ=b + Σⱼ (yᵢ × wᵢⱼ)"] 
  c [ style=invis ]
  d [ style=invis ]
  b -> a [ label="yⱼ = 1 / ( 1 + e⁻ᶻ)"]
  c -> b [ label=yᵢ]
  d -> b [ label=yᵢ₊₁]
}

logit.png

output

  • some smooth non linear function
  • approx 0 when z big and negative
  • approx 1 when z is big and positive
  • changes smoothly inbetween
    • gives it nice deriviative

derivatives

we need the derivatives of the output of the logistic neuron with respect to the weights for learning we first need to find

  • the derivative with respect to the logit
  • the derivative of logit wrt weight

    the derivative of the logit with respect to wᵢⱼ is xᵢ

    δz / δwᵢⱼ = xᵢ
    

  • the derivative of logit with respect to xᵢ

    similarly the derivative of the logit with respect to xi is wᵢⱼ although this appears not to be needed

    δz / δxᵢ = wᵢⱼ
    

  • the derivative of the output with respect to the logit
    yⱼ = 1 / ( 1 + e⁻ᶻ)
    δy / δz = y × (1 - y)
    

  • first find the derivative of the output with respect to the weight

    use chain rule

    yⱼ = 1 / ( 1 + e⁻ᶻ)
    δyⱼ / δwᵢⱼ = (δz / δwᵢⱼ) × (δyⱼ / δzⱼ) = xᵢ × yⱼ ×(1 - yⱼ)
    

    • explain chain rule
  • then find the derivative of the error with respect to the weights

    so the way that the error changes as we change a weight is the sum over all the training cases ₙ of the value on the input line xᵢⁿ multiplied by and the residual tⁿ – yⁿ multiplied by the slope of the logistic function yⱼⁿ × (1 – yⁿ)

    note that ⁿ represents values for a particular training cast t represents the target output for a neuron ᵢ represents an input layer ⱼ represents an output layer wᵢⱼ represents a weight connecting an output from layer ᵢ to a neuron in layer ⱼ

    yⱼ = 1 / ( 1 + e⁻ᶻ)
    δEⱼ / δwᵢⱼ =
    Σₙ (δyⁿ / δwᵢ) × (δE / δyⁿ) = - Σₙ xᵢⁿ × yⱼⁿ × (1 - yⁿ) × (tⁿ - yⁿ)
    
    import Data.Graph
      
    --http://web.engr.oregonstate.edu/~erwig/papers/abstracts.html#JFP01
    
                                                                               
    type VertexId = String
    type ParentVertexIds = [VertexId]
    type VertexDef a = (a, VertexId, [VertexId])                                  
      
    data LayerDef = LayerDef {getLayerId::String, getLayerSize::Integer}
    data NetworkDef = NetworkDef {getLayerDef::LayerDef, getSubLayerDefs::[NetworkDef]}
    
    data TrainingCase = TrainingCase { trainingInputs::[Double] , trainingExpectedOutputs::[Double]}
    data Neuron  = Neuron { neuronBias::Double} deriving Show
    
      
    buildVertexDef :: ParentVertexIds -> String -> Integer -> VertexDef Neuron
    buildVertexDef parentlayerkeys lid neuronidx = (Neuron 0 0 0,"layer"++lid ++ show neuronidx, parentlayerkeys)
      
    buildLayerDef :: ParentVertexIds  -> LayerDef -> [VertexDef Neuron]
    buildLayerDef pkeys ldef = map (buildVertexDef pkeys lid) [1..n]
                               where lid = getLayerId ldef
                                     n = getLayerSize ldef
      
    buildNetworkDef :: ParentVertexIds ->NetworkDef -> [VertexDef Neuron]
    buildNetworkDef pkeys (NetworkDef ldef [])   = buildLayerDef pkeys ldef                    
    buildNetworkDef pkeys (NetworkDef ldef subs) =  concat $ thislayer : subdefs                         
                                                      where thislayer = buildLayerDef pkeys ldef
                                                            thislayerkeys = map (\(_, k, _) -> k) thislayer
                                                            subdefs =   map (buildNetworkDef thislayerkeys) subs
      
    myNetWorkSchema::NetworkDef
    myNetWorkSchema = NetworkDef (LayerDef "i" 5) [layerj] 
                     where layerj = NetworkDef (LayerDef "j" 5) []
      
    myNetWorkDef::[VertexDef Neuron]                     
    myNetWorkDef = buildNetworkDef [] myNetWorkSchema                                   
      
    myNetwork :: (Graph,Vertex -> VertexDef Neuron,VertexId -> Maybe Vertex)
    myNetwork = graphFromEdges myNetWorkDef
    
    
    neuronWeights :: Neuron  -> [Double]
    neuronWeights n = undefined
    
    neuronInputs :: Neuron -> [Double]
    neuronInputs n = undefined
    
    neuronLogit :: Neuron  -> Double
    neuronLogit n = b + $ sum ws 
      where zs = zipWith (*) is ws
            ws = neuronWeights n
            is = neuronInputs n
            b  = neuronBias n
    
    neuronOutput:: Neuron -> Double
    neuronOutput = 1 / (1 + exp (-z))
               where z =logit n
    
    ewrtws :: TrainingCase -> Neuron -> [Double]
    ewrtws tc n = undefined
    

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s