Neural Networks

David Wallace Croft
croft@alumni.caltech.edu

2005-04-15


I have relocated this webpage to http://www.orbs.com/david/research/neural/. All updates will be made at the new website.

Frustrated Synapse Learning

In the Spring of 1993, I invented a neural network learning algorithm which I called Frustrated Synapse learning. My novel idea was to combine Hebbian learning with an additional rule that would make a synaptic weight more negative if it were triggered during the hyperpolarization of the target neuron. Through a number of computer simulation experiments that Summer, I observed that the ability of the algorithm to overcome the stability-plasticity dilemma and to stabilize fully recurrent networks.

In the Fall of 1993, I entered Caltech as a graduate student and threw myself into my simulations and course work in an effort to determine whether this was the learning rule used by real-life biological neurons. During that time, I was able to discuss the learning rule with many of my professors and classmates and increase the sophistication of my models, as demonstrated in some of my notes, student papers, and presentation slides from that period as listed below. When I left graduate school to enter industry in 1995, I was even more convinced that this learning algorithm was biologically plausible.

Upon returning to the field in 2003, I was pleasantly surprised to learn that this learning rule was demonstrated to exist in biological neurons through experiments performed by neuroscientists in 1997. What I have previously labeled Frustrated Synapse, Phase Covariance, or Hebbian Phase learning is now known as antisymmetric Hebbian or spike timing dependent synaptic plasticity (STDP).

Synchronicity and Periodicity Research

Other Neural Network Papers

  • 1994-03-19: Representational Systems
  • 1994-02-09: Could a Computer Feel Pain?
  • 1994-01-12: A Neural Network Primer
  • Neural Projects

    • 1993 - 1997
      Theoretical analysis of biologically realistic neuron model with dendritic tree using the simulation tool Neuron. B. Mel, E. Niebur, and D. Croft, "Why Neurons Make Bad Coincidence Detectors but Good Periodicity Detectors", presented at the 1995 Neurosciences Meeting.

    • 1993 - 1997
      Research of the neural network Phase Correlation learning algorithm, novel activation and learning rules for spiking neurons.

    • 1996
      Implementation of neural network and genetic algorithm simulations in the Java programming language.
      http://www.croftsoft.com/portfolio/

    • 1995 Jun - 1996 Jul
      Systems Engineer
      Tanner Research Inc., Pasadena, CA
      http://www.tanner.com/

      • Design and implementation of parameterizable VLSI layout language software code in C for the automated generation of digital neural network and subthreshold analog VLSI neuromorphic circuits as part of the Neural Network Silicon Compiler research contract. Demonstrated at the 1996 NSF Workshop on Neuromorphic Engineering.
      • Design and fabrication of scalable, programmable, stochastic pulse CMOS VLSI Digital Neural Network Architecture (DNNA) circuitry.
      • Laboratory testing of analog and digital CMOS VLSI chips for speech processing and neural network applications.
      • Documentation of reusable VLSI circuit layout language code components and cell libraries in HTML.
      • Wrote the "Fuzzy Logic Silicon Compiler" government research proposal, identifying low-power analog circuits to be used for Fuzzy Logic processing.
      • Experience with the full suite of EDA tools for VLSI design including schematic editors, layout editing, and simulators in the process of carrying circuit designs from concepts to the test bench.

    • 1995
      With lab partners, injected mRNA for neural channels into frog oocytes and later observed neural spiking when current was injected.

    • 1994 - 1995
      Design, fabrication, and testing of a novel analog VLSI depolarizing-hyperpolarizing neuron with an analog synapse adapted using an integrated learning algorithm with floating gate tunneling and injection. Presented in a talk at the kickoff for the NSF Center for Neuromorphic Systems Engineering at the California Institute of Technology.

    • 1994
      Developed simulation software in MatLab for the implementation of a spatiotemporal filter for accurate velocity estimation over a range of spatial frequencies using passive and integrate-and-fire neuron models.

    • 1990
      Design, implementation, and demonstration of the neural network ART-1 learning algorithm for pattern recognition. System included photodiode input, digital to analog conversion, RS-232 serial I/O circuitry, serial I/O software, software implementation of the ART-1 learning algorithm, and graphical output.

    Demos

    • Backprop_XOR: a neural network simulation of backpropagation error learning
    • Dice: click the mouse and watch a simple neural net learn whether to flee or fight
    • Insight: goblins hunt kobolds in the dark using a neural network

    Links

    Books

    • Software Agents and Soft Computing: Towards Enhancing Machine Intelligence
      1997; Hyacinth S. Nwana and Nader Azermi (Eds.)
    • Intelligent Java Applications for the Internet and Intranets
      1997; Mark Watson
    • Advances in Knowledge Discovery and Data Mining
      1996; Fayyad, Piatetsky-Shapiro, Smyth, and Uthurusamy (Eds.)
    • Data Mining with Neural Networks: Solving Business Problems -- from Application Development to Decision Support
      1996; Joseph P. Bigus
    • An Introduction to Natural Computation
      1996 June (draft); Dana H. Ballard
    • C++ Neural Networks & Fuzzy Logic, 2nd Edition
      1995; Valluru Rao and Hayagriva Rao
    • Neural Networks for Pattern Recognition
      1995; Christopher M. Bishop
    • Neural Networks: A Comprehensive Foundation
      1994; Simon Haykin
    • Analog VLSI: Signal and Information Processing
      1994; Mohammed Ismail and Terri Fiez
    • Biophysics of Computing: Information Processing in Single Neurons
      1994 September 08 (draft); Christof Koch
    • Advanced Methods in Neural Computing
      1993; Philip D. Wasserman
    • The Book of Genesis: Exploring Realistic Neural Models with the GEneral NEural SImulation System
      1993; James M. Bower and David Beeman
    • From Neuron to Brain, 3rd Edition
      1992; Nicholls, Martin, and Wallace
    • Frontiers in Cognitive Neuroscience
      1992; Stephen M. Kosslyn and Richard A. Andersen (Eds.)
    • Principles of Neural Science, 3rd Edition
      1991; Kandel, Schwartz, and Jessell
    • Introduction to the Theory of Neural Computation
      1991; Hertz, Krough, and Palmer
    • Visual Perception: The Neurophysiological Foundations
      1990; Lothar Spillmann and John S. Werner (Eds.)
    • The Synaptic Organization of the Brain, 3rd Edition
      1990; Gordon M. Shepherd (Ed.)
    • Mind and Cognition: A Reader
      1990: William G. Lycan (Ed.)
    • The Representational Theory of Mind: An Introduction
      1990: Kim Sterelny
    • Methods in Neuronal Modeling: From Synapses to Networks
      1989; Christof Koch and Idan Segev (Eds.)
    • Analog VLSI and Neural Systems
      1989; Carver Mead
    • Neural Computing: Theory and Practice
      1989; Philip D. Wasserman
    • Eye, Brain, and Vision
      1988; David H. Hubel
    • Robot Vision
      1986; Berthold Klaus Paul Horn
    • An Introduction to the Mathematics of Neurons
      1986; F. C. Hoppensteadt
    • The Human Brain Coloring Book
      1985; Diamond, Scheibel, and Elson


    Back

    http://www.alumni.caltech.edu/~croft/research/neural/
    David Wallace Croft, croft@alumni.caltech.edu
    First posted 1997-10-10.