Axons of presynaptic neurons (input X) connect to branches of dendrites (horizontal rectangles) by synaptic layers; the membrane layer (vertical rectangles) sums the dendritic activations, and transfers the sum to the soma body (black sphere). Synaptic layers have four different connection cases, namely, the direct connection, the inverse connection, the constant 0 connection and the constant 1 connection. Credit: Kanazawa University Artificial neural networks are machine learning systems composed of a large number of connected nodes called artificial neurons. Similar to the neurons in a biological brain, these artificial neurons are the primary basic units that are used to perform neural computations and solve problems. Advances in neurobiology have illustrated the important role played by dendritic cell structures in neural computation, and this has led to the development of artificial neuron models based on these structures.
The recently developed approximate logic neuron model (ALNM) is a single neural model that has a dynamic dendritic structure. The ALNM can use a neural pruning function to eliminate unnecessary dendrite branches and synapses during training to address a specific problem. The resulting simplified model can then be implemented in the form of a hardware logic circuit.
However, the well-known backpropagation (BP) algorithm that was used to train the ALMN actually restricted the neuron model's computational capacity. "The BP algorithm was sensitive to initial values and could easily be trapped into local minima," says corresponding author Yuki Todo of Kanazawa University's Faculty of Electrical and Computer Engineering. "We therefore evaluated the capabilities of several heuristic optimization methods for training of the ALMN."
After a series of experiments, the states of matter search (SMS) algorithm was selected as the most appropriate training method for the ALMN. Six benchmark classification problems were then used to evaluate the ALNM's optimization performance when it was trained using the SMS as a learning algorithm, and the results showed that SMS provided superior training performance when compared with BP and the other heuristic algorithms in terms of both accuracy and convergence speed.
Evolution process of states of matter search (SMS) are based on the physical principle of the thermal-energy motion ratio. The whole optimization process is divided into the following three phases: the gas state (50 percent), the liquid state (40 percent) and the solid state (10 percent). Each state has its own operations with different exploration-exploitation ratios. The gas state is a pure exploration at the beginning of the optimization process. The liquid state simultaneously possesses the exploration and exploitation searching, and the solid state focuses on the exploitation simply at the latter part of the optimization process. The algorithm optimized in this way can achieve a suitable balance between exploration and exploitation. Credit: Kanazawa University "A classifier based on the ALNM and SMS was also compared with several other popular classification methods," states Associate Professor Todo, "and the statistical results verified this classifier's superiority on these benchmark problems."
During the training process, the ALNM simplified the neural models through synaptic pruning and dendritic pruning procedures, and the simplified structures were then substituted using logic circuits. These circuits also provided satisfactory classification accuracy for each of the benchmark problems. The ease of hardware implementation of these logic circuits suggests that future research will see the ALNM and SMS used to solve increasingly complex and high-dimensional real-world problems.
ALNM is capable of simplifying the neural models by synaptic pruning and dendritic pruning during the training process. Then, the simplified structures of ALNM can be substituted by logical circuits, which merely consists of the 'comparators' and logic NOT, AND and OR gates. Among them, the 'comparator' works as an analog-to-digital converter which compares the input with the threshold θ. If the input X exceeds the threshold θ, the 'comparator' will output 1. Otherwise, it will output 0. When implemented on hardware, these logical circuits can be adopted as efficient classifiers to solve the six benchmark problems. Credit: Kanazawa University
Explore further: Scientists improve deep learning method for neural networks
More information: Junkai Ji et al, Approximate logic neuron model trained by states of matter search algorithm, Knowledge-Based Systems (2018). DOI: 10.1016/j.knosys.2018.08.020