Analysis of Recurrent Neural Network RNN ​​Algorithm of Artificial Intelligence Machine Learning

Foreword: For the content of artificial intelligence machine learning algorithms, please refer to the previous related articles on the public account "Technology Optimizes Life". There are three main categories of artificial intelligence machine learning: 1) classification; 2) regression; 3) clustering. Today we will focus on the recurrent neural network (RNN) algorithm.

Recurrent Neural Network (RNN) has now become one of the important research objects of neural network experts in the world. It is an artificial neural network in which nodes are directionally connected into a ring. It was originally developed by Jordan and Pineda. Williams, Elman is equivalent to a neural network structure model proposed in the late 1980s.

Juergen Schmidhuber and his students led the research on RNNs of recurrent neural networks, including SeppHochreiter, who discovered the problem of vanishing gradients encountered by high-depth networks, and later invented the long and short-term memory (LSTM) recurrent network; and Alex Graves, he currently works at DeepMind. Two other well-known researchers are Felix Gers, who invented the LSTM forget gate; Justin Bayer, who invented a method that allows the LSTM topology to automatically evolve according to specific problems.

The internal state of Recurrent NeuronNetwork (RNN) can show dynamic timing behavior. Different from the feedforward neural network, RNN can use its internal memory to process arbitrary timing input sequences, which makes it easier to process such as unsegmented handwriting recognition and speech recognition.

Recurrent Neural Networks (RNN) can realize parallel and sequential calculations, and in principle can calculate anything that can be calculated by traditional computers. But unlike traditional computers, RNN has similarities with the human brain; the human brain is a large feed-forward network composed of neurons connected to each other, which can learn to send sensory signals that last a lifetime in a certain way. The input stream is converted into a valid motion output sequence. The human brain is an outstanding model, because it can solve many problems that computers are not capable of.

The essential feature of the recurrent neural network RNN ​​is that there are both internal feedback connections and feedforward connections between processing units. From the system point of view, it is a feedback dynamic system, which reflects the dynamic characteristics of the process in the calculation process, and has stronger dynamic behavior and computing capabilities than the feedforward neural network.

1) Fully recurrent network

2) Hopfield network (Hopfield network)

3) Elman networks and Jordannetworks

4) Echo state network

5) Long short term memery network

6) Two-way network (Bi-direcTIonal RNN)

7) Continuous network (ConTInuous－TIme RNN)

8) Hierarchical RNN (Hierarchical RNN)

9) Recurrent mulTIlayer perceptron

10) Second Order Recurrent Neural Network (Second Order Recurrent Neural Network)

11) Pollack's sequential cascaded networks (Pollack's sequential cascaded networks)

The biggest difference between the recurrent neural network RNN ​​and the feedforward neural network FNN is that there is a feedback loop. In addition, the memory modes of the two are completely different. After finalizing, the feedforward neural network will generate a static model of the learned data, and then accept new examples for accurate classification or clustering. The recurrent neural network RNN ​​will generate a dynamic model, that is, a model that will change over time, that is, the output of step n-1 is fed back to the neural network, which affects the output of step n, and every subsequent step is also And so on. This change allows the model to perform accurate classification based on the context of the input sample.

The recurrent neural network RNN ​​contains the hidden state that determines the previous classification result in the same sequence. In each subsequent step, this hidden state will be added to the input data of the new step to generate a new hidden state, and then a new classification result will be obtained. Each hidden state will be recycled to generate adjusted subsequent hidden states.

The human memory model can also consider the context and recycle the cognition of the past state to properly interpret new data. Humans are affected by short-term memory and previous feelings, and retain different "hidden states."

Basic structure of RNN:

RNN is a neural network that models sequence data, that is, the current output of a sequence is also related to the previous output. The specific form of expression is that the network will memorize the previous information and apply it to the calculation of the current output, that is, the nodes between the hidden layers are no longer unconnected but connected, and the input of the hidden layer not only includes the output of the input layer It also includes the output of the hidden layer at the previous moment. The following is an example diagram of an RNN model, where:

st is the hidden state (memory) at time t, based on the hidden state at the previous time and the current input: st = f (Uxt + Wst? 1), where f is generally a nonlinear activation function. When calculating s0, you need to use s ? 1.

ot represents the output at time t, ot=softmax (Vst);

In RNN, all layers share the same parameters. It reflects that each step in the RNN is doing the same thing, but the input is different, so the parameters that need to be learned in the network are greatly reduced.

RNN allows operations on the sequence of vectors: the input can be a sequence, and the output can also be a sequence. In the most generalized case, both input and output can be sequences. The output sequence is understood as the degree of confidence that the RNN predicts about the next state of the sequence.