What is a Recurrent Neural Network (RNN)?

A recurrent neural network (RNN) is a learning model that is trained to process and convert a sequential data input into a specific sequential data output. Sequential data is data where sequential components interrelate based on complex semantics and syntax rules—for example, sentences or time-series data. An RNN is a software system composed of many interconnected components that mimic how humans process sequential data, such as when translating text from one language to another.

RNNs are mainly legacy architecture, replaced by transformer-based artificial intelligence (AI) and large language models (LLMs), which perform better in processing sequential data.

• Read about neural networks
• Read about deep learning
• Read about transformers in artificial intelligence
• Read about large language models

Here are some common RNN types.

One-to-many

This RNN type channels a single input to multiple outputs. It enables linguistic applications such as image captioning by generating a sentence from a single image.

While RNNs can be configured in a one-to-one architecture, where one input sequence is associated with one output, this is better handled by a feedforward neural network.

Many-to-many

The model uses multiple inputs to predict multiple outputs. For example, you can create machine translation or speech recognition agents with an RNN, which analyzes a sentence and correctly structures the words in a different language.

Many-to-one

Several inputs are mapped to an output. This is helpful in applications such as sentiment analysis, where the model predicts customers’ sentiments such as positive, negative, and neutral, from input testimonials.

RNNs are one of several different neural network architectures.

Recurrent neural network vs. feed-forward neural network

Like RNNs, feed-forward neural networks are artificial neural networks that pass information from one end to the other. Feedforward networks can perform classification, regression, or recognition tasks, but it can’t remember the previous input that it has processed. For example, it forgets Apple by the time its neuron processes the word is. The RNN overcomes this memory limitation by including a hidden memory state to retain previous inputs.

Recurrent neural network vs. convolutional neural networks

Convolutional neural networks are artificial neural networks designed to process spatial data by modeling the functions of the human eye. You can use convolutional neural networks to extract spatial information from videos and images by passing them through a series of convolutional and pooling layers in the neural network that can detect elements such as edges and patterns. A fully connected layer then classifies the object or objects in the image.

RNN architecture has laid the foundation for broader ML models to support language processing. Several RNN variants have emerged that share its core principles of memory retention but improve its functionality in different ways.

Stacked RNNs

A stacked RNN is built by layering multiple recurrent neural networks on top of one another. Each layer functions as an independent unit. The output produced by each layer becomes the input for the next one. This “stacking” allows the combined network to learn and predict more complex sequences.

Bidirectional recurrent neural networks

A bidirectional recurrent neural network (BRNN) processes data using both forward and backward layers of hidden nodes. The forward layer operates just like a standard RNN. It starts at the beginning of the sequence, stores inputs in the hidden state, and uses them to predict the next output as it moves forward.

The backward layer performs the same operation, but from the end of the data sequence. It uses both the current input and the future inputs to update the current hidden state. By combining both directional layers, the BRNN can consider both past and future contexts to improve prediction accuracy.

For example, consider the word “lead” in these two sentences:

• “The lead singer performed well.”
• “The lead pipe was heavy.”

A standard RNN moving forward only would not know how to classify “lead” because it hasn’t seen the rest of the sentence yet.

In a BRNN, the forward layer processes “The” and has learned the pattern that a subject is coming next. Simultaneously, the backward layer reads from the end of the sentence and works its way backward through the sequence. In the first sentence, it sees “singer” and “performed,” signaling that “lead” likely refers to a person. In the second, it sees “pipe” and “heavy,” signaling that “lead” likely refers to a metal. By combining these insights, moving both backward and forward, the BRNN can more accurately predict the correct meaning of contextual input.

Long short-term memory

Long short-term memory (LSTM) is an RNN variant that enables a model to expand its memory capacity to accommodate longer horizons. A basic RNN only remembers the immediate past input. It can’t use inputs from several earlier time steps to improve its prediction.

Consider the following sentences: “Tom is a cat. Tom’s favorite food is fish.” When you’re using an RNN, it might generate various foods, like “cake” or “pizza,” while attempting to predict the last word. A very basic RNN may not remember that Tom is a cat.

LSTM networks add special memory blocks called cells to the hidden layer. Cell state is the longterm memory for the sequence. Each cell is controlled by an input gate, output gate, and forget gate, which enables the layer to remember helpful information. For example, the cell remembers the words “Tom” and “cat” enabling the model to better predict the word “fish.”

Gated recurrent units

A Gated Recurrent Unit (GRU) is a type of RNN designed for selective memory retention. They were developed as a simplified version of LSTMs. The model incorporates update and reset gates into its hidden layer. These gates control the flow of information, determining what to store in or remove from memory. GRUs do not include an output gate.

ML engineers have made significant progress in natural language processing (NLP) applications since the original development of RNNs and their variants. However, the RNN model holds several limitations.

Read about natural language processing

Exploding gradient

An RNN often incorrectly predicts outputs during initial training. You need to perform several iterations of backpropagation in order to adjust the model’s parameters to reduce the error rate. Remember that the gradient of each weight shows how the weight affects the loss function.

An exploding gradient occurs when gradients multiply and grow exponentially during backpropagation, causing the RNN to become unstable. The model then updates its weights too much, too fast. The training process then diverges too much and fails to find a solution.

Vanishing gradient

The vanishing gradient problem is a condition where a model’s gradients approach zero in backpropagation. When this happens, the RNN fails to learn effectively from the training data and can’t perform well in real-life applications because its weights are never adjusted appropriately.

RNNs are most at risk of vanishing and exploding gradient issues when they process long data sequences.

Slow training time

An RNN processes data sequentially, which limits its ability to process large volumes of text efficiently. As each step is dependent on the step before, RNNs are not capable of parallel processing for a sequence. For example, an RNN model can analyze a buyer’s sentiment from a few sentences. However, it requires a far longer amount of time to summarize a page of an essay.

Transformers are deep learning models that use self-attention mechanisms in a neural network architecture. They can perform sequential data processing, like RNNs, but with processing in parallel.

Self-attention

Instead of relying on hidden states to store past information, as RNNs do, transformers use a mechanism called self-attention. This technique allows the model to look at every word in a sequence simultaneously and compute how strongly each word relates to the others. By analyzing these relationships all at once, the model can more effectively understand contextual information even in very long input sequences.

Parallelism

Transformers solve the gradient issues that RNNs face by enabling parallelism during training. By processing all positions in the sequence simultaneously, transformers avoid vanishing gradients, as the gradients can flow freely to all positions without passing through other steps. They are also optimized for parallel computing infrastructure, which graphics processing units (GPUs) can provide. Parallelism enables transformers to scale massively and handle complex NLP tasks by building larger models.

Generative AI on Amazon Web Services (AWS) provides services, tools, and resources that you can use to build, manage, and scale traditional AI applications with advanced transformer-based technology. For example:

Amazon Bedrock simplifies generative AI development by enabling the customization and deployment of industry-leading foundation models (FM) securely and efficiently.

Amazon Sagemaker delivers an integrated experience for analytics and AI with unified access to all your data. Collaborate and build faster from a unified studio using familiar AWS tools for model development, generative AI, data processing, and SQL analytics. Access all your data whether it’s stored in data lakes, data warehouses, or third-party or federated data sources, with governance built in to meet enterprise security needs.

AWS Trainium is an ML accelerator that you can use to train and scale deep learning models affordably in the cloud.

Get started with generative AI on AWS by signing up for a free account today.