What Is an Autoencoder?

An autoencoder is a deep-learning algorithm that converts complex, real-world data into a simplified mathematical representation for further processing by artificial intelligence (AI) systems. Real-world data like text and images have many attributes or dimensions that define them. For example, a word in a sentence has attributes like spelling, grammar, and context that require mathematical symbolization (or vector representation) before software systems can process them. Autoencoders convert such data to and from vector representations. They are an integral part of generative AI systems and are also used independently for various data processing tasks.

Autoencoders allow engineers to overcome several challenges in deep learning and data processing applications.

Data denoising

Noise, or unwanted data, degrades the integrity of a video, image, or audio file. An autoencoder detects and removes noise from the original data when trained with specific datasets. For example, you can train an autoencoder model with images of cars. Then, it becomes exceptionally good at recognizing cars in images and disregards blurring, shadows, and other noise at the pixel level.

Dimensionality reduction

Autoencoders help data scientists simplify machine-learning development by reducing input data into a more compact representation. For example, you can use a high-definition image to train a regular machine-learning model to recognize faces. Still, minor changes in the input might significantly affect the model's performance. With an image-denoising autoencoder, you can reduce the image's resolution while maintaining critical information. Moreover, the model is less susceptible to subtle pixel variations.

Anomaly detection

Autoencoders can detect anomalies in data by comparing input data against its compressed representation. When an autoencoder trained to process cat images is presented with an image of an elephant, the model reports a substantial reconstruction error. A reconstruction error indicates that an autoencoder struggles to rebuild the original image despite being trained to do so. When this happens, the model might have detected an outlier among the input data.

Feature clustering

An autoencoder is designed to remove unnecessary information as the original data passes through subsequent layers of neurons. It reduces the data's dimensionality to a few subsets. Engineers can easily segregate these subsets into several classes to group similar features together. We call this feature clustering, which helps machine learning teams train AI models.

An autoencoder is a neural network that compresses and remembers knowledge learned from the original data. Unlike other neural networks, the autoencoder doesn't have an equal distribution of neurons in its hidden layers. Instead, the autoencoder is designed with fewer neurons in the deeper layer to enable sufficient knowledge compression. You can picture an autoencoder as two gradually narrowing funnels that meet in the middle.

Components

Autoencoders consist of an encoder, code, and decoder.

• An encoder is a neural network spanning several hidden layers that compresses the original data into a reduced representation.
• The code is the autoencoder's compressed bottleneck region and the intersecting point between the encoder and decoder. It is the latent space that holds the encoded representation of the input in a lower dimension.
• A decoder is an expanding neural network that reconstructs image data from the compressed version to its original dimension.

All autoencoders require training before they can compress and reconstruct image data to its original form. When training an autoencoder, engineers feed the algorithm with an unlabeled dataset and evaluate the reconstructed output.

As data traverses through the autoencoder, information loss is expected. Engineers measure the reconstruction loss—a value that indicates the discrepancy between the original and output data. Then, they continuously adjust the model until it can consistently reconstruct the real-world training data. To do that, the model must be adequately sensitive to the input changes but not to the extent of overfitting. Overfitting is a condition where the model performs well in training but not with new data.

Example

Autoencoders work well on data that has interdependent features, such as images. Consider a functional autoencoder with a training dataset of faces. After sufficient training and readjustment, the autoencoder can take the image of any face (outside its training dataset) and recreate it. It can be used to complete partially missing, hidden, or blurry faces. However, it gives an error if given the input image of an animal’s face.

We share several variations of autoencoders that allow engineers to reconstruct compressed image data differently.

Undercomplete autoencoder

The undercomplete autoencoder is the simplest form of the autoencoder model. It compresses the input image into the latent space and reconstructs the encoded representation into its original dimension. Without using additional conditions, the model relies solely on the reconstruction error to predict the same image as the input.

Convolutional autoencoder

A convolutional autoencoder uses layers of convolutional neural networks (CNNs) in the encoder and decoder. CNNs are a deep-learning algorithm capable of extracting and preserving spatial information. When added to an autoencoder, the CNN layers reduce spatial and temporal losses when processing image data. The model gently extracts the image features by passing the data through convolutional layers. Convolutional autoencoders retain the relationships between pixels as they are while reducing data dimensionality. They are ideal for image recognition, classification, and generation applications.

Sparse autoencoders

Sparse autoencoders have a different architecture than other autoencoder variants but share similar goals. Rather than compressing data through hidden layers of decreasing interconnected neurons, the sparse autoencoder maintains a linear node count throughout the entire neural network. To compress input data into the latent space, it selectively activates specific neurons to encode or decode data that passes through. This allows the sparse autoencoder to vary the latent space formed, corresponding to the type of data it processes. Because of its unique architecture, the sparse autoencoder applies reconstruction loss and sparsity functions to regulate the number of neurons it activates.

Denoising autoencoders

Image-denoising autoencoders are architected similarly to a basic autoencoder but specifically trained to remove noise from image data. Rather than passing an original image, engineers use a slightly corrupted image to train the model. The autoencoder tries to reconstruct a noise-free version of the image and compare the output with the original data. As the model doesn't learn from the original image, it cannot reconstruct the output from the latent space representation or learned manifold. Instead, it gradually adjusts the learned manifold until the output matches the original data. Then, the denoising autoencoder applies the reconstruction method learned on unfamiliar data in real-world applications.

Contractive autoencoders

Contractive autoencoders allow the encoding neuron to extract key features without being affected by variations in the input image. Unlike a denoising autoencoder, the contractive autoencoder desensitizes itself against noises. When training the model, engineers add a penalty function to the reconstruction loss so that the model ignores small changes in the original images. Eventually, the contractive autoencoder learns not to map minor changes in the input images to its latent space.

Variational autoencoders

Variational autoencoders were designed to overcome the limitations of other autoencoders. Usually, autoencoders map image features to a single representative value in the latent space, which the decoder uses to reconstruct new and realistic data. For example, a red apple is represented by a value of 1, while a green apple is 2 in the vector space. When generating a unique image from the learned vector, the model might randomly choose a value, 5, which it decodes as a blue apple. As we know, a blue apple doesn't exist. Engineers might train the autoencoder with more images to improve accuracy and likeness, but this leads to overfitting when the model overlearns similar patterns.

The above example is a simplified explanation of the limits of autoencoders, which variational autoencoders overcome. Instead of representing the dimensional space with numbers, a variational encoder uses probability distribution. It applies a bell-curve distribution to the feature values in a continuous latent space. When decoding from the learned representation, the decoder samples a value along the latent vector distribution. This allows the autoencoder to generate an accurate and realistic image that looks similar to the original.

Generative AI allows users to generate unique content such as text, images, audio, and video. Autoencoders, specifically variational autoencoders, offer similar capabilities. The decoder block of a variational autoencoder is a generative model. It became an important component of subsequent deep learning models, such as the generative pre-trained transformer (GPT). The GPT architecture features a decoder block, allowing the model to sample from the probabilistic distribution of encoded representation and generate new and realistic content.

While transformer-based models lead to revolutionary developments in natural language processing and text generation, the variational autoencoder remains relevant in image generation models and applications.

AWS provides purpose-built managed services to help organizations build, train, deploy, and scale their generative AI applications. With enterprise-grade security, you can integrate applications with industry-leading foundational models to provide a personalized customer experience and accelerate AI strategies.

• Amazon Bedrock allows you to build generative AI applications easily with top-performing deep learning models like Stable Diffusion, Llama, and Amazon Titan.
• Use Amazon Sagemaker Hyperpod to reduce training time by splitting foundational model training across thousands of machine-learning accelerators.
• Deploy your generative AI workloads on AWS Trainium and AWS Inferentia to maximize performance at an affordable cost.

Get started with autoencoders and generative AI by signing up for an AWS account today.