What are Embeddings in Machine Learning?

Q: How are embeddings created?

Engineers use neural networks to create embeddings. Neural networks consist of hidden neuron layers that make complex decisions iteratively. When creating embeddings, one of the hidden layers learns how to factorize input features into vectors. This occurs before feature processing layers. This process is supervised and guided by engineers with the following steps: Engineers feed the neural network with some vectorized samples prepared manually. The neural network learns from the patterns discovered in the sample and uses the knowledge to make accurate predictions from unseen data. Occasionally, engineers may need to fine-tune the model to ensure it distributes input features into the appropriate dimensional space. Over time, the embeddings operate independently, allowing the ML models to generate recommendations from the vectorized representations. Engineers continue to monitor the performance of the embedding and fine-tune with new data.

Embeddings are numerical representations of real-world data that machine learning (ML) and artificial intelligence (AI) systems use to understand complex knowledge domains. As an example, computing algorithms understand that the difference between 2 and 3 is 1, indicating a close relationship between 2 and 3 as compared to 2 and 100. However, real-world data includes more complex relationships. For example, a bird-nest and a lion-den are analogous pairs, while day-night are opposite terms. Embeddings convert real-world data into complex mathematical representations that capture inherent properties and relationships between real-world data. The entire process is automated, with AI systems learning embeddings during training and using them as needed to complete new tasks.

Why are embeddings important?

Scatter plot of machine learning terms positioned in a three-dimensional embedding vector space, showing related concepts clustered together.

Embeddings enable deep-learning models to understand real-world data domains more effectively. They simplify how real-world data is represented while retaining the semantic and syntactic relationships. This allows machine learning algorithms to extract and process complex data types and enable innovative AI applications. Here are some ways embeddings help in processing.

Reduce data dimensionality

Data scientists use embeddings to represent high-dimensional data in a low-dimensional space. In data science, the term dimension typically refers to a feature or attribute of the data. Higher-dimensional data in AI refers to datasets with many features or attributes that define each data point. This can mean tens, hundreds, or even thousands of dimensions. For example, an image can be considered high-dimensional data because each pixel color value is a separate dimension.

When presented with high-dimensional data, deep-learning models require more computational power and time to learn, analyze, and infer accurately. Embeddings reduce the number of dimensions by identifying commonalities and patterns between various features. This consequently reduces the computing resources and time required to process raw data.

Train large language models

Embeddings improve data quality when training large language models (LLMs). For example, data scientists use embeddings to clean the training data from irregularities affecting model learning. ML engineers can also repurpose pre-trained models by adding new embeddings for transfer learning, which requires refining the foundational model with new datasets. With embeddings, engineers can fine-tune a model for custom datasets from the real world.

Build innovative applications

Embeddings enable new deep learning and generative artificial intelligence (generative AI) applications. Different embedding techniques applied in neural network architecture allow accurate AI models to be developed, trained, and deployed in various fields and applications. For example:

With image embeddings, engineers can build high-precision computer vision applications for object detection, image recognition, and other visual-related tasks.
With word embeddings, natural language processing software can more accurately understand the context and relationships of words.
Graph embeddings extract and categorize related information from interconnected nodes to support network analysis.
Multimodal embeddings extract contextual information across multiple inputs including text, audio, and video. Multimodal embeddings are shared vector representations that incorporate multiple data types within the same embedding space. Modern AI applications use multimodal embeddings to search, compare, and retrieve information more efficiently.

Computer vision models, AI chatbots, large language models, and AI recommender systems all use embeddings to complete complex tasks that mimic human intelligence.

Enable retrieval augmented generation (RAG)

Embeddings allow software developers to build a retrieval augmented generation (RAG) system that integrates generative AI with an internal knowledge base. For example, when you ask a question in a semantic search system, it searches for document embeddings stored in its vector database. The system uses a reranker to score and order the top candidates before generating its answers. A reranker is a stage two scoring model in RAG that ML applications use to improve their response accuracy.

What are vectors in embeddings?

Diagram showing source documents converted into text tokens such as cup of milk and pinch of salt, then transformed into numerical embedding vectors.

Machine learning models cannot interpret information intelligibly in their raw format and require numerical data as input. They use neural network embeddings to convert real-world information into numerical representations called vectors. Vectors are numerical values that represent information in a multi-dimensional space. They help ML models to find similarities among sparsely distributed items.

Every object an ML model learns from has various characteristics or features. As a simple example, consider the following movies and TV shows. Each is characterized by the genre, type, and release year.

The Conference (Horror, 2023, Movie)
Upload (Comedy, 2023, TV Show, Season 3, Episode 5)
Tales from the Crypt (Horror, 1989, TV Show, Season 7, Episode 7)
Dream Scenario (Horror-Comedy, 2023, Movie)

ML models can interpret numerical variables like years, but cannot compare non-numerical ones like genre, types, episodes, and total seasons. The embedding layer encodes non-numerical data into a series of values that ML models can understand and relate. For example, the following is a hypothetical representation of the TV programs listed earlier.

The Conference (1.2, 2023, 20.0)
Upload (2.3, 2023, 35.5)
Tales from the Crypt (1.2, 1989, 36.7)
Dream Scenario (1.8, 2023, 20.0)

The first number in the vector corresponds to a specific genre. An ML model would find that The Conference and Tales from the Crypt share the same genre. Likewise, the model will find more relationships between Upload and Tales from the Crypt based on the third number, which represents the format, seasons, and episodes. As more variables are introduced, you can refine the model to condense more information in a smaller vector space.

How do embeddings work?

Embeddings convert raw data into continuous values that machine learning models can interpret. Conventionally, ML models use one-hot encoding to map categorical variables into forms they can learn from, using discrete values. The encoding method divides each category into rows and columns and assigns them binary values. Consider the following categories of produce and their price.

Fruits	Price
Apple	5.00
Orange	7.00
Carrot	10.00

Representing the values with one-hot encoding results in the following table.

Apple	Orange	Carrot	Price
1	0	0	5.00
0	1	0	7.00
0	0	1	10.00

The table is represented mathematically as numerical vectors [1,0,0,5.00], [0,1,0,7.00], and [0,0,1,10.00].

One-hot encoding expands the dimensional values of 0 and 1 without providing information that helps models relate the different objects. For example, the model cannot find similarities between an apple and an orange, despite being fruits, nor can it differentiate between an orange and a carrot as fruits or vegetables. As more categories are added to the list, the encoding results in sparsely distributed variables with many empty values that consume enormous memory space.

Embeddings vectorize objects into a low-dimensional space by representing similarities between objects with numerical values. Neural network embeddings ensure that the number of dimensions remains manageable with expanding input features. Input features are traits of specific objects that an ML algorithm is tasked to analyze. Dimensionality reduction allows embeddings to retain information that ML models use to find similarities and differences from input data. Modern embedding methods, such as Matryoshka and Binary Embeddings, further reduce the vector’s dimension while maintaining accuracy.

What are embedding models?

Embedding models are algorithms trained to encapsulate information into dense vector representations in a multi-dimensional space. Data scientists use embedding models to enable ML models to process and analyze high-dimensional data. These are common embedding models used in ML applications.

Principal component analysis

Principal component analysis (PCA) is a dimensionality-reduction technique that reduces complex data types into low-dimensional vectors. It finds data points with similarities and compresses them into embedding vectors that reflect the original data. While PCA allows models to process raw data more efficiently, information loss may occur during processing.

Singular value decomposition

Singular value decomposition (SVD) is an embedding model, or matrix factorization technique, that transforms a matrix into its three singular matrices. The resulting matrices retain the original information while allowing models to better comprehend the semantic relationships of the data they represent. Data scientists use SVD to enable various ML tasks, including image compression, text classification, and recommendation.

Word2Vec

Word2Vec is an ML algorithm trained to associate words and represent them in a continuous vector space. Data scientists feed the Word2Vec model with massive textual datasets to enable natural language understanding. The model finds similarities in words by considering their context and semantic relationships.

There are two variants of Word2Vec—Continuous Bag of Words (CBOW) and Skip-gram. CBOW allows the model to predict a word from the given context, while Skip-gram derives the context from a given word. While Word2Vec is an effective word embedding technique, it cannot accurately distinguish contextual differences of the same word used to imply different meanings.

BERT

BERT is a transformer-based language model trained with massive datasets to understand languages. Like Word2Vec, BERT can create word embeddings from input data it was trained with. Additionally, BERT generates contextual embeddings that differentiate the meanings of words when applied to different phrases. For example, BERT creates different embeddings for ‘play’ as in “I went to a play” and “I like to play.”

How are embeddings created?

Engineers use neural networks to create embeddings. Neural networks consist of hidden neuron layers that make complex decisions iteratively. When creating embeddings, one of the hidden layers learns how to factorize input features into vectors. This occurs before feature processing layers. This process is supervised and guided by engineers with the following steps:

Engineers feed the neural network with some vectorized samples prepared manually.
The neural network learns from the patterns discovered in the sample and uses the knowledge to make accurate predictions from unseen data.
Occasionally, engineers may need to fine-tune the model to ensure it distributes input features into the appropriate dimensional space.
Over time, the embeddings operate independently, allowing the ML models to generate recommendations from the vectorized representations.
Engineers continue to monitor the performance of the embedding and fine-tune with new data.

How can AWS help with your embedding requirements?

AWS offers a range of services to help build, train, and deploy your AI models for custom business applications, including with embeddings:

Amazon Bedrock is a fully-managed service that offers a choice of high-performing foundation models (FMs) from leading AI companies, along with a broad set of features to build generative artificial intelligence (generative AI) applications.

Amazon Bedrock’s Titan Multimodal Embeddings G1 is a managed, multimodal embedding neural network that processes text and image data. Titan Embeddings translates text into a numerical representation. The Titan Embeddings model supports text retrieval, semantic similarity, and clustering. The maximum input text is 8K tokens, and the maximum output vector length is 1536.

Amazon Nova is a portfolio of AI offerings—including Nova models, fast and cost-effective foundation models for text and multimodal needs; Nova Forge, a new service to build your own frontier models; and Nova Act, a new service to build agents that automate browser-based UI workflows.

Amazon SageMaker can also be used to create embeddings. Amazon SageMaker is a hub wherein you can build, train, and deploy ML models in a secure and scalable environment.

Get started with embeddings on AWS by creating a free account today.

What are Embeddings in Machine Learning?