Build taxonomy-based contextual targeting using AWS Media Intelligence and Hugging Face BERT

As new data privacy regulations like GDPR (General Data Protection Regulation, 2017) have come into effect, customers are under increased pressure to monetize media assets while abiding by the new rules. Monetizing media while respecting privacy regulations requires the ability to automatically extract granular metadata from assets like text, images, video, and audio files at internet scale. It also requires a scalable way to map media assets to industry taxonomies that facilitate discovery and monetization of content. This use case is particularly significant for the advertising industry as data privacy rules cause a shift from behavioral targeting using third-party cookies.

Third-party cookies help enable personalized ads for web users, and allow advertisers to reach their intended audience. A traditional solution to serve ads without third-party cookies is contextual advertising, which places ads on webpages based on the content published on the pages. However, contextual advertising poses the challenge of extracting context from media assets at scale, and likewise using that context to monetize the assets.

In this post, we discuss how you can build a machine learning (ML) solution that we call Contextual Intelligence Taxonomy Mapper (CITM) to extract context from digital content and map it to standard taxonomies in order to generate value. Although we apply this solution to contextual advertising, you can use it to solve other use cases. For example, education technology companies can use it to map their content to industry taxonomies in order to facilitate adaptive learning that delivers personalized learning experiences based on students’ individual needs.

Solution overview

The solution comprises two components: AWS Media Intelligence (AWS MI) capabilities for context extraction from content on web pages, and CITM for intelligent mapping of content to an industry taxonomy. You can access the solution’s code repository for a detailed view of how we implement its components.

AWS Media Intelligence

AWS MI capabilities enable automatic extraction of metadata that provides contextual understanding of a webpage’s content. You can combine ML techniques like computer vision, speech to text, and natural language processing (NLP) to automatically generate metadata from text, videos, images, and audio files for use in downstream processing. Managed AI services such as Amazon Rekognition, Amazon Transcribe, Amazon Comprehend, and Amazon Textract make these ML techniques accessible using API calls. This eliminates the overhead needed to train and build ML models from scratch. In this post, you see how using Amazon Comprehend and Amazon Rekognition for media intelligence enables metadata extraction at scale.

Contextual Intelligence Taxonomy Mapper

After you extract metadata from media content, you need a way to map that metadata to an industry taxonomy in order to facilitate contextual targeting. To do this, you build Contextual Intelligence Taxonomy Mapper (CITM), which is powered by a BERT sentence transformer from Hugging Face.

The BERT sentence transformer enables CITM to categorize web content with contextually related keywords. For example, it can categorize a web article about healthy living with keywords from the industry taxonomy, such as “Healthy Cooking and Eating,” “Running and Jogging,” and more, based on the text written and the images used within the article. CITM also provides the ability to choose the mapped taxonomy terms to use for your ad bidding process based on your criteria.

The following diagram illustrates the conceptual view of the architecture with CITM.

The IAB (Interactive Advertising Bureau) Content Taxonomy

For this post, we use the IAB Tech Lab’s Content Taxonomy as the industry standard taxonomy for the contextual advertising use case. By design, the IAB taxonomy helps content creators more accurately describe their content, and it provides a common language for all parties in the programmatic advertising process. The use of a common terminology is crucial because the selection of ads for a webpage a user visits has to happen within milliseconds. The IAB taxonomy serves as a standardized way to categorize content from various sources while also being an industry protocol that real-time bidding platforms use for ad selection. It has a hierarchical structure, which provides granularity of taxonomy terms and enhanced context for advertisers.

Solution workflow

The following diagram illustrates the solution workflow.

The steps are as follows:

1. Amazon Simple Storage Service (Amazon S3) stores the IAB content taxonomy and extracted web content.
2. Amazon Comprehend performs topic modeling to extract common themes from the collection of articles.
3. The Amazon Rekognition object label API detects labels in images.
4. CITM maps content to a standard taxonomy.
5. Optionally, you can store content to taxonomy mapping in a metadata store.

In the following sections, we walk through each step in detail.

Amazon S3 stores the IAB content taxonomy and extracted web content

We store extracted text and images from a collection of web articles in an S3 bucket. We also store the IAB content taxonomy. As a first step, we concatenate different tiers on the taxonomy to create combined taxonomy terms. This approach helps maintain the taxonomy’s hierarchical structure when the BERT sentence transformer creates embeddings for each keyword. See the following code:

def prepare_taxonomy(taxonomy_df):
    
    """
    Concatenate IAB Tech Lab content taxonomy tiers and prepare keywords for BERT embedding. 
    Use this function as-is if using the IAB Content Taxonomy
    
    Parameters (input):
    ----------
    taxonomy_df : Content taxonomy dataframe

    Returns (output):
    -------
    df_clean : Content taxonomy with tiers in the taxonomy concatenated
    keyword_list: List of concatenated content taxonomy keywords
    ids: List of ids for the content taxonomy keywords
    """
    
    df = taxonomy_df[['Unique ID ','Parent','Name','Tier 1','Tier 2','Tier 3']] 
    df_str = df.astype({"Unique ID ": 'str', "Parent": 'str', "Tier 1": 'str', "Tier 2": 'str', "Tier 3": 'str'})
    df_clean = df_str.replace('nan','')
    
    #create a column that concatenates all tiers for each taxonomy keyword
    df_clean['combined']=df_clean[df_clean.columns[2:6]].apply(lambda x: ' '.join(x.dropna().astype(str)),axis=1)
    
    #turn taxonomy keyords to list of strings a prep for encoding with BERT sentence transformer
    keyword_list=df_clean['combined'].to_list()
                       
    #get list of taxonomy ids
    ids = df_clean['Unique ID '].to_list()                  
            
    return df_clean, keyword_list, ids

taxonomy_df, taxonomy_terms, taxonomy_ids = prepare_taxonomy(read_taxonomy)

The following diagram illustrates the IAB context taxonomy with combined tiers.

Amazon Comprehend performs topic modeling to extract common themes from the collection of articles

With the Amazon Comprehend topic modeling API, you analyze all the article texts using the Latent Dirichlet Allocation (LDA) model. The model examines each article in the corpus and groups keywords into the same topic based on the context and frequency in which they appear across the entire collection of articles. To ensure the LDA model detects highly coherent topics, you perform a preprocessing step prior to calling the Amazon Comprehend API. You can use the gensim library’s CoherenceModel to determine the optimal number of topics to detect from the collection of articles or text files. See the following code:

def compute_coherence_scores(dictionary, corpus, texts, limit, start=2, step=3):
    """
    Compute coherence scores for various number of topics for your topic model. 
    Adjust the parameters below based on your data

    Parameters (input):
    ----------
    dictionary : Gensim dictionary created earlier from input texts
    corpus : Gensim corpus created earlier from input texts
    texts : List of input texts
    limit : The maximum number of topics to test. Amazon Comprehend can detect up to 100 topics in a collection

    Returns (output):
    -------
    models : List of LDA topic models
    coherence_scores : Coherence values corresponding to the LDA model with respective number of topics
    """
    coherence_scores = []
    models = []
    for num_topics in range(start, limit, step):
        model = gensim.models.LdaMulticore(corpus=corpus, num_topics=num_topics, id2word=id2word)
        models.append(model)
        coherencemodel = CoherenceModel(model=model, texts=corpus_words, dictionary=id2word, coherence='c_v')
        coherence_scores.append(coherencemodel.get_coherence())

    return models, coherence_scores

models, coherence_scores = compute_coherence_scores(dictionary=id2word, corpus=corpus_tdf, texts=corpus_words, start=2, limit=100, step=3)

After you get the optimal number of topics, you use that value for the Amazon Comprehend topic modeling job. Providing different values for the NumberOfTopics parameter in the Amazon Comprehend StartTopicsDetectionJob operation results in a variation in the distribution of keywords placed in each topic group. An optimized value for the NumberOfTopics parameter represents the number of topics that provide the most coherent grouping of keywords with higher contextual relevance. You can store the topic modeling output from Amazon Comprehend in its raw format in Amazon S3.

The Amazon Rekognition object label API detects labels in images

You analyze each image extracted from all webpages using the Amazon Rekognition DetectLabels operation. For each image, the operation provides a JSON response with all labels detected within the image, coupled with a confidence score for each. For our use case, we arbitrarily select a confidence score of 60% or higher as the threshold for object labels to use in the next step. You store object labels in their raw format in Amazon S3. See the following code:

"""
Create a function to extract object labels from a given image using Amazon Rekognition
"""

def get_image_labels(image_loc):
    labels = []
    with fs.open(image_loc, "rb") as im:
        response = rekognition_client.detect_labels(Image={"Bytes": im.read()})
    
    for label in response["Labels"]:
        if label["Confidence"] >= 60:   #change to desired confidence score threshold, value between [0,100]:
            object_label = label["Name"]
            labels.append(object_label)
    return labels

CITM maps content to a standard taxonomy

CITM compares extracted content metadata (topics from text and labels from images) with keywords on the IAB taxonomy, and then maps the content metadata to keywords from the taxonomy that are semantically related. For this task, CITM completes the following three steps:

1. Generate neural embeddings for the content taxonomy, topic keywords, and image labels using Hugging Face’s BERT sentence transformer. We access the sentence transformer model from Amazon SageMaker. In this post, we use the paraphrase-MiniLM-L6-v2 model, which maps keywords and labels to a 384 dimensional dense vector space.
2. Compute the cosine similarity score between taxonomy keywords and topic keywords using their embeddings. It also computes cosine similarity between the taxonomy keywords and the image object labels. We use cosine similarity as a scoring mechanism to find semantically similar matches between the content metadata and the taxonomy. See the following code:

def compute_similarity(entity_embeddings, entity_terms, taxonomy_embeddings, taxonomy_terms):
    """
    Compute cosine scores between entity embeddings and taxonomy embeddings
    
    Parameters (input):
    ----------
    entity_embeddings : Embeddings for either topic keywords from Amazon Comprehend or image labels from Amazon Rekognition
    entity_terms : Terms for topic keywords or image labels
    taxonomy_embeddings : Embeddings for the content taxonomy
    taxonomy_terms : Terms for the taxonomy keywords

    Returns (output):
    -------
    mapping_df : Dataframe that matches each entity keyword to each taxonomy keyword and their cosine similarity score
    """
    
    #calculate cosine score, pairing each entity embedding with each taxonomy keyword embedding
    cosine_scores = util.pytorch_cos_sim(entity_embeddings, taxonomy_embeddings)
    pairs = []
    for i in range(len(cosine_scores)-1):
        for j in range(0, cosine_scores.shape[1]):
            pairs.append({'index': [i, j], 'score': cosine_scores[i][j]})
    
    #Sort cosine similarity scores in decreasing order
    pairs = sorted(pairs, key=lambda x: x['score'], reverse=True)
    rows = []
    for pair in pairs:
        i, j = pair['index']
        rows.append([entity_terms[i], taxonomy_terms[j], pair['score']])
    
    #move sorted values to a dataframe
    mapping_df= pd.DataFrame(rows, columns=["term", "taxonomy_keyword","cosine_similarity"])
    mapping_df['cosine_similarity'] = mapping_df['cosine_similarity'].astype('float')
    mapping_df= mapping_df.sort_values(by=['term','cosine_similarity'], ascending=False)
    drop_dups= mapping_df.drop_duplicates(subset=['term'], keep='first')
    mapping_df = drop_dups.sort_values(by=['cosine_similarity'], ascending=False).reset_index(drop=True)
    return mapping_df
                                               
#compute cosine_similairty score between topic keywords and content taxonomy keywords using BERT embeddings                                               
text_taxonomy_mapping=compute_similarity(keyword_embeddings, topic_keywords, taxonomy_embeddings, taxonomy_terms)

3. Identify pairings with similarity scores that are above a user-defined threshold and use them to map the content to semantically related keywords on the content taxonomy. In our test, we select all keywords from pairings that have a cosine similarity score of 0.5 or higher. See the following code:

#merge text and image keywords mapped to content taxonomy
rtb_keywords=pd.concat([text_taxonomy_mapping[["term","taxonomy_keyword","cosine_similarity"]],image_taxonomy_mapping]).sort_values(by='cosine_similarity',ascending=False).reset_index(drop=True)

#select keywords with a cosine_similarity score greater than your desired threshold ( the value should be from 0 to 1)
rtb_keywords[rtb_keywords["cosine_similarity"]> 50] # change to desired threshold for cosine score, value between [0,100]:

A common challenge when working with internet-scale language representation (such as in this use case) is that you need a model that can fit most of the content—in this case, words in the English language. Hugging Face’s BERT transformer has been pre-trained using a large corpus of Wikipedia posts in the English language to represent the semantic meaning of words in relation to one another. You fine-tune the pre-trained model using your specific dataset of topic keywords, image labels, and taxonomy keywords. When you place all embeddings in the same feature space and visualize them, you see that BERT logically represents semantic similarity between terms.

The following example visualizes IAB content taxonomy keywords for the class Automotive represented as vectors using BERT. BERT places Automotive keywords from the taxonomy close to semantically similar terms.

The feature vectors allow CITM to compare the metadata labels and taxonomy keywords in the same feature space. In this feature space, CITM calculates cosine similarity between each feature vector for taxonomy keywords and each feature vector for topic keywords. In a separate step, CITM compares taxonomy feature vectors and feature vectors for image labels. Pairings with cosine scores closest to 1 are identified as semantically similar. Note that a pairing can either be a topic keyword and a taxonomy keyword, or an object label and a taxonomy keyword.

The following screenshot shows example pairings of topic keywords and taxonomy keywords using cosine similarity calculated with BERT embeddings.

To map content to taxonomy keywords, CITM selects keywords from pairings with cosine scores that meet a user-defined threshold. These are the keywords that will be used on real-time bidding platforms to select ads for the webpage’s inventory. The result is a rich mapping of online content to the taxonomy.

Optionally store content to taxonomy mapping in a metadata store

After you identify contextually similar taxonomy terms from CITM, you need a way for low-latency APIs to access this information. In programmatic bidding for advertisements, low response time and high concurrency play an important role in monetizing the content. The schema for the data store needs to be flexible to accommodate additional metadata when needed to enrich bid requests. Amazon DynamoDB can match the data access patterns and operational requirements for such a service.

Conclusion

In this post, you learned how to build a taxonomy-based contextual targeting solution using Contextual Intelligence Taxonomy Mapper (CITM). You learned how to use Amazon Comprehend and Amazon Rekognition to extract granular metadata from your media assets. Then, using CITM you mapped the assets to an industry standard taxonomy to facilitate programmatic ad bidding for contextually related ads. You can apply this framework to other use cases that require use of a standard taxonomy to enhance the value of existing media assets.

To experiment with CITM, you can access its code repository and use it with a text and image dataset of your choice.

We recommend learning more about the solution components introduced in this post.

• Discover more about AWS Media Intelligence to extract metadata from media content.
• How to use Hugging Face models for NLP using Amazon SageMaker
• Guidance for Contextual Intelligence for Advertising on AWS

About the Authors

Aramide Kehinde is a Sr. Partner Solution Architect at AWS in Machine Learning and AI. Her career journey has spanned the areas of Business Intelligence and Advanced Analytics across multiple industries. She works to enable partners to build solutions with AWS AI/ML services that serve customers needs for innovation. She also enjoys building the intersection of AI and creative arenas and spending time with her family.

Anuj Gupta is a Principal Solutions Architect working with hyper-growth companies on their cloud native journey. He is passionate about using technology to solve challenging problems and has worked with customers to build highly distributed and low latency applications. He contributes to open-source Serverless and Machine Learning solutions. Outside of work, he loves traveling with his family and writing poems and philosophical blogs.