What Is Stemming in NLP?

The goal of stemming is to identify the common base word of all its variants by removing affixes (prefixes and suffixes). For example, stemming algorithms aim to convert ‘burning’ to ‘burn’ by removing ‘-ing’ and ‘re-run’ to ‘run’ by removing ‘re-.’

The Lovins stemmer, published in 1968, is widely considered the first official stemming algorithm. It features a long list of suffixes, which are matched to English words by longest-case first to truncate original words into stemmed words. The stemmer was designed for information retrieval.

Several advances and improvements have occurred since then.

Porter stemmer

The Porter stemming algorithm removes only suffixes to reduce inflectional forms of words. It identifies patterns of consonants and vowels within a word and systematically reduces them by referencing a predefined list of suffixes. For example, suffixes, such as ‘-ing,’ ‘-ed,’ ‘-ly,’ and others, are removed based on specific conditions like word length or the presence of certain consonant-vowel sequences.

Pros and cons

Porter stemming is efficient because it uses a compact set of rules. However, the algorithm does not retain the stemmed word’s original meaning or grammatical correctness. It may truncate words excessively or confuse words with similar endings but different meanings. For example, ‘universe’ and ‘university’ have the same stem, ‘univers,’ although they have different meanings.

Snowball stemmer

Designed by the original creators of Porter stemmer, the Snowball stemmer addresses several previous limitations to make it more adaptable and practical for NLP applications. One of its primary design goals was to ensure more straightforward implementation in various programming languages. To that end, it was built in Snowball code, a domain-specific programming language that provides a structured and concise representation of stemming rules. Anyone can modify, extend, and implement the code in different contexts.

Pros and cons

Snowball code allows developers to generate equivalent stemmers in multiple languages with minimal effort. The algorithm has been extended to include stemmers in French, German, Spanish, Dutch, and more, with cross-platform implementations across technologies.

The Snowball stemmer also introduces several functional improvements over the Porter Stemmer.

• It incorporates rules for handling prefixes and suffixes for various word transformations.

• The conditions that govern when and how stems are derived have been refined.

These improvements result in a more accurate and consistent stemming process.

Lancaster stemmer

The Lancaster stemmer is more sophisticated than the previously mentioned stemming algorithms. It has a more complex ruleset and iteratively removes or replaces suffixes until it reaches a stem. For instance, when processing the word ‘friendlier,’ the algorithm first removes ‘lier’ and replaces it with ‘ly,’ resulting in ‘friendly.’ It then processes ‘friendly’ again, removing the ‘ly’ suffix to produce the stem ‘friend.’

Pros and cons

The Lancaster stemmer’s ruleset is compact and concise, making it easier to store and implement than algorithms with broader linguistic coverage, like Snowball. Despite being iterative, it is also efficient because it applies simple pattern-matching rules and does not require computationally expensive operations.

However, it tends to reduce words excessively, so they become too short or meaningless.

Krovetz stemmer

Like other stemmers, the Krovetz Stemmer uses predefined rules to strip common suffixes from words. However, before applying suffix removal rules, it checks whether the word exists in a built-in dictionary. If the word is found, no stemming is performed, as the word is assumed to already be in its base form. If the word is not in the dictionary, it then applies rules to remove suffixes and attempts to find a valid word in the dictionary after each transformation.

Pros and cons

Because it outputs valid dictionary words, the Krovetz Stemmer avoids creating non-words or overly truncated stems. However, it is less effective for domain-specific vocabularies or informal text (e.g., slang or abbreviations) that may not appear in the dictionary. It is also designed for English and is not readily adaptable to other languages.

Regexp stemmer

A regexp stemmer is a simple, customizable stemmer based on a list of predefined regular expressions to remove prefixes and suffixes. For instance, regular expressions might match ‘un$,’ ‘$ing,’ and ‘$s,’ and the algorithm steps through the list, removing the letters as coded in the expression.

Pros and cons

A regexp stemmer allows users to define their own patterns for domain-specific use cases. For example, a regexp stemmer can handle unique vocabulary in specialized fields like medicine, law, or finance. It also gives users complete control over which transformations are applied, minimizing the risk of unwanted changes caused by overly aggressive or incorrect general-purpose rules.

However, the regexp stemmer lacks linguistic awareness and cannot account for the grammatical or morphological word structures. It simply matches patterns without understanding the underlying language rules. Snowball is preferred when linguistic accuracy and comprehensive support for natural languages are critical.

Lemmatization is another text-preprocessing technique that reduces words to their root form. However, lemmatization and stemming take different approaches to the problem. Stemming only looks at word structure and letters in the word, while lemmatization takes a more holistic view—it tries to find stems that are meaningful, semantically correct, and retain grammatical information related to the original word.

Meaningful roots

Stemming does not always result in a dictionary-defined root word. For instance, ‘retraceable’ may transform to ‘trace’, making it a full stem, whereas ‘computing’ may transform to ‘comput’, a partial stem.

In contrast, lemmatization reduces words to their base dictionary form word called lemma. For example, ‘computing’ would result in ‘compute.’

Context retention

Stemming does not retain word context. For example, the words ‘computer’ and ‘computing’ are both changed to ‘compute,’ thereby losing their original meaning. However, lemmatization changes ‘computer’ to ‘computer’ as a noun and ‘computing’ to ‘compute’ as a verb.

Similarly, lemmatization of ‘mice,’ the plural form, results in ‘mouse,’ the singular form, whereas when stemming, ‘mice’ typically reduces to ‘mic.’

Performance

The lemmatization process uses in-document context and can vary the form of the word to produce a resulting lemma. It handles different inflectional forms using sophisticated techniques, including a complete morphological analysis of words, whereas stemming uses only partial morphological analysis. However, it is more computationally complex, making it a slower process with a larger memory footprint. Choosing lemmatization or stemming as a technique to standardize words will depend on whether accuracy or speed is more important.