Statistical techniques were relatively good at Natural Language Processing (NLP) tasks like text classification. For tasks like translation, there was still much room for improvement.
A recent technique that has advanced the field of Natural Language Processing (NLP) for tasks like translation is deep learning.
When you want to translate text, you shouldn’t just translate each word to another language. You may remember translation services from years ago that translated sentences too literally, often resulting in interesting results. Instead, you want a language model to understand the meaning (or semantics) of a text, and use that information to create a grammatically correct sentence in the target language.
Understand word embeddings
One of the key concepts introduced by applying deep learning techniques to NLP is word embeddings. Word embeddings solved the problem of not being able to define the semantic relationship between words.
Before word embeddings, a prevailing challenge with NLP was to detect the semantic relationship between words. Word embeddings represent words in a vector space, so that the relationship between words can be easily described and calculated.
Word embeddings are created during self-supervised learning. During the training process, the model analyzes the cooccurrence patterns of words in sentences and learns to represent them as vectors. The vectors represent the words with coordinates in a multidimensional space. The distance between words can then be calculated by determining the distance between the relative vectors, describing the semantic relationship between words.
Imagine you train a model on a large corpus of text data. During the training process, the model finds that the words bike and car are often used in the same patterns of words. Next to finding bike and car in the same text, you can also find each of them to be used when describing similar things. For example, someone may drive a bike or a car, or buy a bike or a car at a shop.
The model learns that the two words are often found in similar contexts and therefore plots the word vectors for bike and car close to each other in the vector space.
Imagine we have a three-dimensional vector space where each dimension corresponds to a semantic feature. In this case, let’s say the dimensions represent factors like vehicle type, mode of transportation, and activity. We can then assign hypothetical vectors to the words based on their semantic relationships:
Leave a Reply