Word Embeddings

Word Embeddings transform text into a sequence of vectors.

Background

Word Vectors

Word vectors are dense representations of words (in contrast to a huge sparse binary matrix), that allows to feed text into neural networks.

A Word embeddings is the transformation of all input words into word vectors.

This approach is also referred to as Word2Vec.

Skip-Grams

Skip-grams are simply pairs of words that are 1 or more words apart. Skip grams are frequently used to calculate word embeddings:

1. Tokenizing the data follows the same principles as in a Bag-of-words. To represent spatial relationships between words.

2. Consider words and their context by generating pairs of words up to N words apart.

3. Count the co-occurence of words X (a huge sparse matrix)

4. Optimize a loss function for the word vectors w over words i and j

\[J \approx \sum_{ij}(w_i^T w_j - log X_{ij})^2\]

Pretrained embeddings

Because calculating a word embedding from scratch is costly (millions of tokens, 1 million parameters and more), it is often useful to start with pretrained word embeddings:

• BERT (state-of-the-art in 2019)

• GloVe (Stanford).

• Gensim / Doc2Vec – applies the embedding principle to text paragraphs

• spacy also has pretrained Word Vectors.

Embedding Layers

An embedding layer transforms one-hot encoded input (usually with a huge number of dimensions) to much fewer dimensions, e.g. representing a word as a 50 or 200-dimensional vector.

There are plenty of uses and architectures for Word Embeddings. The majority of these have an Embedding Layer either as the first layer (to understand text) or as the last layer (to produce text).

Here is a typical architecture

To condense the information in a longer text sequence, you can use one or more 1D convolutional layers.

Cosine Similarity

You can compare word vectors using the cosine similarity:

import numpy as np

def cosine_similarity(a, b):
    """Returns cosine similarity of two word vectors"""
    return a.dot(b) / (np.linalg.norm(a) * np.linalg.norm(b))

Embeddings for other types of data

Embeddings can be used for other types of sparse data as well, e.g. to build recommender systems (see the parallel to NMF).

Links

---

Examples

Create GenSim Vectors

from gensim.models import word2vec

corpus = [
   "She loves you yeah yeah yeah",
   "see you later alligator",
   "see you later crocodile",
   "i just call to say i love you",
   "and it seems to me you lived your life like a candle in the wind",
   "baby you can drive my car",
   "we all live in the yellow submarine"
]

tokenized = [s.lower().split() for s in corpus]

wv = word2vec.Word2Vec(tokenized, size=7, window=5, min_count=1)
words = list(wv.wv.vocab)
len(words)

print(wv['crocodile'])

wv.wv.most_similar('crocodile')

Download Pretrained GenSim Vectors

Warning

This is a 4 GB file that will be loaded to memory. It will require a lot of RAM.

import os
from keras.utils import get_file
import gensim
import subprocess


MODEL = 'GoogleNews-vectors-negative300.bin'
path = get_file(MODEL + '.gz', 'https://s3.amazonaws.com/dl4j-distribution/%s.gz' % MODEL)
if not os.path.isdir('generated'):
    os.mkdir('generated')

unzipped = os.path.join('generated', MODEL)
if not os.path.isfile(unzipped):
    with open(unzipped, 'wb') as fout:
        zcat = subprocess.Popen(['zcat'],
                          stdin=open(path),
                          stdout=fout
                         )
        zcat.wait()


model = gensim.models.KeyedVectors.load_word2vec_format(unzipped, binary=True)

Similarity of Words

model.most_similar(positive=['espresso'])

Vector Addition

See also

The code examples above are adopted from the Deep Learning Cookbook by Douwe Osinga (https://github.com/DOsinga/deep_learning_cookbook).

Exercises

Exercise 1

Load a set of pretrained word vectors into a Python program.

Exercise 2

Calculate cosine similarities between pairs of words.

Exercise 3

Calculate the word vector resulting from

glove - hand + foot

Calculate the cosine similarity between the resulting vector and what you think might be the real answer.

Exercise 4

Calculate cosine similarities between word pairs like (eagle, good) to find out whether there are good or evil animals.

How do you interpret the results?