Contents
S23/Session 23/Deep learning/~1 week

Embeddings and sequences

By the end you canYou represent words as dense vectors that carry meaning and understand how networks process sequences of text.

All modulesS23 · Embeddings and sequences▾

One-hot treats every word as something isolated: dog and cat are as different as dog and television. Embeddings put words into a space where closeness means close meaning. It's the idea all of modern NLP rests on. This module gives you the intuition, so the big models later aren't magic.

01

The problem with one-hot representation

In classic NLP (S16), a word was a position in a huge vector of zeros with a single 1. The problem: all the vectors are equally far from each other. The model has no way to know that king and queen are related, or that good and excellent are close. The meaning is lost completely.

The vectors are also enormous (as big as the vocabulary, tens of thousands of dimensions) and sparse (almost all zeros). Inefficient and with no semantic meaning. Embeddings solve both problems.

02

Embeddings: meaning as position in space

An embedding is a dense, short vector (say 100-300 numbers), learned for each word, so that words with close meaning have close vectors. They're learned from contexts: words that appear in similar contexts get similar vectors. Dog and cat both appear next to food, fur, animal, so they end up close.

Word2Vec learns such vectors in two variants: skip-gram (from a word you predict the context) or CBOW (from the context you predict the word). A famous result: vector arithmetic works. The vector king minus man plus woman lands close to queen. Meaning becomes geometry.

vec(king) - vec(man) + vec(woman) ≈ vec(queen)
Meaning relations become directions in space. The man-woman difference is the same direction as king-queen.
03

FastText: important for Romanian

Word2Vec treats each word as a whole. FastText goes by subwords (pieces of letters): it represents a word from its fragments. That matters a lot for Romanian, with its rich inflection. Merg, mergem, mergeau share subwords, so they get related vectors, even if one of the forms is rare or unseen at training.

You can use pretrained vectors (ready-trained Word2Vec, FastText, GloVe) as feature extractors, without training anything: you take the word vectors and put a simple model on top. With little data, it's much better than starting from scratch.

04

Sequences: RNN, LSTM, GRU

An embedding gives a word meaning, but a sentence is a sequence where order matters. Recurrent networks (RNN) process the text word by word, holding a state that summarizes what they've seen so far. The problem: on long sequences, the gradient vanishes and the network forgets the start of the sentence.

LSTM and GRU solve this with gates: mechanisms that decide what information they keep, what they forget and what they let through. That way they can hold context from further back. You don't have to implement them from scratch, but understand conceptually why they work: the gates protect important information from being erased step by step.

RNNs and LSTMs have largely been replaced by transformers in top-end NLP, but they stay important for understanding the idea of sequence processing and the long-range memory problem, which is exactly what transformers came to solve better.

Remember
  • One-hot loses the meaning; all words are equally far apart.
  • Embeddings are dense vectors where closeness = close meaning, learned from context.
  • Word-vector arithmetic works: meaning relations become directions.
  • FastText uses subwords, essential for Romanian's rich inflection.
  • RNNs process sequences but forget over long ranges; LSTM/GRU hold on with gates.
Index
From one-hot to a dense vectorWord2Vec (skip-gram, CBOW), GloVe, FastText, and why subwords matter for RomanianWord-vector arithmeticRNN, the vanishing gradient, LSTM and GRU conceptuallyPretrained models as feature extractors
If you want more