If you represent words with one-hot vectors, you’re making a quiet assumption: every word is equally unrelated to every other word. Distributed representations fix that by doing something simple and powerful: they represent each word as a dense vector learned from context, so similarity becomes part of the data, not something you fake later.
Here’s the mental model:
- A word embedding is a learned feature vector
Think of embeddings as a big table where each word points to a row of numbers. Those numbers aren’t “definitions.” They are features the model learned because they help it predict language patterns.
- Context is the training signal
A classic training idea is: predict nearby words.If two words often appear in similar contexts (“movie” and “film”), the model learns vectors that end up close together. That’s the logic behind approaches like skip-gram and CBOW.
- Similarity becomes geometry
Once words live in a vector space, you can measure relationships with distance and direction. This turns messy text into something models can work with naturally:
- classification becomes easier (better features)
- search and retrieval improve (semantic similarity)
- clustering makes sense (topics and groups form)
- recommendations get smarter (user/item text can be embedded too)
- Why this matters beyond “cool math”
Distributed representations are one of the reasons NLP stopped relying so heavily on manual feature engineering. When the representation is strong, you often need less hand-crafted logic, and downstream models become more stable and generalizable.
The practical takeaway: Embeddings don’t just encode words. They encode relationships, and relationships are what language is made of.