What Are Embeddings

When Spotify picks your next song, it's reading a map. When a CIA computer is looking for Jason Bourne's face in a CCTV feed, it's reading a map too. What kind of maps? Do they have north and south? These maps are called embeddings and are also a big part of what makes LLMs like ChatGPT able to grasp the meaning of things. Embeddings are maps that, just like regular maps, help us find out how close or far apart things are, or in which direction to go to get from point A to point B.

In maps places are defined by two numbers, longitude and latitude. They represent how far north or south a point is from the equator, and how east or west it is from the Greenwich meridian. It is with those two numbers that we can find out how far our destination is.

Take this ship, for example. Its crew is plagued with scurvy. They desperately need to get to the closest shore and get some lemons. If they don't reach land in 2 days or less, half of the crew will die. Click on the different ports to find their coordinates and distances and navigate to the closest one. There's no time to lose!

That was close, but we did it. By using the coordinates and Pythagoras we were able to find the nearest port and our lovely crew can move on to pillaging, conquering and spreading the word of God.

All in a nice retro console look. The colours on the console, by the way, also have coordinates, but instead of having a north-south axis and an east-west one, each colour is represented by three numbers that represent how much red is in a colour, how much green and how much blue. You might have heard of this system as RGB. The axes only go one way, 0 is none to 255 is all. Pure green would be [0, 255, 0], white would be [255, 255, 255] and black would be [0, 0, 0].

With all the possible combinations between the three values we can show over 16 million different colours. That's why most screens use it. Under every point in a screen there are 3 small lights, one for each of the primary colours.

If we want to mix two colours we just sum their components.

We can try mixing —adding— red and green:

  [255,   0,   0]  red
+ [  0, 255,   0]  green
  ---------------
  [255, 255,   0]  yellow

We can visualise this in a 3-dimensional map, where each colour is a point placed at the intersection of its red, green and blue values. Similar colours are closer together, black and white are at opposite corners.

This cube doesn't show all 16.7 million colours, it shows 54. One reason is that it would be difficult to see the individual points. The other reason is that those 54 are all the colours that the NES could represent.

Game developers and designers back then had to ask themselves a question: which colour in this very limited palette is closest to the one I want?

Quite useful indeed!

Embedding concepts.

A colour splits neatly into ingredients: this much red, this much green, this much blue. A song doesn't. A breakup ballad and a stadium anthem can use the same guitar, the same drums, the same voice, and still sound nothing alike. Two songs feel close when both are sad, or when both make you want to dance.

Sad and happy are the two ends of one line. Sleepy and danceable are the two ends of another. It's north and south on the map again, with the equator in the middle, except the ends are ideas.

Wave Race 64, Nintendo's 1996 jet ski game, puts lines like these on its tuning screen. Before a race you set your engine somewhere between Dash (quick off the start) and Top End (slow to get going, faster once it does). Grip runs from Loose (faster, but more likely to slip out of a turn) to Tight (holds the turn, gives up some speed). A third slider sets the handling.

Which end is the good one? Neither. You pick the one that suits how you ride, and that changes what the middle means. On the colour cube, a red of 128 is half the red you could have. On a Wave Race slider the middle is Normal, where every jet ski starts, and it means balanced: some speed, some control. Hold on to the idea of a line where both ends are worth having. You'll need it at the end.

The game has four riders, and each one's jet ski leans its own way. Set the sliders to how you like to ride and see who ends up closest.

Ayumi Stewart has 5 stars for acceleration and 2 for top speed, so she sits out towards Dash. Miles Jeter turns better than anyone and grips worse than anyone. Dave Mariner is the hardest to knock off course and the hardest to steer. Ryota Hayami has no big strengths and no big weaknesses, which is why the manual rates him for beginners and advanced riders alike. Who's the best rider? Whoever sits closest to your sliders.

Football games use the colour kind of axis, and far more of them. In EA Sports FC every player is rated on about 30 attributes, from sprint speed and stamina to finishing and sliding tackles, each out of 99. Like the colour channels, they only go one way, from none of a skill to as much as the game allows. To keep the player cards readable, the game squashes them into six: pace, shooting, passing, dribbling, defending, physical.

Six is too many for a cube. It fits on a radar chart.

I love the spider chart. You can see the shape of a player, and compare two players, or two teams, in an Augenblick. A winger spikes out towards pace and dribbling, a centre-back towards defending and physical. Add a third player and the shapes pile up into a tangle. With 30 attributes instead of six you couldn't draw it at all, so this is where the drawing stops.

You don't need to see a space to measure it. Remember how you found the nearest port: take the difference on each axis, square it, add the squares up, take the square root. Nothing in that recipe cares how many axes there are. For a colour you add up three differences. For a footballer you add up 30. The distances still come out, and close still means similar.

Imagine Florian Wirtz gets injured. Liverpool agreed to pay up to £116.5m for him in 2025, and now they need someone who plays like him. These are the eight players closest to him in the game's data. Pick one to see them side by side with Wirtz.

Three of the eight play on the right flank: Olise, Saka, Foden. The distance only looks at the ratings, and by the ratings all three play a lot like Wirtz. The real find is fourth on the list. Dani Olmo is almost as close as Saka, and the game values him at €62m, less than half what it puts on Wirtz. That's a Schnäppchen if I've ever seen one.

You've been using embeddings since the scurvy ship.

An embedding describes things as lists of numbers, so that things that are alike end up close together. The name comes from maths: you embed each thing in a space by giving it a spot. A port got two numbers. A colour got three. A jet ski got three sliders, a footballer about 30. Once something is a list of numbers you can ask "what's close to this?" and answer it with the same Pythagoras that got the crew to their lemons.

That question runs a lot of what you use every day. Spotify keeps its songs on a map, and your next song is one that sits close to what you've been playing. A news site's "you might also like" box looks for articles close to the one you just read. A search engine that finds "budget airlines" when you typed "cheap flights" is matching two points that sit close together, even though the words are different.

Faces work the same way. The CCTV system hunting Jason Bourne turns every face in the feed into a list of numbers and flags the ones that land near his (Google's FaceNet, from 2015, used 128 numbers per face). Language models do it with words. Before ChatGPT does anything else with what you type, it turns each word, or piece of a word, into a list of thousands of numbers.

So far every number you've used had a clear meaning: longitude, red, grip, pace. Someone chose each axis and gave it a name. Spotify's map has no street names. Nobody decided that axis 17 means sad or that axis 203 means danceable. A computer placed millions of songs so that the ones people play together land close to each other, and the axes are whatever came out. You can still measure distances on a map like that, and your next song still comes from nearby. You just can't read the labels, because there aren't any.

This is where the Wave Race sliders come back. On a map without names, meaning can still turn up as a direction: walk one way and the songs get sadder, walk the other and they get happier. Two ends, like Loose and Tight, except nobody printed them on the screen. You have to find them and name them yourself.

Part 2 starts there. In Embeddings in Fewer Dimensions I take the footballers' 30 numbers, squash them down to two and see what the squashing keeps and what it throws away.