Intro

I have been spending a lot of hours on AI, but it feels like all I do nowadays is ask AI and patiently wait for some sort of response back from agents. The more I do this, the more I feel like I am getting dumber... or maybe I truly am getting dumber. So I made a decision to take a little more time to understand a keyword or concept that I think is crucial for artificial intelligence OR super intelligence.

So what the hell is a transformer? I am a lazy person, so I decided to use Perplexity to get the definition of a transformer. The only transformer I know is the electrical transformer, which has nothing to do with AI. Oh, btw, Perplexity is still the best research agent out there, imo.

Transformer

According to Perplexity (which I believe got it from Stanford):

A transformer is a type of neural-network architecture that learns relationships between pieces of data—such as words in a sentence—to build context-aware representations. It is the foundation of many modern language models, including GPT and BERT. Transformers were introduced in 2017 and are also used for images, audio, and other kinds of data.hai.stanford

It sounds like a transformer is crucial. One key point I want to understand: in any other kind of engineering, a transformer changes a characteristic of its input. So what does that mean in AI? What does a transformer change?

According to Perplexity (citing IBM):

The key idea is attention: when processing one part of the input, the model calculates which other parts are relevant to it.ibm

I am still a little confused, but after I did a bit more research, I came to a conclusion: before a transformer even starts, the input gets turned into numbers (words → tokens → vectors).

What the transformer changes is those numbers. It updates each word's numbers based on the words around it, so "bank" in "river bank" ends up different from "bank" in "bank account."

What does that mean? In this world, a transformer is nothing more than a mathematical blueprint. It shows how numbers are related to each other: attention scores how relevant each word is to every other word. Words with similar meanings also end up with similar numbers. A large language model uses all of this to compute the most likely word to come next.

 
 To sum it up so far: text is split into tokens, the basic units a model works with. Each token gets an ID number, and each ID is turned into a vector called an embedding, which captures meaning. Next, attention lets each token's vector update itself based on the other tokens around it. AND yes. "update" Vector numbers are not constant. It can be changed.

graph LR
A[Text] --> B[Tokens] --> C[Vectors] --> D[Transformer: attention] --> E[Next word]