If there is one A.I. platform wearing the crown today, it is OpenAI's ChatGPT (https://chat.openai.com). But how does a GPT actually work? How is it - for heaven's sake - possible for such a system to understand and generate human language so well?
GPTs, or "Generative Pre-trained Transformers", are a type of A.I. that uses machine learning, more specifically transformer architectures, to generate content. No, not the alien car robots that can change shape... 🤖
Let's go through it word by word:
1️⃣ Generative: "Generative" means that these systems are capable of creating something, in this case text. This is different from AI models that predict or recognise, such as classifying images as a dog or a cat. A generative model can produce new sentences or even entire articles that are completely original and therefore not copies of training data. Other popular Generative A.I. platforms: Dall-E or Midjourney (images), Synthesia (video), AlphaCode or GitHub Copilot (code).
2️⃣ Pre-trained: "Pre-training" refers to how these models are developed. Before they are used for specific tasks, they are trained on enormous datasets of text. GPT-3 is estimated to contain 45 terabytes of plain text. For comparison, that is 129 million books averaging 275 pages. An average large city library contains around 1 to 2 million books, so that is a lot of libraries put together ...and GPT-4 was trained on an even larger dataset! You do not just do that quickly on your computer at home... 😉 This pre-training enables the model to acquire broad knowledge of language and grammar.
3️⃣ Transformers: This is where it gets a little more difficult. This is a specific type of architecture in machine learning, best known for its efficiency in handling sequential data such as text, where one word follows another and together forms a certain context. Transformers stand out because of their ability to take the context of an entire piece of text into account, rather than individual words or sentences. They do this using "attention mechanisms", which allow the model to identify and focus on the most relevant parts of a text.
💡 How do GPTs work?
When a GPT model generates a piece of text, it starts with an input (such as a question or an opening sentence), known to many as the "prompt". The model then uses what it learned during pre-training to predict what the next words or sentences should be. In doing so, it takes the context of the entire input into account, enabling it to produce relevant and coherent text.
✉ Hopefully this gives a little more insight into exactly how ChatGPT produces its answers. Feel free to send me a message if you are interested in AI!
