GPT is not a mystery anymore
Yesterday I came across this YouTube video explaining what exactly a GPT does behind the scene. It kinda reveals the mystery of the black box, which is known as Grokking. This is a weird phenomenon which indicates when a GPT model moves from learning to creating. Interestingly, this transition is not immediate. There is usually a long delay between those two events. It was first discovered by OpenAI by mistake—no wonder great discoveries are made by mistakes.
The video follows the research of a two years old article in which researchers used a simplified GPT to recreate the transition from learning to creating. What was mind blowing is that Fourier series like patterns appeared when GPT develops an understanding of the problem.
Isnt it crazy!!!
So what ChatGPT has now is some complex combination of mathematical functions representing human knowledge. This made me hopeful as, contrary to the claim, its not an intelligence but mathematical function giving output for an input.