GPT-3 is a third generation neural network language model created by the
OpenAI research laboratory in San Francisco.
Neural networks can either be created in software or hardware and each neuron has inputs and a result output that can be connected to other neurons. Each neuron sums a weighted average of its inputs that can result in it 'activating' - that is producing a result output - or not.
"Backpropagation" refers to a way of training neural networks - that is getting each neuron to adjust the weights of its inputs.
Think of it like back in school when you had to find the minima or local minima of a function in two dimensions of x and y. Then translate that concept in your mind to finding the minima in some 3D space, x,y,z, such as a valley in a hilly or mountainous landscape. Then make the conceptional leap to some much higher dimensional space with peaks and valleys in that space still having minima. That is akin to what goes on in the algorithms of a neural network where the inputs are x,y,z,a,b,c,d and so on. Just as you used the differentiation of gradients to help find minima, at their heart neural networks are doing the same high school maths. There is nothing organic, wet or brain-like squidgy about them.
Machine learning as a field has been around for decades. Funding and research in it has waxed and waned over the years.
The approach of using neural networks for machine learning had been touted by some computer scientists but year after year the reality of the utility of neural networks would fall far short of the hopes of the designers. There was a period that computer scientists refer to as the "AI Winter" from the mid 1980's to the mid 2000s where funding and interest largely dried up. To declare one was involved in AI research to other computer scientists and engineers during that period one would be judged as a having a career
with no future prospects.
However, in the past decade, neural networks have started to show increasing promise in various areas. Some researchers say that the problem before was partly because the statistical methods they rely on to find those local minima seemed to need a diet of large datasets.
There simply wasn't enough data before.
Enter the Internet.
Now programs such as GPT-3 have been expanded to have hundreds of billions of neurons and fed rich diets such as the entire contents of Wikipedia, they start to become interesting and I dare say would beat the pants off most of us when it comes to general knowledge.
GPT-3 was introduced in May 2020 and since then several videos have appeared on the Internet demoing it.
This first video is in some ways the least interesting, simply because we see more of the human interviewer than the GPT-3 AI he is interviewing.
Nevertheless, it is worth watching first as Eric Elliott provides some background on GPT-3 and context on how the video was produced including the use of the post-processed avatar :-
https://youtu.be/PqbB07n_uQ4
However, this playlist of 11 short videos between Dr. Alan D. Thompson and a GPT-4 AI named 'Leta' provides a continual one-on-one interaction between human and computer.
There are some cool moments, like when Thompson asks Lena, "If the sky were the sea, what would it make the birds?" and she responds, "Flying fish".
Enjoy!
https://www.youtube.com/watch?v=5DBX...-U0zxCgbHrGa4V