ChatGPT went all in on increasing the amount of data to work with to boost the AI capabilities, but just like all other tech sectors, it seems pretty clear that we find diminishing returns with each doubling in the amount of data.
Currently we have reached that point where only trillion dollar companies are capable of building better and better AI fast.
General AI even now can regurgitate the best answers, but it hasn't shown the capacity to create something truly new, that comes with the randomness that emerges from biological evolution.
General AI is limited by the total knowledge of humanity being fed into it, beyond which it cannot grow no matter what. It is the world's smartest answering machine.
Will AI be able to replace humanity? No.
Will AI be able to make robotic equivalents to human labor? Yes.
Today, the most advanced AI in the world is Gemini by google.
It is taking the efforts of the 4th most valued company in the world to keep developing and upgrading AI further.
The current AI boom was a random discovery and it will slow down as fast as it rose up.
General AI will very likely peak at being 2-3 times smarter than the smartest human, then go no further.
This is because it cannot create anything new, it can only pattern match all the data that already exists out there.
Until and unless we give AI the ability to "mutate" like human DNA does, it will reach an upper limit and stagnate at that point.
Conclusion:
AI won't take over the world. It will be another tool to help humans create more things faster, and finish up all the current backlog of research projects.
Jump in the discussion.
No email address required.
What a bummer, there was some hope that this wouldn't be the case
https://gwern.net/scaling-hypothesis
I wonder whether this doesn't invalidate the scaling-hypothesis, but actually is just a result of us using up all high quality training data and now trying to throw the trash into the model too.
Jump in the discussion.
No email address required.
I think of it more as moving towards a core. Just as we see with empires and technologies and nature, you have a large boom, then a stagnation, then a receding area which is more like a regrouping of the better pieces, for example - people figuring out how to get similar quality AI output with lower number of parameters to create a smaller sized AI, the issue currently would primarily be that there isn't enough data and storage out there to feed a single AI forever. Sure google could afford that much data storage space to exponentially increase the number of parameters a bunch of more times, but even they would run out of space at some point.
At least that's my take on it.
As I understood it there was no high quality data, they just threw everything they could from the beginning. The issue is, to give an AI the internets worth of data, you would have to create a second internets worth of storage. You would also need a supercomputer powerful enough to process all that data.
Basically you are going to have one super specialized company leading in AI like we do with semiconductors.
Jump in the discussion.
No email address required.
These are text based models, storage isn't that much of an issue. Text is very easy to store and compress. Even storing all images on the internet probably wouldn't be a problem and could be done with at most a few million dollars (archive.org manages to do that just fine and they're not rich). Video is the real beast that will be too expensive for most but the biggest corpos.
Of course you can't exclude all low quality data and go through it by hand, but you can definitely heuristically control the ratio of it. One trivial example to lower the average quality would be to include the youtube kids comment section. Each source has a degree of trust (established news papers, reddit, twitter, irc-logs and spam-emails) bottoming out at text that is most likely already machine generated or code by junior programmers that are somewhat cheap to filter.
Especially AI generated text is becoming an issue as SEO spammers are picking up.
Jump in the discussion.
No email address required.
I think Grandma answers your question better than I do.
https://rdrama.net/h/nerdshit/post/281855/ai-development-is-going-to-slow/6650211#context
What you said makes sense. Thanks for educating me.
AI turned out to be halfway blowout just like autonomous cars.
I think we have reached that point where meaningful progress can only be made when the entire planet is collaborating on a project like we did with the ozone layer and are now trying to do with climate change and plastic pollution.
Jump in the discussion.
No email address required.
More options
Context
More options
Context
More options
Context
More options
Context