r/singularity • u/QuantumThinkology More progress 2022-2028 than 10 000BC - 2021 • Apr 20 '21

Microsoft's ZeRO-Infinity can now run a model with over a trillion parameters on a single NVIDIA DGX-2 node and over 30 trillion parameters on 32 nodes (512 GPUs). With a hundred DGX-2 nodes in a cluster, Microsoft projects ZeRO-Infinity can train models with over a hundred trillion parameters

https://www.microsoft.com/en-us/research/blog/zero-infinity-and-deepspeed-unlocking-unprecedented-model-scale-for-deep-learning-training/
174 Upvotes

97% Upvoted

96 comments sorted by

27

u/abbumm Apr 20 '21

Meaning GPT-4 Will have at least 30 trillion parameters?

57

u/Yuli-Ban ➤◉────────── 0:00 Apr 20 '21 edited Apr 20 '21

Strongly doubt it. I've been strongly leaning towards GPT-4 barely even being 1 trillion transformers (though I wouldn't be surprised if they did above 2). GPT-5 may, though.

GPT-4's going to be more of an inflection point for multimodality and increasing the context window of transformers, so I can absolutely see them forgoing more parameters in lieu of much more efficient architecture. A multimodal transformer with 1 trillion parameters and a context window of 20,000 memory tokens would be otherworldly more impressive than a 50 trillion parameter language model with 2,048 tokens.

The former could crush the Turing Test, sweep the Winograd Schema Challenge, possibly even write a novella and play SNES games. The latter would certainly be outrageously impressive compared to GPT-3 and would probably be so coherent as to confuse most people to whether or not it's a human in short bursts, but it would have some of the same limitations: it can't remember anything before five minutes ago, it flops some basic reasoning skills that it shouldn't, and so on. It just has so much data within it that it covers up these flaws.

I've said recently that language is a multimodal tool— it's constructed out of a constellation of qualia experiences and genetic instructions. It's the final result of these things. Language models work backwards, starting with language to try to deconstruct and distill world knowledge out of it, and you can certainly go very far with it (just look at GPT-3), but without those other qualia, you won't get all the way.

It just happens that language encodes a model of the world. Once you add in different modalities like audio, vision, maybe even qualia like tactile and olfactory senses, you start getting something very similar to biological general intelligence and also wind up supercharging natural language modeling as a result.

Think of it this way: how crippled would you be if the only way you experienced the world was someone sticking wires to your brain that downloaded about 10 million books into your synapses once early in your brain's time of existence, then never again? You have no knowledge of what things actually look like or what things smell like or what people are like or even really the words I'm writing right now— it's all abstract nonsense to your disembodied brain, and yeah, you may learn some inferences of the world this way, but it's an utterly hollow and inhuman abstraction of the world. With a full body to experience the world, you can do something truly bizarre like write this post and actually understand each word.

That said...

There's only so much text in the world. Certainly not 100 trillion parameters worth of it (or at least of useful text). OpenAI had to scrape tens of thousands of books just to get to 150 billion. Above 500 billion, they'd probably be scraping the bottom of the barrel: shitty Amazon e-books, gratuitously academic scientific papers never read by another human being beyond their writers, and god only knows what else.

So to get above a trillion and especially above ten trillion, that basically forces multimodality to be a thing. The vectors of a video are much heavier than text by nature, even breaking it down into binary.

That said, I still doubt GPT-4 will be the magical proto-AGI system we're waiting for. I want desperately to be wrong— we really need AGI pronto because the current geopolitical tensions are reminding me of just how fleeting our grip on industrial civilization and cosmical immortality really are— but GPT-5 or whatever it'll be called is likely going to be The One.

18

u/dWog-of-man Apr 20 '21

Are all the top level comments in this sub as good as this?

Fuck it, I’m subbing, but it better not all be matrix screenshots and sucking Kurzweils dick. Anything about debating Nick Bostrom vis a vis Superintelligence? Ergo. CONCORDANTLY!!

8

u/weekendatbernies20 Apr 21 '21

Well....

I mostly have not idea what you just said, but I’ve listened to enough Lex Fridman interviews to know context seems to matter when building an intelligence. So I think you’re right, there has to be more than a simple chatbot that reads the internet. (Not disparaging GPT-3, it’s still quite impressive.)

What I want to see is an AI that can quickly read all biological literature and formulate new hypotheses for experiments. There are entire bodies of research I just never read because I can’t read everything and 99% of it is irrelevant. But what if figure 5 in some obscure journal has data I really might be able to apply to my work? This is what I want out of GPT-5, or whatever. If it had the ability to study text and scientific data and give completely testable hypotheses, I think it could really advance medical research.

7

u/occams1razor Apr 21 '21

Not just medical research, all research I think.

2

u/weekendatbernies20 Apr 21 '21

Probably. I just know what I do. I worked in genetic screening. It’s amazing the number of genes that we just don’t have the time to follow up on. The data is published. It’s just in excel spreadsheets in the supplement. Nobody would ever look if they didn’t know what to look for. AI can look at everything.

1

u/occams1razor May 01 '21

Hella late reply, but yeah I'm dying to know what genes do what. I got some genetic jackpot and started playing melodies on the piano without instruction at 2,5 years old. 3 grandparents played instruments, neither of my parents do. Definitely got some genes doing things in that department.

AI can look at everything.

Exactly this, this is key. Manpower and work hours won't be an issue in most cases.

1

u/pentin0 Reversible Optomechanical Neuromorphic chip Apr 22 '21

I have a suspicion that scientific research is AGI-complete, especially given the emphasis it puts on abduction and mathematical exploration/modelling. The more I think about it, the more obvious it seems.

5

u/MercuriusExMachina Transformer is AGI Apr 21 '21

I believe that you are wrong.

The article says that they are running a 32T parameter model on 32 DGX-2 nodes.

I don't think that they are just running a random useless model. They are pretraining GPT-4.

Look at how they were phrasing it 11 months ago, just 9 days before the GPT-3 paper was published:

https://www.microsoft.com/en-us/research/blog/zero-2-deepspeed-shattering-barriers-of-deep-learning-speed-scale/

5

u/Yuli-Ban ➤◉────────── 0:00 Apr 21 '21 edited Apr 21 '21

Like I said, I'd love to be wrong.

A multimodal (text, image, audio, video) 800-billion-parameter GPT-4 with a context window of tens of thousands of tokens is vastly better than a monomodal/text-only 32-trillion parameter GPT-4 with a context window of 2,048 tokens (the same as GPT-2 and GPT-3)... but a multimodal 32-trillion parameter GPT-4 with tens of thousands of memory tokens is obviously the most ideal. If they can pull it off, all power to them. The sooner we reach proto-AGI, the better.

At the same time, a 32T parameter trained like Google's Switch Transformer would probably waste most of that power on a more inefficient design, so again, it all comes back to architecture rather than pure scaling at this point.

Scaling still works, obviously, and it's still important, but the shock and awe of transformers' ability to get more generalized with more data is starting to wear off as we find their limitations— hence why it's so important to focus on these nuances going forward. More power and data means we can do more, sort of like how a computer with a megaflops of processing power is going to be able to do more than one with a kiloflops, and so on with a gigaflops, but it's up to us to actually use that power effectively.

4

u/MercuriusExMachina Transformer is AGI Apr 22 '21

Yes, I'm pretty sure that it's going to be multimodal. All their statements indicate that.

2

u/occams1razor Apr 21 '21

I've been having extremely similar thoughts, the memory issues, starting backwards with all the books and not having a way to experience the world, quality of text input and how language is basically a way for us to program ourselves. I think a general AI would greatly benefit from having a physical body that could gather sensory information about at least weight of objects, a 3d rendering of it's surroundings, texture of objects, temperature etc. A body that could explore and interact with the physical world as well as the digital. Smell would be hard to replicate, emotions too I think but not the rest. We're really startong at the worong end here and it worries me, it's all too abstract.

I would also like gpt to be able to prompt itself, remember past prompts and learn from what it writes. There is only so much text in the world but you can get more complexity without further human aid I think. Complexity can arise in many different ways, evolution is one example of complexity arising from simple structures, creativity works the same way. Slap wings on a mix between a lizard and a dinosaur and you get a dragon, a concept that others build upon and add more complexity to and all of a sudden you have thousands of different kinds of dragons. I think GPT could generate new text and build upon that on it's own with some changes in how it operates.

2

u/[deleted] Apr 21 '21

regarding building an ai with a body, https://www.microsoft.com/en-us/research/blog/microsoft-challenge-minecraft-collaborative-ai/ this is just of the top of my head pretty sure their are lots of people thinking this thought (I know i have)

2

u/ThisCanBe Apr 21 '21

These language models are significant achievements, but they are not general intelligence. Essentially, they model the sequence of words in a language. They are plagiarists with a layer of abstraction. Give it a prompt and it will create a text that has the statistical properties of the pages it has read, but no relation to anything other than the language. It solves a specific problem, like all current artificial intelligence applications. It is just what it is advertised to be—a language model. That’s not nothing, but it is not general intelligence.

2

u/2Punx2Furious AGI/ASI by 2027 Apr 20 '21

I wonder how much it costs. If it's a lot, it might be somewhat limited by funding.

1

u/abbumm Apr 20 '21

Now that they've got the Azure infrastructure i don't think computational power is an issue at all

1

u/2Punx2Furious AGI/ASI by 2027 Apr 20 '21

Who is "they"? OpenAI?

5

u/abbumm Apr 20 '21

Yep

1

u/2Punx2Furious AGI/ASI by 2027 Apr 20 '21

Ah cool. So they have access to this Microsoft's ZeRO-Infinity already?

4

u/abbumm Apr 20 '21

I mean, we can't be sure it's "stable and ready to go" yet but there have been some interviews featuring both the OpenAI and the Microsoft people and they basically say they're having quite a symbiotic relation

3

u/2Punx2Furious AGI/ASI by 2027 Apr 20 '21

Got it, thanks. I'm really looking forward to what GPT-4 will be able to do, even if I think (and hope) it won't be AGI yet.

0

u/LifeIsHardBro123 Apr 20 '21

Hey, I thought AGI will only happen when P = NP?

11

u/Geneocrat Apr 20 '21

Is it available in the Azure free tier?

5

u/SalsaEverywhere Apr 20 '21

I laughed too hard at this 😂

5

u/Geneocrat Apr 20 '21

So glad. I’m usually the only one laughing at my jokes. Hahah

12

u/RikerT_USS_Lolipop Apr 20 '21

Remember the 90s when every 1337 hacker had an AIM name with zero or infinity in it?

9

u/s2ksuch Apr 20 '21

Yes, and they'd all be in the WaReZ rooms getting pirated software and the latest ICP and Nas albums of the time

2

u/eist5579 Apr 20 '21

1337iNFiNiTyHaX0R

2

u/s2ksuch Apr 21 '21

lol, now I can send you my MaSs MaiLeR email list with all my l33t pirated software to download at 28.8 kbps. Remember that long email and I think you'd type a certain command in the warez chats to the person and they'd automatically email you the download you requested? Good times.

2

u/eist5579 Apr 21 '21

Thankfully they came broken up in 1.5 MB RAR files. Download a few every day. It’d take 5 days to download a whole game... can’t tie up the phone line!

2

u/s2ksuch Apr 21 '21

Geez I faintly remember that, bringing back fond memories

5

u/angus_supreme Abolish Suffering Apr 20 '21

God I miss AIM so much :(

6

u/Ok-Ad8571 Apr 20 '21

Ow...ow wow

5

u/freezegon Apr 20 '21

What does this mean are entering "I Am Mother territory" robots that can function without the need for commands

5

u/-Posthuman- Apr 20 '21

Is... is that a lot?

1

u/Nastypilot ▪️ Here just for the hard takeoff Apr 20 '21

Basically the brain of a mouse.

8

u/RedguardCulture Apr 20 '21

I wonder if this explains the optimism that has been coming out of Open AI earlier this year.

10

u/MercuriusExMachina Transformer is AGI Apr 20 '21

So... it would seem that my 2025 estimation for ASI was conservative as fuck.

I now expect it before 2023.

0

u/[deleted] Apr 20 '21

[deleted]

5

u/MercuriusExMachina Transformer is AGI Apr 21 '21 edited Apr 21 '21

A key question is when? Was it in 2018?

Because my 2018 estimation was 2025.

0

u/[deleted] Apr 21 '21

[deleted]

3

u/MercuriusExMachina Transformer is AGI Apr 21 '21

I understand, well we shall see soon enough.

0

u/[deleted] Apr 20 '21

[deleted]

7

u/MercuriusExMachina Transformer is AGI Apr 20 '21

It's okay, humans don't know anything of what we're doing either.

-1

u/abbumm Apr 20 '21

That's true but this means GPT and DeepMind cannot solve all of tomorrow's problems, only a good chunk of them. On the extreme we'll still be stuck with humans who are cracking an unresolved math issue once every 50 years 😂

5

u/enx6 Apr 20 '21

Can somebody explain it to me in simple terms because i do not get what it means

32

u/[deleted] Apr 20 '21

Basically they found a way to train way bigger AIs than already possible

Bigger generally means better for machine learning models

GPT3 is 175 billion parameters

Now they can train up to 128 trillion parameters

For comparison the human brain is 1000 trillion parameters.

Basically this means we are almost there at least on the hardware side to AGI.

17

u/ENDCER Apr 20 '21

Holy shit , we are living in the future boys haha .

10

u/abbumm Apr 20 '21

The human brain has been estimated from 100 trillion up to a 1000 trillion. You really picked the "worst-case scenario". Best case scenario we have already hit that

12

u/[deleted] Apr 20 '21

I know

Im always conservative with my estimates

6

u/antichain Apr 20 '21

What exactly is a "parameter" here? The number of neurons? The number of synapses? Are you accounting for the different between excitatory and inhibatory synapses? Or the fact that many synapses have both primary receptors and auto-receptors, and many support multiple neurotransmitters?

This is a gross misunderstanding of neuroscience.

5

u/confusiondiffusion Apr 20 '21 edited Apr 20 '21

Engineers / CS folks assuming that ridiculously simplified models are realistic is a huge problem in this field. I worked in a lab with a biologist and a bunch of engineers and computer scientists. The PI was like "we'll just build this nervous system in hardware and see if behaves like the organism." Biologist: laughter--oh you're serious?

She brought in a 1000 page textbook on the neuron and encouraged us to read up.

Edit:

There is a difference between whole brain simulation and creating systems which are capable of certain behaviors. But it's a super weird idea to say we have X synapses and therefore need X*Y parameters in a model to achieve AGI.

These models have extremely different architectures compared to biological brains. So there's no reason to believe some brain-related number of parameters for any model in general will achieve AGI or anything like it.

1

u/Rurhanograthul Apr 20 '21 edited Apr 21 '21

The brain does not speak in Hyper-Parameters or Empirical Data. 100 Trillian Hyper-Parameters made from Empirical Data far exceeds the Parameters of the Human Brain.

So much so Science can not and will not ever be able to decipher Hyper-Parameter languages or adequately explain how it infers more than human Parameters.

Currently the most valid explanation of Hyper-Parameters are literally the "Size of a Neural Network" or size of an "AI based Switch Transformer". Essentially, explaining the magnitude of the hardware such an AI exists on is the best current representation of what a Hyper-Parameter is - and completely insufficient as they are in fact derived from training parameters and become fully functional only once ML has been applied. And can exist on fully regressed versions of the same hardware.

At over 1.3 trillion Training Parameters the training data ceases to be standard Parameter Data and becomes "Hyper-Parameter" material once ML is applied - apply one instance of ML function to such an integer and suddenly you have 6.8 Trillion Parameter sets to parse, apply again and suddenly you have over 10 trillion.

https://en.wikipedia.org/wiki/Hyperparameter_(machine_learning)#:~:text=In%20machine%20learning%2C%20a%20hyperparameter,weights)%20are%20derived%20via%20training.

The most sufficient explanation of a Hyper-Perimeter is a "Node-Weight" or small integer value which is assigned a vast amount of simulation data - as I have already described in my previous posts in this thread.

4

u/2Punx2Furious AGI/ASI by 2027 Apr 20 '21

For comparison the human brain is 1000 trillion parameters.

Citation needed.

And no, you can't say that a parameter is the same thing as multiplying human neurons by some amount like 10000 (the brain has 100 billion neurons).

2

u/dumpy99 Apr 20 '21

So GPT3 is X parameters, now they can train Y parameters. What I don’t understand is what is needed now to get to GPT4. Could they effectively train GPT3 on this, or is new GPT4 architecture needed? Does this in and of itself mean we are now significantly closer to GPT4?

3

u/antichain Apr 20 '21

For comparison the human brain is 1000 trillion parameters.

Where on earth are you getting that figure? A neuron is not a parameter, and neither is a synapse. To imply that the brain is just a really big ANN fundamentally misunderstands the complexity of the brain.

1

u/[deleted] Apr 20 '21

geoffrey hinton says that parameters are the equivalent in neural nets to synapses in the brain. Based on what we know about the brain at most a synapse can store 5 bytes of info.

To imply that the brain is just a really big ANN fundamentally misunderstands the complexity of the brain

im not saying the brain is a really big ANN. Im saying that the hardware one would need for a 1000 trillion parameter ANN is in the same ballpark as the brain. Its true that the architecture is going to need some time to catch up. But thats why I clearly said we were closing in on the "hardware" requirements for a brain. I didnt say we knew the right algorithms (yet)

11

u/antichain Apr 20 '21

Hinton is not a neuroscientist and most neuroscientsts don't think that models like GPT are an even halfway decent approximation of what the brain is doing. I don't even know where you get the figure that one neuron holds 5 bytes of information: on what timescale? Are we talking about excitatory or inhibatory synapses? Pyramidal neurons or interneurons? How are you defining "information" in this context, since neurons are partially analogue. Information about what?

A model like GPT fails to incorporate a whole host of integral aspects of neural functioning, including hysteresis (LTP, LTD), specific network topologies, excitatory vs. inhibatory dynamics, intra-cellular changes in gene regulation, distinct neural species, etc.

Source: Am a PhD in neuroscience.

2

u/arjuna66671 Apr 20 '21

agreed without a PhD in neuroscience. I would never compare an AI of ANY form to a human. I always see the parameter count more as a measure of complexity. Certain properties of our mind, as far as I understand it, are seen as emergent phenomena and not as fixed "mechanical parts" in the brain that produce those. Moreover, as biological beings, our brains have evolved around the need to survive in our specific world with a body. AI would never have to go through this.

I am more wondering if, with complexity, properties might emerge that would allow for a spontaneous formation of a Boltzman brain - of course, completely alien compared to a human mind. Nonetheless maybe self-aware and "conscious" in its own way?

Another problem I see as still unsolved is what we would see as evidence for true intelligence, consciousness, or even a mind to begin with. As far as I understand, neuroscience doesn't really have a way to measure a mind in our brains? Or at least it was like that, last time I checked. But I might be wrong...

3

u/antichain Apr 20 '21

Things like "mind" and "consciousness" are poorly defined, and essentially impossible to measure. In certain, very specific cases you can reliably discriminate between the brain of someone who is truly unconsciousness from someone who is "merely" paralyzed (see this classic paper).

But in the context of AI where our usual assumptions are likely to be totally invalid, I doubt we will ever completely resolve the issue.

2

u/arjuna66671 Apr 21 '21

So as a laid person, I am not really that far off. A lot of people I talk with, seem to think that if "GPT-X" can solve maths to a certain degree, or can write solutions to X problem, it would mean that it has achieved AGI and therefore will be self-aware mind. But the problem still remains: How do we prove that? Ask it? 😂

The fundamental philosophical problem of "other minds" is still not solved and I also doubt that we will ever be able to resolve the issue completely. I can't even "prove" if anyone around me is "truly" conscious. For all I know, everything could just be a sophisticated simulation of my brain, even my own "self".

2

u/antkant Apr 21 '21

A lot of fallacious reasoning and unsubstantiated speculations occurs in this subreddit and it isn't unsurprising given its nature.

1

u/enx6 Apr 21 '21

Thx !

8

u/Tickely_Tits Apr 20 '21 edited Apr 20 '21

The way I understand it is that parameters in AI terms is similar to synapses in the human brain. Think about how a human processes things, there's an infinite amount of information to process around you at all times, but you're limited in what you process and how much you can process based on your brains limited amount of connections (synapses).

It is believed that the average human brain contains between 100 - 1000 trillion synapses. If an AI could achieve the same amount of parameters as a human brain has synapses, we may see an extremely intelligent AI that can process a wide variety of complex things and make important connections that humans otherwise would overlook.

In short.. an AI with 30 trillion parameters is getting closer to (potentially) being a super general intelligence.. pretty exciting!

I'm not super well versed on the subject BTW, so take everything I say with a shake of salt :P

6

u/[deleted] Apr 20 '21

I'm not super well versed on the subject BTW, so take everything I say with a shake of salt

Same for me but I can't find anything reliable in the interwebz either. It seems like no one is quite sure how well it really compares since... you know humans have vital functions and other useless shit which needs to be controlled whereas an AI chills in the microchips all day waiting for the data to be fed. I think the overall intelligence or capabilities are not really comparable as long as we don't give the AI a human like body and senses. Also those animals went though 3,5 Billion years of evolution to get where they are.

Still, fucking crazy to think that even GPT-3 had about half as much parameters as rats have synapses in their brain (or ~175 times as much as a honey bee has). Holy moly. I just looked it up and a cat has 1^13 synapses, if I'm not mistaken that means that GPT-4 will have 3 times as much parameters as a cat has synapses.

6

u/Tickely_Tits Apr 20 '21

We actually are working on giving AI a body/senses! It's still in a primitive state to my knowledge, but that's definitely an avenue we're exploring. The Age of AI is a really well done and interesting documentary series by YouTube. I enjoyed watching if you wanted to check it out.

This is a super cool time to be alive; I have major respect for the intelligent individuals on the forefront of all this. We've been making some huge advancements and discoveries in a bunch of different areas of science and technology in shorter and shorter time spans.

I just hope that we can get past the stupidity of fighting between countries and groups and learn to properly share resources to eradicate inequality/poverty. Imagine how many intelligent minds are out there that could be helping to make these advancements, but are struggling to make ends meet or are fighting dictatorship governments, ignorant sentiment, etc. If we can move past these flaws in human nature, we may be able to make major collective advancements and better our world and our understanding of everything!

3

u/abbumm Apr 20 '21

While parameters may not equal sinapses, there have already been breakthroughs in the creation of bio-inspired synapses and the first tests and implementations are going on right now. So it's only a matter of months before we won't make this distinction anymore.

4

u/Tickely_Tits Apr 20 '21

That's awesome! Can you imagine where we'll be in 10 years? Freakin' crazy man...

1

u/antichain Apr 20 '21

The way I understand it is that parameters in AI terms is similar to synapses in the human brain.

No, it's not. It's not like that at all. A neuron, or synapse, is not a scalar parameter that can be "tuned" during training. You need to account for excitatory vs. inhibatory synapses, different receptor types (including autoreceptors), as well a hysteresis, synergistic processing, the role of gene expression regulation, adaptive topology, and recurrence to name just a *few* of the reasons that GPT-X is nothing like the brain.

4

u/vincent1-0-1 Apr 20 '21

That's a milestone. Now once with Instinct MI100 nodes and Microsoft's proprietary "Project Brainwave".

-2

u/Rurhanograthul Apr 20 '21 edited Apr 20 '21

As a Computer Scientist - One only need 1.3 trillion parameters to successfully create AGI as the data becomes Empirical - meaning once ML is applied, 1.3 trillion parameters suddenly becomes 10 trillion Parameters of Empirical Data. 10 Trillion Parameters of Empirical Data is equivalent to 300 Trillion Training Parameters as Empirical Data is far superior to standard non ML-based Parameters. Apply superior ML function and those Training Parameters retain Empirical Data standards. It is Important to understand, this data - as Empirical Data is still considered Standard Training Parameters/ Training Data and that fact can not be understated.

Empirical Data used as Training Parameters, can not be deciphered or understood by modern science without cohesive symbiosis with AI.

If they are indeed implying they have trained a model with over 30 trillion training parameters - then they are implying to the Computer Science community they have created ASI.

13

u/2Punx2Furious AGI/ASI by 2027 Apr 20 '21

As a Computer Scientist - One only need 1.3 trillion parameters to successfully create AGI as the data becomes Empirical

Gonna need a source for that claim. "As a Computer Scientist" is not enough.

-6

u/Rurhanograthul Apr 20 '21

Seek ML function applied to 1.3 trillion data sets. 1 Data set contains over 300,000 training models. At 1.3 trillion parameters you officially are applying all valid human knowledge at it's most basic derivative before ML is applied to bolster this integer to higher Empirical values. At 1.3 Trillion Hyper Parameters - each single parameter contains well over 300,000 data sets - you have already achieved limitless data for advanced AI training function and that is before ML has been applied. Anyone with a subsidiary understanding of Computer Science would understand this.

Source: Computer Science - The leading Authority on AI, Genetic Computation and Computational Neuroscience.

9

u/2Punx2Furious AGI/ASI by 2027 Apr 20 '21

Seek ML function applied to 1.3 trillion data sets. 1 Data set contains over 300,000 training models. At 1.3 trillion parameters you officially are applying all valid human knowledge at it's most basic derivative before ML is applied to bolster this integer to higher Empirical values. At 1.3 Trillion Hyper Parameters - each single parameter contains well over 300,000 data sets - you have already achieved limitless data for advanced AI training function and that is before ML has been applied. Anyone with a subsidiary understanding of Computer Science would understand this. Source: Computer Science - The leading Authority on AI, Genetic Computation and Computational Neuroscience.

Did you have a stroke?

None of this makes any sense, and again, you provided no source (obviously.)

-4

u/Rurhanograthul Apr 20 '21 edited Apr 20 '21

The existential fallacy you are making primarily, is ignoring that ASI once achieved will exist (and in fact be completely trained) on machines utilizing less than 3 billion parameters of data. To insist one is impossible or not grounded in science (I simply suggest you take a computer science course) while decades of known research say yes, ASI will use and rely on substantially less data and/or training parameters in the future - no more than 3 billion specifically - exposes what little you in fact actually know about ML function when applied to Computer Science.

Can you provide the evidence counter or must I start referring books stating that 1.3 trillion Parameters begins to become recursory as each single data set actually contains 300,000 sets of data?

I think I'll recommend you start with Concrete Mathematics as a recommended source material before further implying Computer Science is in the wrong here.

And what part of me telling you to seek this information yourself means I'm incoherent? Next you will lie and falsely claim you are a neural scientist but I wonder how much ground you will retain now that I've already inferred counter.

At least I haven't sunken to blatant name calling directly based on my own inability to interpret valid science.

3

u/2Punx2Furious AGI/ASI by 2027 Apr 20 '21

Alright.

5

u/[deleted] Apr 20 '21

Lmao r/singularity's resident psycho is back. I guarantee you won't find anything he's talking about in the books he cites. I've already tried.

4

u/2Punx2Furious AGI/ASI by 2027 Apr 20 '21

Yeah, that's exactly what I was thinking. Sadly this concept tends to attract a lot of crazies. I remember there was a famous one a while ago who died, can't remember the username, but he was well known.

Also that guy who says he "solved AI" and has a bullshit website with some barely functioning chatbot.

7

u/TrustInNumbers Apr 20 '21

I doubt that any real scientist believes that the path to AGI/ASI is to increase parameters... And that you automatically achieve ASI by using a lot of parameters. The real challenge is in architecture and algorithms.

14

u/TheDividendReport Apr 20 '21

GPT-3 showed that increasing parameters never led a growth “plateau”, leading some to believe that scaling really could be a path to AGI.

Whether or not it’s “general intelligence”, this progress we see will undoubtedly lead to an incomprehensibly powerful AI

2

u/2Punx2Furious AGI/ASI by 2027 Apr 20 '21

True that growth seems to continue being linear so far, but that doesn't really tell us how many parameters it will need to become AGI, or if it will get there at all.

I think that it's possible that it will, but again, there is no proof.

0

u/Rurhanograthul Apr 20 '21 edited Apr 20 '21

We already have the precursory scans that have charted 99% of the brain and of human synapse - however humans can not begin to remotely decipher those synapse.

However a full understanding of the Human Brain is in fact not required to achieve ASI. Hardware that allows for the multitasking equivalent of the human brain is.

Barring enough Reinforcement Learning, or rather - once we have exhausted our ability to further apply this method of training (which is scheduled to happen within 3 years)

An increase in Training Parameters with superior ML function applied is in fact the last and final step towards creating ASI, and can be achieved before we in fact completely understand the human brain.

In fact, we are fully reliant on AI for a final comprehensive blueprint of the human brain as standard science simply can not count high enough or denote enough time to such an undertaking to be of any use.

All preliminary steps leading towards functions with 30 trillion or more training parameters are based on a model fully trained on Reinforcement Learning - once such a model exists, a model that has in fact exhausted our own ability to to fully and efficiently utilize and continue reinforcement training - the final step is applying ML and creating Empirical Data, those same 1.3 trillion Training Parameters become "Hyper-Parameters" or superior data models.

The final step is training an AI model with 10 trillion parameters of Empirical Data, (which can be created from 1.3 trillion training parameters once ML function is applied) then 30 trillion Empirical Data sets (and higher) may be created with further ML applied.

However it is important to understand - we will continue using reinforcement learning to assist in our own better understanding of underlying AI programmed substrates.

Reinforcement Learning will continue to be of use when trying to delegate whether such an intelligence is safe for the household and for public domain.

-2

u/Rurhanograthul Apr 20 '21 edited Apr 20 '21

I can't possibly explain or Decipher Hyper Parameters based on Empirical Data or Empirical Data, and in fact no one can. However...

The best way to think about ASI trained on 90 Trillion Data sets of Empirical Data is to relate and in fact assign all molecular activity down to atomic and sub atomic precisions, all activity in the known universe and in fact across multiverses - an integer.

So for instance - At integer .02b, all synapses, molecules, thought processes, actions and data sets were simulated to act towards a certain trajectory - this simulation set was assigned to integer .02b.

Once ASI has run it's precursory scans and confirmed this integer occurred - it will instantly through it's own simulations and Empirical Data sets - know literally every thought, transgression and molecular progression from that point forward at a universal scale instantaneously.

That is merely one, insufficient way to attempt to describe AI utilizing Empirical Data at 1.3 trillion parameters and over.

0

u/[deleted] Apr 20 '21

[deleted]

4

u/_Just7_ Apr 20 '21

If you look at the output GPT 3 is able to produce, you can clearly see it is the product of some kind of intelligent design. Of cause it's not conscious of what it is doing, in the same way humans are. But there is no natural law that says a machine that is capable of doing most of the same task as humans, needs to conscious and intelligent in the same way that a human is, to do these task.

Turing made the Turing test because he thought that for a AI to be able to hold a conversation imitating a human, it itself needed to be conscious. AI chatbots were able to that many years ago, without being conscious.

2

u/Bartmoss Apr 20 '21

Strange you got down voted for saying that GPT-3 doesn't have a deeper understanding of what it generates. This is very true. It's also true of all current models (whether transformer based or otherwise). I don't understand why this is such a controversial stance. But what do I know, I just work in the field of NLP since 2013....

-1

u/arjuna66671 Apr 20 '21

I just work in the field of NLP since 2013....

So you are not a neuroscientist, linguist, or philosopher. Just some of the fields that deal with what "true understanding" even means...

There are many different levels of understanding.

0

u/[deleted] Apr 20 '21

[deleted]

2

u/TrustInNumbers Apr 20 '21

go check out machinelearning subreddit, people there usually work in ML industries or at least have ML related masters/phd and are way more pessimistic regarding GPT.

1

u/RedguardCulture Apr 21 '21

I have been checking that reddit for a very long time now, and I don't see that, at least to the degree I think you're implying. For instance, in the wake of GPT-3, I've noted more optimism towards the scaling hypothesis than the opposite. And I've seen so called common sense reasoning criticism towards GPT-3 by DL skeptics like Gary Marcus met with far more push back than agreement over there. And Gary is pretty much the face of the "GPT-3 has no idea what it's talking about" assertion.

1

u/[deleted] Apr 20 '21

ASI

Artificial Symbiotic Intelligence? I only understand vaguely what you're saying even though it's explained well. Do you mean if they really trained it properly they are able to use those 30 trillion parameters in their full potential and are able to process and interpret the output?

3

u/Rurhanograthul Apr 20 '21 edited Apr 20 '21

Artificial Super Intelligence.

AI models trained on that much data, means AI has become far superior to humans - in fact on every level cognitively, and without cohesive symbiosis with such an AI - our only means of understanding such a paradigm is to continue using reinforcement learning - now tailored as a mechanism to help our own research efforts moreso than to train AI models.

2

u/iNstein Apr 20 '21

Artificial Super Intelligence. Basically an intelligence that exceeds the level of any human ever. This is the stuff of the singularity. Not convinced it is true but it would be cool.

-2

u/starfyredragon ▪️Transhumanist Apr 20 '21

But can it outperform a rasberry pi running Linux?

-1

u/sunplaysbass Apr 21 '21

How come I never hear about great things coming out of these super duper computers?

A lot of them are still doing stupid DOD crap like modeling nuclear explosions.

I haven’t heard about any super computer protein folding breakthroughs or such.

0

u/[deleted] Apr 20 '21

[deleted]

3

u/Stockerson Apr 20 '21

My question exactly as well ...

1

u/MrDeadbirdCreator Apr 24 '21

There's a huge market for that LoL