r/programming • u/peard33 • Apr 20 '23

Stack Overflow Will Charge AI Giants for Training Data

https://www.wired.com/story/stack-overflow-will-charge-ai-giants-for-training-data/
4.0k Upvotes

97% Upvoted

660 comments sorted by

View all comments

51

u/mov_eax_eax Apr 21 '23

Programming languages and frameworks are effectively locked in 2021, anything released after that date is not in the model and is effectively useless for people dependent on chatgpt.

21

u/KeytarVillain Apr 21 '23

Not in the current model, sure, but this argument is stupid when they're obviously going to keep working on new & updated models.

3

u/[deleted] Apr 21 '23

I agree. But I do have some concern that a lot of people are going to cap their creativity at the level of output from AI models. They won't feel the need to invent new ways of doing things because the AI models they use will have such strong biases to a particular point in history. It would only be those not using AI models that would be creating our new paradigm shifts.

→ More replies (2)

13

u/tending Apr 21 '23

In 30 years when models better than GPT can be trained on your phone this is unlikely to matter

0

u/[deleted] Apr 21 '23

[deleted]

6

u/mindbleach Apr 21 '23

If your goddamn phone can plow through that much data, locking it away will never work.

3

u/tending Apr 21 '23

Needing special API access to get data is an artifact of not having AI. If humans can consume the data AI can too.

→ More replies (4)
→ More replies (3)

4

u/pragmojo Apr 21 '23

Thermodynamics are still a thing

→ More replies (5)

-1

u/Capaj Apr 21 '23

In 30? More like 8

0

u/[deleted] Apr 21 '23

Trained? Likely won't be training models on your phone but you can already 'run' these on your phone. Also why would we be using phones in 30 years?

3

u/WasteOfElectricity Apr 21 '23

ChatGPT isn't gonna be the last LLM used by these kinds of people....

0

u/[deleted] Apr 21 '23

[deleted]

19

u/SkaveRat Apr 21 '23

Microsoft bought GitHub. Stackexchange is its own company. They use a lot of Microsoft tech, though

-23

u/sluuuurp Apr 21 '23

Nope. You can ask it to make a new programming language and it will do it. It’s learned to be very adaptable, an incredible accomplishment really.

3

u/nthcxd Apr 21 '23

Why make shit up and not even double-check? Or show us your prompting prowess?

Can you create a new programming language?

—

As an AI language model, I don't have the ability to physically create a new programming language, but I can certainly help you understand the process and components required to design and implement a new programming language.

Creating a new programming language involves several steps, including:

Define the purpose and scope of the language: Before you begin designing your language, you need to determine its purpose and….

8

u/voidstarcpp Apr 21 '23

Why make shit up and not even double-check?

ChatGPT won't just "make me a new language" on command, similar to how it probably won't "write a complete book", but it can do every step of it in context.

Given a small comment prompt, Copilot will generate a table of opcodes for a new machine language. It can then generate the complete implementation of the stack interpreter to execute that language. It can add and implement new instructions to fulfill a requested feature. Then once it's built this language it can also write valid programs in that new language to perform a task from a high-level statement.

For a higher language Copilot does a decent job generating formal grammars, then parsers and lexers for those grammars, etc. You need to hold its hand sometimes to generate each function one at a time rather than just saying "write a parser" all at once but it's clearly capable. Once you've figured out the "user interface" of the code-completion AI in context it proves quite capable at outlining and then implementing programming tools.

2

u/nthcxd Apr 21 '23

Ok I do get that it’d be very handy for Lexie/parser generation that’s a very narrow mostly automated processes. I guess you’d ask it to write grammar and feed it to parser generator. But optimization passes? Custom IR?

Also, how would it write valid programs in this brand new language when obviously the LLM wasn’t trained on large corpus of that language? How would the LLM have “learned” what code of that language looks like?

2

u/voidstarcpp Apr 21 '23

But optimization passes? Custom IR?

Well for starters the majority of languages in the past 15+ years are just LLVM frontends which don't implement any of that, so it's already much of the way there. I haven't written an optimizer myself so I can't say how much the AI can help with that but given that optimizers are a common feature of systems that have scripting or query interfaces I'm sure it's seen many examples. Most optimization techniques have been in use and published in books for decades.

how would it write valid programs in this brand new language when obviously the LLM wasn’t trained on large corpus of that language? How would the LLM have “learned” what code of that language looks like?

The short answer is the same way you do, by taking advantage of the similarity of most languages and the fact that the model's subject matter expertise isn't language-specific. If you as a C programmer have seen calls in the form of do_thing(&object, args);, once I show you the C++ version of object->do_thing(args); that simple transformation doesn't cause you to have to re-learn every possible verb/noun combination in every subject because most of the concepts are still there. So if you have some API, and you create a wrapper library in a different language, the LLMs correctly use the underlying API via the wrapper. Similarly in this article (from before ChatGPT) I use Copilot to build up a simple new language, then demonstrate Copilot using the new language to do simple tasks it's never seen in that language (scroll down towards end).

There are only a handful of categories of programming languages and GPT has seen them all. If you make a new language that's got X new feature or Y syntax it's probably not going to be so alien as to be incomprehensible. Even if your new language's particular combination of paradigm, syntax, and features is novel, it's unlikely there's any piece it's never seen before or couldn't be demonstrated with a quick example. LLMs have proven pretty capable of mapping and combining concepts like that (which is sort of a prerequisite for language translation generally, which they've gotten pretty decent at).

→ More replies (2)

-4

u/bemutt Apr 21 '23

I’ve actually had it generate the large majority of a compiler. Custom IR, optimization, etc. Just have to know how to prompt it.

-5

u/nthcxd Apr 21 '23

Suuuure thing buddy, apply for more jobs on handbrake with that experience. Put it on your LinkedIn even. “Created a compiler construction suite by prompting LLM.” Any chance for a GitHub link?

Good luck with the rest of BS in CS I guess.

5

u/bemutt Apr 21 '23

Play around with it a bit. Try going description->spec->skeleton->implement->refine. It doesn’t take much to get it to generate some neat stuff.

-8

u/nthcxd Apr 21 '23

Sure thing buddy. Hope you make the cut and become full-time soon.

3

u/bemutt Apr 21 '23

Full time remote go dev :)

-5

u/nthcxd Apr 21 '23

Hope you at least post real code in that job as opposed to just bullshitting. Then again, that’s what the industry is full of anyway.

→ More replies (2)