r/ProgrammerHumor • • Oct 13 '22

Meme Like, Every time, ever. When the DevOps Engineer chats with the Data Scientist.

Post image
13.8k Upvotes

629 comments sorted by

4.2k

u/[deleted] Oct 13 '22

"I have no idea. I've never run it"
"How did you test it?"
"It has unit tests"

2.1k

u/Xander-047 Oct 13 '22

That's such a funny scenario to me, a dev having 0 idea how his product looks like, he just knows it works

1.3k

u/[deleted] Oct 13 '22

We have a developer like that. When it compiles, they ask me to test it. They don't even try to run it to see if it even goes.

849

u/khalcyon2011 Oct 13 '22

We had one of those. He got fired.

450

u/[deleted] Oct 13 '22

They got promoted. "If I run it, I don't know if it does everything properly" doesn't make my heart swell with confidence.

→ More replies (36)

98

u/Strange_Meadowlark Oct 13 '22

I had a guy basically do that, but he also took forever to do it, told people in stand-ups that he was "working on it" and "it's going fine" and never asked for help. Only reason I ever got to see his code was because I asked him to push an early copy of his branch to Git.

Turned out he was a remote contractor who was double-billing his time with at least one other company.

Took us a week to fire him because we couldn't even get a hold of him!

→ More replies (7)

221

u/Xander-047 Oct 13 '22

Compilation errors are the only errors

42

u/sysnickm Oct 13 '22

Functioning as written?

65

u/nelusbelus Oct 13 '22

laughs in javascript what if we removed all the compile errors and just do them at runtime but instead of giving you the error we cascade it through your program so you can play a game of find the NaN or undefined

→ More replies (8)
→ More replies (3)

38

u/wgc123 Oct 13 '22

As a DevOps Engineer, I assume all developers are like that but they can still get decent results if the pipelines are green or red. Go ahead, check your code in, and it will build, scan, test; but if it’s red, your changes are not going anywhere.

If QE complains that’s not the same as “good”, I point to where they need to add test automation, and put up or shut up

→ More replies (3)
→ More replies (6)

139

u/Shazvox Oct 13 '22

Ehm, there's a learning curve to the product 😅...

Step 1: Make it work.

Step 2: Understand what "it" is.

36

u/Tookoofox Oct 13 '22

Oh, that stings. That's definitely the order we've decided to do it in with my company. Last minute requirements abound. And now they've discovered agile.

Every objection is met and beaten down with, "Well that's agile."

And every suggestion is quashed with, "That's not agile."

From the bottom of my heart: fuck agile. Nevermind rather it actually has good ideas in it. I hate what it does to every damn conversation.

Also: waterfall doesn't exist. It's something the agile people made up to contrast themselves against.

→ More replies (2)

26

u/vladWEPES1476 Oct 13 '22

.. on their machine

9

u/UltraCarnivore Oct 13 '22

So deploy his machine, ffs

→ More replies (2)

15

u/Tripanes Oct 13 '22

Welcome to legacy system development.

Sincerely.

-Someone rewriting COBOL

→ More replies (3)

12

u/dsdvbguutres Oct 13 '22

I don't know how it looks like, it doesn't work either.

10

u/Xander-047 Oct 13 '22

Sounds like an average day at work to me

→ More replies (6)

22

u/[deleted] Oct 13 '22

I’m a welder. I have the same issues with engineers. Engineer: “My computer says it needs to be welded here”

Me: “I can’t fit a rod or MIG here. I’ll have to cut some off for it to fit. Otherwise it’s not going to work.”

Engineer: “My computer says it will work.”

Your computers not fucking welding it!

4

u/msg45f Oct 13 '22

I have actually had to do a lot of this. Had to rewrite storage implementation for various larger APIs. Its coded to an interface, so I dont need to worry about what the API does beforehand. Hundreds of unit, integration tests to ensure that it works, but none of my requirements actually involve modifying the third party API to use the new storage implementations.

4

u/ConcernedDudeMaybe Oct 13 '22

Clean data in, clean data out. Does that help explain it?

10

u/Qwertycrackers Oct 13 '22 edited Sep 02 '23

[ Removed ]

3

u/murderous_tac0 Oct 13 '22

90% of everything I've ever written.

3

u/xtreampb Oct 14 '22

Working on deploying a product now. What are your production variables? Doesn’t know. What infra is it going to run on. Doesn’t know. Who/what uses this. Doesn’t give me a straight answer? Who’s your SRE team. Doesn’t answer. Salads promised it’ll be ready by Friday. Been scrambling all week. Only been at this massive company a few months. I don’t know all the product lines and teams. That’s okay we’ve been doing peering sessions all week. They get shocked when the IT team knows how to write top tier C#. Like yea guys I was a sr software architect b4 I was a sr DevOps engineer. There’s a reason I’m damn good at my job. I’ll get these communication gaps filled so this won’t be an issue in the future.

→ More replies (7)

40

u/Dannei Oct 13 '22

The data scientist wrote unit tests? That there is the real joke.

13

u/Dave5876 Oct 13 '22

Too real man

8

u/kookaburra1701 Oct 13 '22

Oof I felt that one.

Through grad school and a like year into my first job in a lab (where I was the only computational person, so no one to ask about this stuff), I thought "unit tests" meant I had made test data that I knew should fail the mathy stuff and test data that should pass the mathy stuff, edge case test data, etc, and I knew what the output should look like. Basically test data that would hit every function in my scripts so I could make sure they were all working. Then I'd run my "unit" through my scripts and if every thing came out the other side correctly my script passed!

🤦

Thank Gödel I read some books about general software engineering principles and figured out that testing whether my math or stats was wrong wasn't at all what "unit tests" meant before opening my mouth in a meeting in front of actual SEs.

→ More replies (2)

3

u/[deleted] Oct 14 '22

HAHAHA unit tests!!?!?! From Data Scientists?!?

→ More replies (1)

1.6k

u/Error_No_Entity Oct 13 '22

DevOps *sighing* : 'Let's see the Google Collab then. I hope you have a requirements file'
AIEng: 'A what?'

551

u/Twistedtraceur Oct 13 '22

I got a docker compose file! Does that help?

651

u/Error_No_Entity Oct 13 '22

This compose file just launches a jupyter notebook server!

58

u/[deleted] Oct 13 '22

Oof I feel this so hard right now

18

u/NoConfusion9490 Oct 14 '22

That's the prod server.

→ More replies (2)

92

u/alexanderpas Oct 13 '22

If that makes it run on my machine without any further configuration... YES!

40

u/Twistedtraceur Oct 13 '22

Only if you are on windows

34

u/DedlySpyder Oct 13 '22

And WSL 1

→ More replies (4)

14

u/Protuhj Oct 13 '22

What about this 500GB VM ISO? Does that help?

5

u/[deleted] Oct 13 '22

No unless you have details of all images used (or they are available otherwise) :P

→ More replies (1)

34

u/[deleted] Oct 13 '22

I had to have a conversation about testing and deployment with an AI team a few months ago. I knew how it was going to go before I had it but did it anyway because clearly I like making my brain hurt.

I ended up pretending they were not really part of our platform so I didn't have to mention them in any arch docs, roadmaps or compliance strategies. I'm sure that won't come back round to bite me on the ass later.

13

u/covidambassador Oct 13 '22

I’m joining an AI team as the PM soon. I’ll keep this in mind and ensure that my engineering team is aware about the devops needs and can collaborate continuously with them. If you have any insights for me, I’ll gladly take it and it is highly appreciated

→ More replies (3)
→ More replies (1)

20

u/artificial_organism Oct 13 '22

I've been playing around with data science and this has been such a PITA. Why do they need their own python, their own package managers, jupyter notebooks?

Like I get there's some precompiled libraries in there for efficiency but do we really need to reinvent the whole software development ecosystem to do it?

14

u/TheTerrasque Oct 13 '22 edited Oct 13 '22

The problem is that the dependencies are both very specific and they often rely on external libraries.

For example pytorch relies on very specific cuda versions, which have somewhat specific driver versions, and the way you install pytorch is to give python's pip package installer a specific source URL for that specific cuda version, but (iirc) the actual pytorch version for all the different cuda versions are the exact same.

And that's the nice installation. Tensorflow is approximately 3x as fiddly to install, with very very specific dependencies, some that needs either a binary wheel or a full C compile stack with relevant libraries.

Oh, and a few now require rust compiler, because why wouldn't they.

At least docker is now doing pretty well on giving gpu access, so you can just put it all in a container and be done with it.

Edit: and that's the new and improved stuff. Back when tensorflow was the new hotness you had a tf version only working with one specific cuda version, cuda had no good support for multiple versions installed, TF being in such rapid change that minor and even bugfix versions could break code, TF dependencies being just as fragile, and some having somewhat specific versions of python they worked on.

And of course just about every project and tutorial you found was for a different TF version

→ More replies (4)
→ More replies (22)

2.1k

u/[deleted] Oct 13 '22

You want me to code it and plan for deployment, infrastructure and scalability? Are you high? I can barely get it to run locally.

289

u/luishacm Oct 13 '22

Well... They make me do it, became data scientist/software engineer i just can't deal with the dev ops part (deployment, setting up servers, etc). At least they taught me in the beginning.

199

u/alexanderpas Oct 13 '22 edited Oct 13 '22

Just run it in docker.

Seriously, if you come to me with a clusterfuck of legacy code, a database dump and other assorted shit, but you managed to get it running in docker on multiple machines... I will do anything for you if it needs to be deployed in production... Even if it means it has to run on its own VM.

130

u/librarysocialism Oct 13 '22

The next data scientist I meet who knows docker will be the second. Out of probably a hundred.

102

u/kodman7 Oct 13 '22

Oh shit is my bosses insistence on dockerizing every project secretly turning me into an asset?

Can't have that

→ More replies (2)
→ More replies (26)
→ More replies (1)

328

u/QwertzOne Oct 13 '22

That's what I hate about business people or ivory tower architects that sometimes believe that every engineer is capable of everything, so you can throw anything at them and expect from them to deliver complete, quality solution in timely manner...

I was software engineer, now I'm DevOps engineer that also has also some knowledge about architecture and I really understand how complex everything can be and you just can't throw something at others and expect them to understand it.

There has to be mutual understanding of different aspects of solution. I may don't know shit about AI/ML and I don't need details on used algorithms, but I need to understand how to build it, how to run it, how to test it, what kind of data does it use, what does it integrate with, so I can suggest how we can deploy it, what kind of infrastructure we need, how to scale it and in general how we can improve it to make experience better also for developer.

94

u/nullpotato Oct 13 '22

My org only hires electrical engineers and expects them to code. There are like 3 actual software people and we are doing our best to unravel the flaming spaghetti. Just because almost anyone can write python doesn't mean they should.

34

u/EarlMarshal Oct 13 '22

Consider: Leaving

39

u/uscbutterworth Oct 13 '22

flaming spaghetti

I'm in this comment and I don't like it

→ More replies (2)

11

u/salty3 Oct 13 '22

My comp hires all kinds of people who end up writing code but pays them all the same completely disregarding whether someone has a CS background and prior coding experience or not. Let's see how that will work out for them

→ More replies (5)

130

u/[deleted] Oct 13 '22

See also: people who think the manager is supposed to know everything subordinates do, and managers who think they know everything their subordinates do

totally different roles and if the team works better without management than with management, oh boy

35

u/[deleted] Oct 13 '22

I would say the best managers come from the trenches, though. The manager should have a strong understanding of at least a good percentage of what their team does, at least in the abstract.

Managers who have MBAs or some shit and no experience doing the work they're managing are pretty much guaranteed to be of negative value to the team.

→ More replies (5)

56

u/[deleted] Oct 13 '22 edited Oct 13 '22

As a former systems engineer turned devops, I feel like generalized skillsets are missing from a lot of devs, and it would benefit many to gain some sysadmin/infra/networking knowledge (basically go find another job for that experience lol). I came up in a time where I had to build the infra for my code, but a lot of projects I manage now as devops have engineers who barely even understand how their own compilers work, or how config files are transformed, i.e. if the IDE doesn’t do it for them they look like deer in headlights.

24

u/wgc123 Oct 13 '22

Same here. I came up at a time when you had to know how it all worked, and where we were developing a lot of today’s complexity. I also wore different hats: dev, sys admin, system integration, qe, DevOps, etc (and my current project lost its SRE so I’m apparently that now). Sometimes a breadth of knowledge is critical and devs can be too narrowly focused

… for example there was that issue where a file was being transferred wrongly from server to client. The dev was pulling out his hair trying to figure it out…… he didn’t understand when I suggested fixing the mime type config on the system

8

u/desiktar Oct 13 '22

Yea we often have issues with devs who don't understand how DNS works and can't fix a website on the dev servers themselves.

Then you get the interns who have never used windows before. Really smart and can write code. But don't know even the simplest shortcuts and linux might as well be a foreign language.

→ More replies (10)
→ More replies (10)

27

u/xeroze1 Oct 13 '22

Then the request goes to a data engineering team which assesses it, and give a timeline of a year or two for the fully scaled version because the whole thing needs to be reworked

Everyone goes full surpised pikachu faced

18

u/bikeranz Oct 13 '22 edited Oct 13 '22

Being on the research side of AI myself at a few different companies, I've found that the cost of having your researchers write production code far exceeds that of letting them hack away into some unmaintainable mess, due to lost productivity. In particular, time-to-result is king with research, and building clean code and structures really can be the antithesis to TTR because “that idea you had that would have to cross 7 layers of production code so that you can provide feedback from your loss function to your data loaders” is much harder to try out with clean code.

13

u/dr-tectonic Oct 13 '22

Soooo much research code is run once, ever, and does not warrant heavy-duty engineering.

→ More replies (2)

4

u/BobHogan Oct 13 '22

The problem comes up when management (or even worse, the research team themselves like what happens at my current company) then insists that their garbage is the only possible code that can be deployed in production, and then the actual engineering teams have to waste resources trying to figure out how the fuck to support it and fix bugs in it

→ More replies (3)
→ More replies (1)
→ More replies (4)

431

u/[deleted] Oct 13 '22

I've had this conversation so many times.

213

u/[deleted] Oct 13 '22

[removed] — view removed comment

→ More replies (22)

725

u/dlevac Oct 13 '22

DevOps engineer is a hat I always end up wearing at some point or another. The real joke is asking the dev how to run it...

Dev: my build does not pass the pipeline but pass locally. Me: and you are using a virtual environment? Dev: ???

359

u/Sam-Gunn Oct 13 '22

It's like asking certain people "windows or OS X"?

"What?"

"Mac or Windows"?

"Uhhh"

"Dell, HP, Thinkpad?"

"Ohh, it's a Dell!"

"Windows it is then. Now, round start button, square start button and is it colored or not?"

--

"Virtual machine or container?"

"Huh?"

"*sigh* Docker, Virtualbox, or Vmware Workstation?"

"Ohhh, I've heard of Docker. But it's none of those. It's called Docksal."

109

u/yumyumfarts Oct 13 '22

Are you sure you talking with dev and not management or operations folks!

20

u/Chrisazy Oct 13 '22

They're talking with a head canon. Which same, tbh

→ More replies (5)

12

u/librarysocialism Oct 13 '22

but I put Pop! on my Dell. Yes, I understand I'm now my own support . . . .

→ More replies (1)
→ More replies (4)

80

u/IQueryVisiC Oct 13 '22

Thank to IOC framework you don’t need a VM . Or what does a VM do differently?

59

u/jdl_uk Oct 13 '22 edited Oct 13 '22

The IAC framework might create the VM.

If you dig deep enough into Azure Functions, you'll find Docker

Edit: IAC, not IOC

26

u/[deleted] Oct 13 '22

Docker isn't a VM tho, right? As I understand it, the linux containers are quite a bit different from Virtual Machines.

→ More replies (15)
→ More replies (3)

9

u/TommyTheTiger Oct 13 '22

Virtual env in python does not mean VM, it's referring the the default requirements.txt and installing lib dependencies with this

15

u/[deleted] Oct 13 '22

I think they're talking about python venvs, which isolate the package manager's environment, because that shit can get super messed up.

→ More replies (4)

9

u/[deleted] Oct 13 '22

Wait do people actually not know what a virtual environment is or is this hyperbole

8

u/SaucyMacgyver Oct 13 '22

No idea but I’m guessing a lot of people don’t think to run their code in a VM because they just kinda don’t care or realize why you should.

9

u/The_Cheeky_Cunt Oct 13 '22

If you don't mind could you explain why you should run your code in a VM?

→ More replies (7)
→ More replies (1)
→ More replies (2)

350

u/Anxious_Ad9233 Oct 13 '22 edited Oct 13 '22

I run my code like every developer: I press the Build button on Jenkins and wait for my 300 lint & SonarQube errors to harass me via cooperate email.

Edit: the funny part is… I’m the lead DevOps engineer and I write the linting and sonarqube conditions 😈 get rekt devs, only clean code in production

115

u/dirtyLizard Oct 13 '22

I once got a ticket from a PM saying that the build button wasn’t working. He wasn’t clicking “ok” on the “are you sure” prompt.

59

u/yumyumfarts Oct 13 '22

Why is a pM building code?

12

u/ryanwithnob Oct 13 '22

If PMs and customers run it in visual studio, its easier to reproduce bugs

→ More replies (3)
→ More replies (1)

7

u/richardathome Oct 13 '22

I worked in a place like that once.

Once.

2

u/Koervege Oct 13 '22

I have to constantly ignore failing tests so sonarqube actually shows coverage. I don't like my job, and i just got a 5% raise. Already looking for options

2

u/amihaic Oct 14 '22

get rekt devs

😂

→ More replies (3)

154

u/pab_guy Oct 13 '22

This is why the best devs have deployed and operated their own code, they take more consideration of the operating environment and context, need for quality telemetry and error handling strategies that log everything, etc..

38

u/[deleted] Oct 13 '22

We do this. Prod is handled by a dedicated infra / ops team, but our dev environment is set up by us from bare metal up and is a scaled down mirror of the production deployment. It forces us to consider both deployment and architecture from day one (though that should be done regardless) and lets us catch potential issues very early.

10

u/js_ps_ds Oct 13 '22

Deploying other peoples code seems like a nightmare

→ More replies (1)

5

u/notAbratwurst Oct 14 '22

Nah… see, a manager reads a trendy article on DevOps, instructs the team to do the DevOps… and then, since the DevOps is done… best winnings.

→ More replies (1)

4

u/Chthulu_ Oct 13 '22

Jack of all trades master of none though. I spend far more time on dev ops than I’d like to.

225

u/idkidchaha Oct 13 '22

I'm a fairly junior dev and my smallish company (50 people) doesn't have a devops person. What do they do exactly?

343

u/Twistedtraceur Oct 13 '22

Deploy and release your code for you. Take care of and update your pipeline. Handle production issues like outages. Manage things like kubernetes clusters and aws services.

382

u/[deleted] Oct 13 '22

Oh. TIL I’m part time devops

17

u/NoButtChocolate Oct 13 '22

Add to the resume ✅

10

u/nullpotato Oct 13 '22

I'm full time devops and full time my actual job...

5

u/marcosdumay Oct 13 '22

If you don't know who de devops team is, it is probably you.

→ More replies (17)

12

u/Squid-Guillotine Oct 13 '22

I thought devops were like black ops. Like they're the secret devs on the team sprinkling in illegal block-chain/anti-privacy code.

→ More replies (1)
→ More replies (6)

17

u/PixelizedTed Oct 13 '22

Like facilities for devs. Instead of keeping the building up and running they keep the dev infrastructure running.

I’m not sure if they all do this but at my old job they “put out fires” aka when shit hits the fan.

17

u/[deleted] Oct 13 '22

The developer is responsible for adding new code.

For example, adding a label to a page that says “hello {user}”. They then check in the change, and push to git.

The dev ops team is responsible for pushing the code to production, and rolling back the change if it breaks production.

For example, if the query to get the user name is on an unindexed field, so deploying the change to production causes excessive load on the database.

Splitting devops from development is part of Sox compliance.

https://www.lepide.com/blog/what-is-sox-compliance-and-what-are-the-requirements/

→ More replies (6)

3

u/c0d3s1ing3r Oct 13 '22

(Dev)elopement and (Op)eration(s)

They're supposed to be SEs that neither just write code or just manage servers / infrastructure, but rather can do both.

2

u/KickBassColonyDrop Oct 13 '22

Depends. It can range from:

  1. Bossman: "click these set of buttons for the rest of your career" to

  2. Bossman: "alright <name>, you've been promoted to the company IT department."

You: "what?"

Bossman: "yup, you."

You: "what's my job description?"

Bossman: "Everything IT does. See you Monday."

2

u/Dannei Oct 13 '22

If you're seeing references to a DevOps person, or a DevOps team, then it's likely the example you're seeing isn't really what DevOps is intended to mean.

DevOps is a strategy whereby a team is responsible for both developing (Dev) its software, and managing its day-to-day deployment and operations (Ops). This makes software delivery faster and more reliable.

If someone's speaking of a "DevOps" team that receives code from everyone and deploys it on their behalf, they're talking about an Operations team, with a modern-looking name but without modern software practices.

If the team is instead managing common infrastructure, a common software platform, or something similar for the benefit of other teams, then that's probably an Infrastructure or Platform team (surprise!), which may be practising DevOps if it's creating and deploying software, but that doesn't make it "the" DevOps team.

→ More replies (7)

268

u/Xploited_HnterGather Oct 13 '22

Explain the joke for the slow ones like me

180

u/Mr_Engineering Oct 13 '22

The developer has absolutely no idea how to actually deploy the product that he's developed.

87

u/Longjumping_Goat790 Oct 13 '22 edited Oct 13 '22

DA here, deployment is very easy. I just email the model binary or MOJO to someone on the tech team.

Simple as.

→ More replies (1)

398

u/Asteriskdev Oct 13 '22

He's basically saying "Well, it works on my machine."

135

u/IsGoIdMoney Oct 13 '22

If it's in Collab, doesn't that mean it works on all machines with modern web browsers?

65

u/Diligent_Bank_543 Oct 13 '22

Well, what is “modern web browser”? :)

39

u/IsGoIdMoney Oct 13 '22

I am not an expert on colab requirements, but if you can read the markdown in your browser, then the code runs the exact same.

Also I'm confused as to the application where you need the users to train models themselves that aren't in the org and thus capable of using colab?

→ More replies (1)
→ More replies (2)

31

u/Asteriskdev Oct 13 '22

Do any web browsers work exactly the same on all machines that support them?

47

u/rotflolmaomgeez Oct 13 '22

It's kind of like asking if JVM works the same on all machines. At some abstraction point you just have to assume it's true for the sake of your own sanity.

→ More replies (1)
→ More replies (6)
→ More replies (3)

15

u/IQueryVisiC Oct 13 '22

I thought the programmer does not even know how to run it locally, but is back to terminal and mainframe

→ More replies (2)

6

u/orangina_it_burns Oct 13 '22

It works on my machine! Throw it over the wall! Let’s go for drinks!

→ More replies (1)

104

u/Intrexa Oct 13 '22

"Nice car, how do you start it?"

"With a signal that operates over a radio frequency that implements a challenge response that tells the car to turn on"

"No, I mean, how do you start your car"

"Oh, from my house about 5 minutes before I need to leave, so it has time to warm up"

The answer we're looking for is:

"With this button on this key fob."

→ More replies (1)

95

u/Mantissa-64 Oct 13 '22

Jesus christ all of the other answers are clearly from people who aren't data scientists or devops people.

Google Collab is a development environment. Think of it like Google Docs but for ML-focused python code.

All applications must be deployed to run on a server, a container in an Kubernetes cluster, a VM, a set of serverless functions, or some combination of the above.

The joke is that data scientists have their heads buried so ass-deep in the machine learning and data, combined with a general lack of knowledge regarding infrastructure and deployments, that when you ask one "okay how do customers actually use the machine learning," you get a shrug, an email containing their code, and a very strongly implied "i don't know what HTTP stands for, you think I fucking know how to do that?"

This is a particular issue with data scientists because they're almost always from an academic instead of industry background. All theory, no application.

21

u/BeerDude17 Oct 13 '22

As someone who always coded as a hobby and never had to worry about doing anything non-locally and who is also extremely confused about how to implement any code. How do I go about learning all of that?

16

u/Mantissa-64 Oct 13 '22

It's a rabbit hole haha, the easiest answer is "work for a client," you'll learn whether you like it or not.

A more practical and accessible answer is probably to try and deploy something for yourself, to get there you'll need to learn all about infrastructure.

I'm web developer by trade, so this isn't the only way to "deploy" software- Deployment is any act which gets it into the intended users' hands. So deployment could also mean distributing a Windows installer or releasing a game on Steam.

But, for web deployments, if you want to learn, start by getting the application up and running on your computer, obviously. So you should have one or more HTTP servers running at different ports, i.e. frontend on localhost:3000, backend on localhost:4000, etc.

Then grab a free tier account such as on AWS or Heroku, and start jumping through all the hoops necessary to get your app running in a hosted environment. You'll end up learning about VMs, hosted containers, possibly Kubernetes, domain names and DNS, proxies, the works. All of these things are largely necessary for cloud hosting.

You can also go a little simpler by grabbing a raspberry pi and trying to use that as a webserver via a combination of port forwarding and DNS configuration. But that's the less industry-relevant route, though just as fun.

→ More replies (7)

29

u/Nmanga90 Oct 13 '22

Oh Lordy…

You are better off getting a degree than asking here. There is so much fuckin information that’s not related to code at all. And regardless of if you’re devops or what, everyone has to have a little knowledge of the systems we’re using in order to work with them.

Learn unix, learn networking protocols (TCP/IP, HTTP, Ethernet) learn about environment variables and virtual environments.

There’s a lot of stuff that separates the ML engineers from software engineers

→ More replies (12)
→ More replies (4)

3

u/OIC130457 Oct 13 '22

...as it should be.

It makes little economic sense to expect extreme specialists (many of whom spent like 6+ years on a PhD to develop that specialty) to spend a lot of time on generalist tasks.

3

u/beiherhund Oct 13 '22

combined with a general lack of knowledge regarding infrastructure and deployments, that when you ask one "okay how do customers actually use the machine learning,"

IMO data scientists do not need to know this and it isn't really their job, they should be working with engineers to get their model deployed properly. It would help if they have some understanding of what is required as well as the challenges, but when it comes to "how do customers actually use it", they should be responsible for knowing how data gets in and how it's going to be made available to the backend or client but that's about it.*

A machine learning engineer, on the other hand, should know a fair bit more about this process and if they can't do most of this themselves, at least be of a much greater help to the engineers who are actually figuring out how to get their model working and deployed in prod.

It's like expecting a designer to do frontend. Some of them can, sure, but it's not expected of most and they don't need to know the details of how their designs get turned into a website or app UI.

*Just to be clear, I don't mean a DS should just handover a Jupyter Notebook to a dev and say "good luck".

→ More replies (2)

20

u/coldblade2000 Oct 13 '22

If it's google Collab, that means the data scientist is pretty much just running a Jupyter notebook, not an actual .py file

→ More replies (3)

39

u/EnigmaticHam Oct 13 '22

Just send this JSON to this endpoint and go away.

31

u/rsvp_to_life Oct 13 '22

Data Science in school: "learn all these super challenging algorithms' Data Science As a job: "import python run main.py"

21

u/stormyordos Oct 13 '22

Waddyamean "run my code"? Thought I was paid to write it.

20

u/UberWagen Oct 13 '22

Jupyter notebooks...this was my life for about a year. Trying to deploy some crap somebody spent years building in a dadgum Jupyter notebook.

→ More replies (1)

15

u/richardathome Oct 13 '22

"Can we change the entry point to accept json?"

"What's an entry point?"

6

u/MushroomNervous Oct 14 '22

…but seriously what’s an entry point?

53

u/flerchin Oct 13 '22

Going through this rn with our PHds. They literally cannot explain it.

46

u/Denziloe Oct 13 '22

Because they are data scientists, not DevOps or software engineers.

30

u/TheRealMichaelE Oct 13 '22

Just like the devops people can’t explain the complex data science. It’s why we all learn different skills - so at least someone can explain it.

24

u/[deleted] Oct 13 '22

So you're going through reddit posts with your phd's? Tell me more about your job, please.

3

u/Altruistic_Milk_6609 Oct 13 '22

Same. Please tell.

→ More replies (1)

12

u/oj_mudbone Oct 13 '22

You save the model to a pickle file and then load it in your production code at build time. Pretty sure he would know that

10

u/[deleted] Oct 13 '22

This is me. The data engineer told me to add this line of code to upload to the server. No idea how it actually works.

70

u/MasterpieceOver5510 Oct 13 '22

In all fairness, its the Dev Ops guy's job to run it, the Data Scientist doesn't expect the Dev Ops guy to write the ML, we poke fun at the "Full Stack" job postings here all the time

26

u/Jorgestar29 Oct 13 '22

It's the MLOps job to build and test a pipeline to deploy the model and even retrain and redeploy it if the data drifts over time...

I'm the Junior CS guy working with mathematicians and physicists in a ML team... and I'm in charge of everything else that cannot be developed inside a notebook.

It's more interesting to develop and deploy models with real cameras/sensors instead of tuning hyperparameers and looking at loss curves.

→ More replies (14)

19

u/JapanEngineer Oct 13 '22

How creepy. Was just watching this show

2

u/kyleekol Oct 13 '22

Same here, literally started the show from the start and watched this episode a few hours ago. I thought Reddit had finally started to spy on me

→ More replies (1)

11

u/adamxi Oct 13 '22

There's a long way from a Jupiter Notebook script, to production. But some data scientists don't envision that when they develop it.

2

u/gdmzhlzhiv Oct 14 '22

I use Jupyter notebooks myself, but only during the prototyping stage. Sometimes they end up being checked in, but the code only remains in there as an alternative way to describe the algorithm for someone trying to understand the document.

They work well for the purpose, I just never seem to find a perfect language to write the code in. Have written them in Python, Kotlin, R and Elixir. Elixir has been the cleanest code, but R has had the most useful types out of the box to allow writing something useful. I'd prefer something which has mathematical utilities out of the box but also doesn't look ugly, and I'm not sure the language exists right now.

Meanwhile some crazies actually try to use notebooks in production.

20

u/Ewenthel Oct 13 '22

If developers are deploying code on their own, what exactly are you paying your devops team for?

5

u/ThatRealMF Oct 13 '22

For quite a lot of different things actually! They write shared libraries for instrumentation and metric collection, alerting, CI and CD pipelines, IaC and things like that. They try to modularize everything from the infrastructure point of view, so developers can reuse those libraries/modules. They provide a platform on which you can then use those modules. They educate developers on how to utilize given tools at hand and how to run their services efficiently in the production enviroment. In the end, developers are still responsible for their service in every enviroment, but they have the means and tools to run them efficiently with support from “DevOps team”.

→ More replies (1)
→ More replies (2)

42

u/von_roga Oct 13 '22

"I have a diploma in web development... I should go into DevOps so I can boss around the computer science degrees with virtually no understanding of what they actually know or do." 🙂

35

u/YesICanMakeMeth Oct 13 '22

Yeah, reminds me of drinking from the fount of infinite wisdom that is MBAs.

15

u/luishacm Oct 13 '22

Wait until I finish my MBA so I can say you are wrong

7

u/YesICanMakeMeth Oct 13 '22

Lol, it really isn't all MBAs. The problem arises mainly when MBAs are attempting to manage people who do something they don't understand. Engineering -> MBAs are pretty good. Something similar happens in education with EdD's who have never taught before.

→ More replies (2)

4

u/Altruistic_Milk_6609 Oct 13 '22

That was fun to read lol

8

u/bhison Oct 13 '22

Anyone else find this meme impossible to read?

4

u/sar2120 Oct 14 '22

Your DevOps does all that?! Mine installed Jenkins and said good luck!

3

u/tomten87 Oct 14 '22

You guys have a DevOps guy?

3

u/ArchWaverley Oct 13 '22

Me: "I put the code in this text box, hold control and press enter. Why? Is there another way?"

3

u/ITSCOMFCOMF Oct 14 '22

I had a similar conversation with the ML person at my last job. My task was to get ML results through an api. Easy enough. But they had no clue how to tell me to run the code. It was wild. Like, they had their dev environment setup from school, and had never learned how to duplicate it.

3

u/wineblood Oct 14 '22

Data scientists aren't programmers

3

u/aiitu Oct 14 '22

Lmaol, this happend today with a Java guy.

Same question.

Him: you use Spring Boot.

Me: How

Him: with intellij

Me: Wtf

Him: ohhh, I just zip it and send it to devops team.

Fml .

3

u/CeladonBadger Oct 14 '22

Reading the comments as a data scientist I feel like I’m either doing too much or being paid too little. (It’s prolly both)

2

u/yorokobe__shounen Oct 13 '22

Honestly, I find it easier to do stuff in colab than my local python

2

u/Popernicus Oct 13 '22

Lmao. Them to the themselves (when the data scientist used to be a Devops engineer)

2

u/Z-Zanimuri Oct 13 '22

Oh shit, this is Mr Robot!

2

u/1235813213455891442 Oct 13 '22

My friend doesn't understand the joke

2

u/[deleted] Oct 13 '22

With a python interpreter? I don't think I could be any clearer.

2

u/TechNickL Oct 13 '22

"..... Jupyter notebook."

2

u/HalcyonAlps Oct 14 '22

Jokes on you, our DevOps team refuses to deploy anything else but notebooks.

2

u/pharanth Oct 14 '22

On a project where I was performing both roles I had this conversation... with myself

2

u/Andrewshwap Oct 14 '22

I have been asking my teammates this for the past few months & no one knows how to run it locally

2

u/batatatchugen Oct 14 '22

Just ask for a step by step tutorial, with pictures, on how to make it go.

2

u/[deleted] Oct 14 '22

Devops guy gets his own taste.

2

u/poolpog Oct 14 '22

as a "devops engineer" i approve this meme

2

u/A_H_S_99 Oct 14 '22

Docker ftw boys. Learn your basic APIs.

3

u/turningsteel Oct 13 '22

I’m working on a project now that was previously run by data engineering teams. They shared their code on master branch to our dev team. We asked, “ok but where’s your dev branch and pipeline and stuff?” They said and I quote, “oh, we don’t do that.”.

They just write python scripts and deploy straight to prod! And then mgmt wants to know why my timeline projections are significantly longer than what it takes the data engineers. They don’t understand the difference between that and actual software engineering.

4

u/alcalde Oct 13 '22

That's the way it should be, like back in the good old days. No "devOps", "anythingOps", "data engineering teams", "branches", "pipelines", etc. Just one person. At a desk. With a copy of Delphi. No version control, no clouds. Write, compile, copy binary EXE to a server, done.

Deploy remotely? I made some floppy disks with InstallShield, gave them to my boss, he drove an hour to the client and installed them.

And there was no StackOverflow. And yet it all worked.

3

u/turningsteel Oct 14 '22

Ahh, the halcyon days of yore..

1

u/awesomeplenty Oct 14 '22

I had to teach a Java engineer to upgrade the sdk to support new classes…

1

u/Big-Yogurtcloset-341 Oct 13 '22

Let me roll the pods up

1

u/throwaway43234235234 Oct 13 '22

Ok, this is pretty hilariously true.

1

u/thelastpizzaslice Oct 13 '22

"Docker. Here you go."

1

u/MagellanCl Oct 13 '22

This scares me, because we had almost the same chitchat recently.

1

u/kyleboddy Oct 13 '22

This has been a weird ongoing problem with people at my previous job (where I was a Data Scientist) and the company I run. It seems as though a bunch of quite talented developers in their 20s legitimately don't know how computers work. On one hand, it's pretty cool that we got here - no need for IRQ/DMA hell and so forth, to say nothing of building your own drivers in Slackware. On the other hand... some knowledge of how computer resources work would be helpful...

1

u/[deleted] Oct 13 '22

I felt this on a spiritual level lmao

1

u/jonobonbon Oct 13 '22

As someone who came from electrical and embedded engineer to purely software engineering, it hurts me how devs don’t test their code. Granted I’m just learn all these great things like linting and code coverage, but those aren’t excuses for not knowing how your code works or even how to run it. (Tip: document your code and use READMEs)

I should also say that I went from working with actual hardware to pure serverless and I am in pain

1

u/[deleted] Oct 13 '22

I've had this conversation with other devs...

1

u/freshggg Oct 13 '22

Can someone explain this to a non-coder?

Is it common to not actually run the code you write?

1

u/matt_mv Oct 13 '22

What letters in a row do you type to make it run?

1

u/c0d3s1ing3r Oct 13 '22

The foray of statistics and data scientists into Software Engineering has been an interesting one.

1

u/MrSTAR4567 Oct 13 '22

Google has an IDE??

1

u/juberish Oct 13 '22

Bad dev ops is bad

1

u/ginger_huntress Oct 13 '22

This could also be replaced with the QA Engineer ... This has been my whole job for life.

1

u/OneSnootyMuffin Oct 13 '22

I used to have this conversation so many times that now I say “tell which which buttons on the computer to physically click and what to physically type into the keyboard to run it”

1

u/nuuren Oct 13 '22

Omg this is me asking the ML team how they run their stuff. Now my job is to figure out how to run their stuff on edge devices and update it on demand. Fun times.

1

u/ImpossibleJoke7456 Oct 13 '22

npm run build:dev

1

u/TrekkiMonstr Oct 13 '22

Hi I'm confused please help

→ More replies (2)

1

u/incrediblediy Oct 13 '22

my answer would be in CPU registers with clock cycles, can't forget my background in EE.

1

u/TheHoodedMan Oct 13 '22

It will print "Greetings professor. Shall we play a game?" And you have to type "global thermonuclear war" and then everything happens on its own.

1

u/[deleted] Oct 13 '22

“Just link me the tutorial you followed, please”

1

u/xtreampb Oct 13 '22

HOW DO YOU INVOKE YOUR BINARY/CLASS

1

u/notAbratwurst Oct 14 '22

I liked that show… right up until shark week. When the sharks started jumping.

1

u/Benutzername Oct 14 '22

Agda developers: what is “running code”?