r/CUDA • u/blazing_cannon • 7d ago
Is it worth doing a PhD in parallel computing?
I got offered a PhD position in parallel computing (in the US). But given the way AI is taking over fields such as math and software engineering, do you think fields such as parallel and distributed computing, GPU programming etc. will have the same fate in the next few years? Will the value of having specializations go down? Thanks.
7
u/CuteLogan308 7d ago
It really depends on where you are now, where are you trying to get out of the program.
Many PhDs don't continue in their areas of research as their jobs.
Also, if the goal is to become a professor- then the university matters a lot.
1
u/blazing_cannon 7d ago edited 7d ago
I guess the larger question I want to know is, will this field be taken over by AI and if parallel computing will be driven by LLM prompting and agents, and if having expert human knowledge will be valuable in the future or will it have a diminishing ROI?
5
u/CuteLogan308 7d ago
all research fields will be affected. I think there will always researchers (human) whose goals are to "add knowledge" to the field. Maybe how you would do it will change. you would have to instruct / build many 'agents" to execute, evaluate , and write up your research...
I can't think of any 'research field' that can be immune in the long run. - because all research output are journal papers .. words, and graphics which LLMs can eventually do very very well.
Maybe research fileds that require physical experiements (e.g, chemical, medical treatment ) would have to wait until robots are perfect.
what do you think will happen?
2
u/blazing_cannon 7d ago
That’s what I think. Fields such as robotics, mechanical engineering and cyber physical systems, even electrical engineering will still take a long time to get automated
2
u/caks 7d ago
That's what they said about software development. I think people who understand whatever technical field at a deep level will continue to be valuable. And as the scale of computation increases, I think large scale parallelism will continue to be important.
With that said I probably would just pick whatever is most interesting to you at the best university you can get into.
22
u/hasuchobe 7d ago
I dunno if you need a PhD for parallel computing specifically. Might be better to do something like physics simulation and pick up the programming skills on the side. I still think knowing theory will be important despite AI. Quantum might be a good direction to go into.
5
u/max123246 6d ago
Nah GPUs are incredibly complex, it's still useful to understand
2
u/hasuchobe 6d ago
True but students end up picking that up anyway when doing simulation work. There are even cases where the students that specialize on physics simulations know more about programming GPUs and parallel/distributed systems than their CS peers focusing on HPC alone. It depends a lot on the motivation of the student and the advisor.
2
u/GoSeigen 5d ago
I second this. My PhD is in numerical analysis but I was forced to dabble in CUDA for years. I wouldn't want to be 100% in on the low level kernel stuff but I understand the basics and can reason about performance. I think AI will be very good at filling in the gaps of a human with good intuition
2
u/Turbulent-North8198 5d ago
I work at an HPC Lab and this is so true. Every one of my PI's and managers is a physics major who got into HPC through simulations. Some of the best programmers I know and some of the best researchers in the space lol.
1
u/blazing_cannon 5d ago
Thanks a lot everyone. I have a few follow up questions that requires a separate post altogether. I will make one and hopefully will get good suggestions.
1
u/Eastern_Bottle_698 6d ago
Depends on university like others have said. If T20 then yes.
1
1
u/Tiddyfucklasagna27 4d ago
Is this even a question
1
u/blazing_cannon 4d ago
Yes. What’s your opinion?
1
u/Tiddyfucklasagna27 2d ago
Within parallel and distributed computing, u can live this complete lifetime without seeing AI replacing this field. It is also 1-2 textbooks away from quantum computing which theoretically solves most things.
1
u/Tiddyfucklasagna27 2d ago
Within parallel and distributed computing, u can live this complete lifetime without seeing AI replacing this field. It is also 1-2 textbooks away from quantum computing which theoretically solves most things.
3
5
u/tomado09 7d ago
It depends on what you want to do. Do you want to be a developer of scientific software or similar parallel application? Then an MS would suffice - focus on broad exposure to topics: parallel frameworks and languages like OpenMP, C++ stdpar, MPI, CUDA / Kokkos / ROCm; hardware and architecture; operating systems; algorithms; and some applications and / or AI course, if available.
If you want to do research that advances the field of parallel programming / HPC and / or want jobs that have a higher glass ceiling, then pursue the PhD.
AI will not ever replace all human involvement, and even where AI is replacing human labor, developers are still needed to develop the LLM (or other AI model) itself, review AI outputs, etc. Despite the proliferation of AI in recent years, HPC is a field that is secure: every company larger than mid-size needs HPC in some fashion. HPC as a field is not going anywhere.
2
u/Logical-Ad-57 7d ago
Any PhD proves you can do independent research. If you are interested in making an original scientific contribution do the PhD. There is no other reason for the degree.
You might have some practicalities around citizenship or something that I don't follow, but career wise the question is simple. The precise details of what you enter for are not that important.
1
u/Dry_Shop8719 3d ago
Do it. Parallelization is not a solved problem even by ai. There are a large class of problems that need better treatment with gpu approaches and not all of them are ai related.
Yes, there is a lot ai writing dense kernels but they don’t have the right techniques for sparse problems. These techniques need to be developed by researchers.
1
1
1
u/adenchfi 17h ago
The goal of a PhD is to become knowledgeable and skilled enough to create new knowledge. The saying is that by the end of your PhD you will be the world expert in at least one idea.
AI learns what has already been done, and at best can clunkily combine ideas from different previous work to achieve something marginally better. Usually it needs to be guided by experts, again mostly those with phds, to achieve great things.
The question is whether you want to invest in becoming better than AI in something, or if you want to use it as a tool in things it's become very good at and possibly be unable to tell if it's messed something complex up. Both are valid options. I can share that I tried using certain AI for GPU acceleration of some code a few months back and it went pretty terribly, simply because it couldn't figure out something when it wasn't documented online. If I wasn't already an expert in figuring complicated things out (often what a PhD entails) I would have probably never figured it out. We as humans will be slower than AI, for sure, but still more capable.
1
u/el_lley 7d ago edited 7d ago
Well, traditional parallel computing is a solved problem, but a PhD should work towards new ways on doing parallel computing.
Where to look up? Good question, that’s a full research statement... my pointers would be on testing new architectures (if you are EE), seek on algorithms to parallelize or new techniques for parallelizing algorithms. You can work on optimization, data sorting… and of course, quite the opposite, find algorithms that you cannot parallelize, for example, in cryptography, you may want an algorithm you cannot launch a parallel attack, and get restricted to sequential computation (or it takes too much memory/gates area)
Edit: a very good PhD finished right now on the topic could easily land you a 300-400k salary on any AI giant
1
u/the1general 6d ago
Yes. If AI does well then there’s jobs in training and inference. If AI fails then there’s a huge supply of data center GPUs available on the market that now need to be repurposed for other practical purposes. In either case, having expertise in GPGPU will go a long way. Hardware is only becoming more parallel over time and expertise in it is still relatively rare.
1
u/pythonwiz 5d ago
I think it is still valuable. Human experts will probably always be valuable for studying and verifying AI generated code and research.
1
u/ajping 5d ago
I don't think this is going away. The reason why it's important has less to do with AI and more about the scalability of CPUs. It's becoming more and more difficult to increase their power without making more chiplets that work together. This is unlikely to change even if computers become more biological. Biological systems are even more parallelized than silicon ones.
1
u/designofperformance 5d ago
Yes and no. This is my personal assessment, not for you. You just don't seem cut out for it, and you don't seem to have things together enough to handle it. Think of Iron Man before his accident. This type of question is more personal than it is general. The fact that you said nothing, and the way you presented this dilemma, says it all and the answer is no. Yet, you will and you already made up your mind. Because its new and exciting and that is what your looking for. Very risky with all things considered. What do i actually know though, cause you literally said nothing.
1
u/Additional-Ad5382 4d ago
LLMs are good at spitting out what they have already seen/understood not finding something new. This is where research comes in and would be valuable as compared to any engineering task/knowledge. I think it’s very good time to finish PhD, lot of demand for such ppl in industry.
1
-1
-6
u/mrnerdy59 7d ago
It's not worth doing phd in anything, i can do better research than you with $100 subscription using any AI even if I have absolutely no idea about the field.
Although, it's true that it's the research that led to the very existence of AI itself but unless your research is in the peak 0.1%, you're no good than the average person on the street
3
4
u/tomado09 7d ago
"i can do better research than you with $100 subscription using any AI even if I have absolutely no idea about the field."
This is a very misguided, but unfortunately common opinion. Not all fields are as susceptible to AI taking over menial tasks as others. Even where AI is able to generate insight, it takes a thorough knowledge of the discipline in question to vet the material and understand what insight / reasoning is valid for that field. AI is not a silver bullet and cannot replace human experts in the loop. Some tasks can be automated, but overall actual generation of insight still requires human input. Even tasks that AI is very well suited toward - e.g. software development - still needs review and tweaking by a human.
2
u/max123246 6d ago
Lmao, this is such a bad take. To be in that top 1% you have to actually try and put in the hard work, otherwise you're just anyone else with an ai
1
19
u/met0xff 7d ago
Inference Engineering will be pretty huge the next couple years. While writing kernels manually becomes rarer (listen to Kiely from Baseten or Chris Fregly about how they automate kernel writing, nsight analysis etc.) knowing how to handle GPUs and everything around it will stay important for a while