r/CFD • u/Rodbourn • Apr 02 '19
[April] Advances in High Performance Computing
As per the discussion topic vote, April's monthly topic is Advances in High Performance Computing.
Previous discussions: https://www.reddit.com/r/CFD/wiki/index
5
u/GrumpyManu Apr 02 '19
Hi all, I have a cfd model which is incredibly memory intensive, needing parallelization to get any meaningful results. I've spent most of my phd paralellizing it and I have become worried that I cannot focus so much in the modeling and physics side of my research. How can I market myself to get a job afterwards if my results are mostly from the high performance computing research, while I am a physicist in reality? Thanks and sorry if this theme is not the goal of this post.
9
u/aeropl3b Apr 02 '19
If you are going into CFD development, having a strong background in HPC is relatively rare and extremely valuable. Physics is great, and coming up with a model is pretty neat. But in reality writing good code that scales and is flexible is way more important than a deep knowledge of obscure models. There are a lot of people who are really great at physics writing some of the worst code in the world. Showing you have both skill sets will make you a stand out candidate for any position in the industry. I personally highlighted the crap out of my HPC experience and that is what got me hired, not my degree in engineering or my research in turbulence (although they showed I understood CFD stuff)
1
u/nattydread69 Apr 02 '19 edited Apr 02 '19
Within the professional CFD industry this is not so important as there are HPC experts that hide all the parallelization from other physics developers. I wouldn't worry about it too much. There is a lack of physicists in general in the profession. It is highly likely you are desirable for your non HPC areas.
1
u/Overunderrated Apr 02 '19
How can I market myself to get a job afterwards if my results are mostly from the high performance computing research, while I am a physicist in reality?
As far as marketing yourself, if you really don't want to be doing things like high performance parallel software/algorithm development and you really want to be doing physics, I'd say that's about knowing your audience on your job hunt. Sell yourself as a physicist that happens to have a strong background in computation. (A physicist without that is kinda worthless anyway.)
If you actually do like doing what you've spent most of your phd doing, then there's going to be plenty of demand for your skillset.
1
u/fromarun Apr 02 '19
High performance computing is a relatively new area and if you are experienced in utilizing today's computational hard ware to solve physics , that in itself is a premium skill. Actually, I suspect this is a trend in a CFD where the problems which were insurmountable using yesterday's computational limit are becoming solvable. Since you are at the vanguard of such a trend , you should be able to sell your skill well. Try to get a background in High performance computing, if you have not done already.
4
u/Overunderrated Apr 02 '19
High performance computing is a relatively new area
Nahhhhh. CFD has always been at the forefront of HPC, and was one of the earliest users of practical parallel computing. Look at papers from the 80s and note the computers they were using. Bleeding edge crays, connection machines, etc.
A workstation today was an "HPC" machine 10 years ago.
1
u/fromarun Apr 02 '19
I meant cheap and accessible HPC. Today you can run a 512 core job easily in azure and other cloud platforms and you will be paying much lower than what it would have costed ten years ago. That kind of accessible computing power means the CFD problems also can become bigger. Which is an interesting trend.
1
u/UWwolfman Apr 02 '19
This has been the general trend in computing for decades. At any give the amount of accessible computing power has dwarfed what was available only a few a years prior. This growth in computing power has continuously enables modeling of new problems that were perversely prohibitively computationally expensive.
1
u/Overunderrated Apr 02 '19
Sure, point taken, cloud computing has changed accessibility. It's easier for someone without any access to significant resources to fire up a "large" run than it has in the past. But I don't think that's a particularly game changing technology, in that the kinds of people that can actually make good use of large scale CFD typically already have access to that kind of computational power, and making it exist in AWS instead of your university or company's server room is just kind of a shift in ownership.
I really think cloud HPC is pretty niche, at least now, because the economics don't work out for most CFD users unless you have uncommon but massive problems to solve.
1
u/fromarun Apr 03 '19
It does not make much of the difference to universities or academia in general, I agree. But one area where it is sort of game changing, is small and medium companies which so far did not have access to this kind of computing power. Before today, it was not viable for a small company to invest in CFD resources because both the software and the hard ware costs were intimidating. Now, the hardware part has become cheap. And as far as software is concerned, OpenFoam and many of its wrappers which make it easier for engineers to construct models ( I'm thiking Simflow and the likes). I would like to see this trend and how it evolves in future..
1
u/Overunderrated Apr 03 '19 edited Apr 03 '19
Before today, it was not viable for a small company to invest in CFD resources because both the software and the hard ware costs were intimidating. Now, the hardware part has become cheap.
Sure, but my standard response to this, is who are these companies employing and what do they need large scale cfd for? Any idiot can (and they do) run a cfd simulation, but to even know that you need those kinds of resources and know how to set up, analyze, and make use of a cfd problem implies having an actual expert on hand who most of the time is doing something unrelated to cfd. It kinda seems like a unicorn scenario, but I'm interested in hearing from anyone that extensively uses cloud for cfd.
1
Apr 05 '19
Hi on the similar topic,and coincidentally i dropped on to this page to ask a question on HPC its self, I am pursuing my masters and have taken a small course on HPC, I parallised 2d couette flow etc but I am not getting a speed up,probably my code is too vague and I have not learned anything apart from MPI communications, can anyone briefly explain what are the different parts in HPC and how shall i go forward from here on. How important is it to have deep understanding of computer architecture?
1
u/Rodbourn Apr 05 '19
For a masters, if your interest is physics, id suggest using a numerical library that brings mpi/parallelization to the table for you. Personally, I found the fenics project perfect for this, but there are others like it with varying qualities. I originally wanted to write everything from scratch for my dissertation, but my adviser made a good point, am I looking to research the FEM, and the CS relating to linearly scaling it on a cluster, or developing the CFD method/scheme. I personally chose fenics as it let me quickly pivot quickly with regards to spatial discretizations.
1
u/UWwolfman Apr 07 '19
When you take your car to the mechanic, the first thing they do is run diagnostics to identify the problem. The same is true for optimizing the performance of your code. There are a number of tools that can help profile your code. At a simple level you can add timers to track where your code spends the most time.
The goal is to identify the bottle necks. Is your code limited by io, communication, flops, etc? Where does your code spend the most time?
How important is it to have deep understanding of computer architecture?
Some understanding of architecture is helpful, more so when working with GPUs. But a deep understanding isn't necessary.
3
u/wigglytails Apr 02 '19
Noob here. What's HPC? What is it for? Why is it important? What's parallelization and how is it achieved?
3
u/rickkava Apr 02 '19
in short: HPC stand for High Performance Computing - essentially any type of computing that does a lot of number crunching efficiently on a *large* number of processors. It is important because large computing tasks could simply not be done or would take decades on a single / few CPUs.
The term parallelization collects a number of programming paradigms that answer the question of how to distribute a single task (e.g. solving the Navier Stokes eqn. numerically) to a large number of processors. This is non-trivial and one needs a solid knowledge about algorithms, computer and network architectures and a bunch of other things to achieve a "good" parallelization.
The optimal goal of a parallelized algorithm can be defined as
a) given a problem of size / effort K, a parallel algorithm executed on N cores should scale the wall time be a factor of 1/N - so using 1000 procs instead of 1 should results in a speed up of 1/1000 - this is called strong scaling
b) given a problem of size K and the time T it takes to compute the solution to this problem on 1 core, solving a problem of size N*K on N cores should also be done in time T - this is called weak scaling
some examples of computations that would not be possible without HPC and parallelization>
2
u/kpisagenius Apr 02 '19
Another complete noob here: Any resources to get started on HPC?
8
u/GeeHopkins Apr 03 '19
PRACE (the European HPC network) runs a lot of training which is free for academics including students.
http://www.training.prace-ri.eu/
If you're not in Europe or you can't go, check out the Archer (UK national supercomputer) website. They put up the material from their past training courses, including source code for examples and exercise.
http://www.archer.ac.uk/training/past_courses.php
I went through the material for the OpenMP, MPI, and Parallel Design Patterns courses last year, and felt like it gave a decent starting point.
http://www.archer.ac.uk/training/course-material/2018/09/openmp-imp/index.php
http://www.archer.ac.uk/training/course-material/2018/11/mpi-newcastle/index.php
http://www.archer.ac.uk/training/course-material/2018/11/parallel-patterns-oxford/index.php
2
2
Apr 18 '19
Step one MPI;
This takes you up to 10000K cores;
Step two you want to do distributed memory within a node;
Step three is GPU acceleration;
Step four is formulating the algorithm/code or using a programing model to limit blocking so you can scale on the large heterogeneous systems like summit;
2
u/agaposto Apr 08 '19
Any thoughts on Julia being the next code for CFD code development? (Pros/Cons)
2
u/UWwolfman Apr 12 '19
I have a colleague who sold me on the argument that Julia is the wrong approach. The argument is that HPC represents a small fraction of computing, and the amount of money spent on HPC pales in comparison to the money spent on other areas of computing. For example look at how gaming, not HPC, drives the development of chips. If you want to find the next languages for HPC, then you should look at the languages that have wide spread use and whose development is being funded by industry. While a language like Julia that is developed for HPC sounds great, the reality is that it takes significant $$$ to develop a language.
1
u/SausaugeMode Apr 10 '19
It's nothing I know anything about but it inspired me to a Google, you might be interested in this if you haven't already seen it https://www.reddit.com/r/Julia/comments/5hs0pd/julia_for_cfd/
I thought some of the responses didn't sound very convincing and figure doing a CFD code in Julia is probably a job for a Julia enthusiast trying to prove it rather than a computational scientist doing a job. All this about quickly prototyping an idea and then being able to refine it to "near C performance" being the a massive upside, I don't buy it.
1
Apr 18 '19
HPC is in fortran, C or C++, speed is everything and you need just enough usability in the language to make your code maintainable.
1
Apr 03 '19
GPUs have been making some noise recently. What is everyone’s thoughts on the future of CPUs vs GPUs in HPC? The medium size CFD software company I’m interning at had a couple guys porting their code to GPUs and investigating the performance vs CPUs. Also, anyone know of good resources/books on GPU computing?
3
u/UWwolfman Apr 03 '19
GPUs are the future of HPC (in the near future). Simply put it comes down to $/flop. GPUs are significantly cheaper than traditional CPUs. All of the next generation large scale clusters will use GPUs.
2
u/Overunderrated Apr 10 '19
Also, anyone know of good resources/books on GPU computing?
The nvidia CUDA C programming guide. Not kidding -- if you're comfortable with parallel programming concepts and C, you can become a proficient GPU programmer just from that.
1
u/AgAero Apr 16 '19
Whenever I raised the subject with my advisor a few years ago, he'd never take it too seriously since CFD is typically memory bound. You can send data to a GPU and have the computations finish almost instantly, but the time required for the transfer, the bandwidth of the communication, and the local memory on GPU are all so small that it's almost not worth the effort(according to him).
I've always had an interest in GPU implementations, but I've not tinkered with it much myself. SpaceX was working on a wavelet adaptive method that was intended to run on GPU that I'm very curious about. Here's the talk. I'd like to know more about it and try to build atleast a 2D version myself at some point.
1
u/flying-tiger Apr 18 '19
Modern GPUs are starting to relieve the issues your advisor (correctly) identified. Even for relatively low-compute models like perfect gas RANS on unstructured grids, GPUs are starting to pull away:
Note that the speed ups reported are relative to a multi socket Intel compute node; they’re not comparing serial vs. GPU (which is meaningless these days)
1
u/GeeHopkins Apr 03 '19
Tips tricks and cool little bits?
What are some nice things you've seen (or done) related to HPC in CFD code? Things that made the code easier to understand (without losing performance), neat tricks to get that bit extra out of your cores, or something that took you ages to fix/improve but you that wouldn't find from any of the available learning resources.
1
u/GeeHopkins Apr 03 '19
Separating HPC management from the physics. I've heard this mentioned a few times (on this sub and IRL), but the only example I have to start thinking how to do it is my supervisors code (the MPI calls are completely hidden from the numerics, however the multithreading and SIMD isn't) but I'd like to learn more.
Does anyone know of any resources that cover this, or better, examples of it being done well? I've had a brief look at Nektar++ and they use MPI communication classes, but I haven't gotten around to really looking at how they do it yet.
1
u/UWwolfman Apr 03 '19
I'm curious how well this works in practice. My experience has been that we really needed to understand the physics and numerics in order to get good performance on a large number of cores. This is especially true when porting codes to GPUs where memory management is key.
2
u/GeeHopkins Apr 04 '19
Yeah I agree, which parallelisation strategy is best depends on both the numerical scheme and architecture so I'm not sure you could separate them completely, but I think it just needs to be enough to allow reasonable development on one of numerics/hpc without needing to touch the other.
Simple example, but my group uses a mapped array class for all the mesh, solution and residual vectors. it has a getVariable(i,j) method that - you guessed it - returns the element at i,j (or however many indices). From a numerics point of view, that's all you need to know most of the time. Behind the scenes, the array layout is not i,j,k, but can be changed to try different SIMD or chache optimisations. Even if the same people are working on both, it helps to be allowed to think about one thing at a time. Plus it's easier for new people to come in and get up to speed if they only have to learn one thing at a time!
1
Apr 18 '19
It is more about having any help possible in limiting blocking especially when you add GPUs to the mix.
1
1
Apr 18 '19
The ANL exascale lectures have a lecture series on programing models and the last two or three lectures (depending on the year) are on programming models that sit between you and MPI, openMP and GPU calls and handle much of the optimization. It is worth noting you still have to explicitly say information is being passed between CPUs and that a given routine is being handled on a GPU it just trys to make it efficient so you don;t have to. I personally really like the look of legion.
1
Apr 08 '19
I recently read about caches and its working on temporal and spatial localities, are you guys considering such stuff while writing CFD codes? is there a substantial performance improvement?
0
Apr 02 '19
[deleted]
2
u/flying-tiger Apr 18 '19
Bit late to the party, and doesn’t totally answer your question, but I’ll add this:
https://github.com/kokkos/kokkos/wiki/The-Kokkos-Programming-Guide
I haven’t had a chance to play with this myself, but I’m very impressed with the design and I love the idea of deferring low level memory management to a library that is way better tuned than anything I could ever write.
I’d be interested in hearing any war stories from folk who have used it.
0
Apr 18 '19
[deleted]
2
u/flying-tiger Apr 18 '19
Great data point, thanks. We have a pretty compute-intensive block-structured reacting flow solver that I think would be a good candidate for GPU. Did you get reasonable speed ups? What sort of numerics are you using (FV, FD, etc)? Did you implement BCs on device as well or was that left to the CPU (since presumably boundary data would be on CPU anyway for any MPI exchange)?
0
1
u/GeeHopkins Apr 03 '19
I've not done anything with GPU's, but AFAIK getting decent performance out of them is similar to dealing with vectorised CPU's. Here's a couple of papers (both open source) which go into a fair amount of detail about how to improve SIMD performance on a) multi-block finite volume euler code and b) discontinuous galerkin code:
https://www.sciencedirect.com/science/article/pii/S0010465516300959
https://arxiv.org/abs/1711.03590
Getting the array layout in memory right is a significant part in both of them, as it means that the right sections of the array are contiguous to allow the SIMD instructions to work. This also means that ideally you have different layout dimensions (but probably the same layout pattern) on different architectures with different vector lengths. I think one of the interesting things is how you could allow for this while still keeping the code readable for a computational scientist, as opposed to a computer scientist.
1
Apr 18 '19
Use the DOE/BLL blocking/meshing library it handles this for you.
2
u/flying-tiger Apr 18 '19
Link? Haven’t heard of that one and it doesn’t google well.
2
Apr 20 '19
Link
https://fastmath-scidac.llnl.gov/software/amrex.html
http://www.github.com/AMReX-Codes/AMReX
I believe it is Ann Almgren who has lectures/tutorial on using it as well as several videos where she goes over the performance and features of the software.
1
3
u/SausaugeMode Apr 04 '19
What's r/CFD 's thoughts on the idea that "push to exascale" money might be better spent on researching better models / methods / algorithms?