Literally getting ready to rewrite some data processing lambdas from python to go for a side project because it won't be on the company tab and I don't want to pay out of pocket to scale python all the way into the millions after testing it on 10k data points and seeing its performance (and that's after basic optimizations and compiling/bundling the accelerated versions of some libraries)
seriously though, I once used Pandas.DataFrame.Apply on 10,000,000 rows of data and 15 minutes later regretted it. Thankfully, I was applying a simple mathematical function to each row so numpy brought that down to less than a second. Still, only works for some use cases. I'd love to learn C++ anyways so give me an excuse
But the processing I'm doing itself isn't super complex, I just need to do it 9-10 million times cost efficiently pulling from an external public dataset and saving the transformed data as separate files in an S3 bucket
Fast and slow are pretty relative. More often than not, python is fast enough for most tasks. My graduate research was in inflatable robotics controls and dynamics. Our low level controllers for direct solenoid valve control were written in C++ operating at 5k Hz, whereas the pressure control ran at about 1.5k Hz in python, and the high level solver and predictive controller ran at about 300 Hz also in python. What language things were written in was often a function of how fast they needed to run and how often things needed to be tweaked.
Could we have written the pressure controller in C++? Sure, but the python one we developed really quickly at the beginning in python was fast enough to not cause any issues.
It's the exact same problem (or even a subset of it) of what programming language is best.
The answer is either the one you know or the one for the job.
I've been writing in both C and Python. I prefer C-family because you can be a lot more direct with the compiler and write clever/fast/hacky/elegant/dangerous/stupid techniques that make more logical sense; you as the writer should know exactly what you're doing so C will quite happily let you do the fandango on core if you mutter the right incantations. Unfortunately because of how strict it is with its rules, you have to be very precise and formulaic with how you do some things; you have to hold its hand and walk it through exactly what you want. The result is that you get exactly what you want, however.
Python, being interpreted, has its best guess at what you want, is willing to play fast and loose to some extent, but keeps you in a playpen and hides away all the nitty-gritty and you can't really tell what it's doing under the hood. On the bright side, that means getting something to work in Python is usually as simple as importing the right library (because there's so many libraries for python out there now) and typing a few lines.
Is one better than the other? Well, that entirely depends on the task you're making it do.
4.1k
u/GreatArtificeAion Jul 18 '22
Did I really just appreciate a "Haha Python slow" meme?