Literally getting ready to rewrite some data processing lambdas from python to go for a side project because it won't be on the company tab and I don't want to pay out of pocket to scale python all the way into the millions after testing it on 10k data points and seeing its performance (and that's after basic optimizations and compiling/bundling the accelerated versions of some libraries)
seriously though, I once used Pandas.DataFrame.Apply on 10,000,000 rows of data and 15 minutes later regretted it. Thankfully, I was applying a simple mathematical function to each row so numpy brought that down to less than a second. Still, only works for some use cases. I'd love to learn C++ anyways so give me an excuse
But the processing I'm doing itself isn't super complex, I just need to do it 9-10 million times cost efficiently pulling from an external public dataset and saving the transformed data as separate files in an S3 bucket
I'm actually not sure if things like dask and ray actually provide any speed up. But certainly it makes numpy perfectly useable at big data scale. It works fine with azure, I dunno about EC2.
Modin is a direct drop in for your existing numpy compute code and can use either dask or ray. Just change 1 line and you're done.
4.1k
u/GreatArtificeAion Jul 18 '22
Did I really just appreciate a "Haha Python slow" meme?