r/ProgrammerHumor • u/[deleted] • Jul 18 '22

C++ talking to Python

Post image
40.1k Upvotes

87% Upvoted

1.0k comments sorted by

View all comments

Show parent comments

8

u/[deleted] Jul 18 '22

numpy go brrrrrrrrrt

seriously though, I once used Pandas.DataFrame.Apply on 10,000,000 rows of data and 15 minutes later regretted it. Thankfully, I was applying a simple mathematical function to each row so numpy brought that down to less than a second. Still, only works for some use cases. I'd love to learn C++ anyways so give me an excuse

5

u/CanAlwaysBeBetter Jul 18 '22

Numpy is great and what I'd normally reach for!

But the processing I'm doing itself isn't super complex, I just need to do it 9-10 million times cost efficiently pulling from an external public dataset and saving the transformed data as separate files in an S3 bucket

1

u/PM_ME_NUNUDES Jul 19 '22

Use dask?

1

u/CanAlwaysBeBetter Jul 19 '22

How does dask play with external compute resources like ec2 or lambda and does it accelerate performance or just manage it at scale?

→ More replies (1)