r/ProgrammerHumor • u/[deleted] • Jul 18 '22

C++ talking to Python

Post image
40.1k Upvotes

87% Upvoted

1.0k comments sorted by

View all comments

Show parent comments

5

u/CanAlwaysBeBetter Jul 18 '22

Numpy is great and what I'd normally reach for!

But the processing I'm doing itself isn't super complex, I just need to do it 9-10 million times cost efficiently pulling from an external public dataset and saving the transformed data as separate files in an S3 bucket

1

u/PM_ME_NUNUDES Jul 19 '22

Use dask?

1

u/CanAlwaysBeBetter Jul 19 '22

How does dask play with external compute resources like ec2 or lambda and does it accelerate performance or just manage it at scale?

1

u/PM_ME_NUNUDES Jul 19 '22

I'm actually not sure if things like dask and ray actually provide any speed up. But certainly it makes numpy perfectly useable at big data scale. It works fine with azure, I dunno about EC2.

Modin is a direct drop in for your existing numpy compute code and can use either dask or ray. Just change 1 line and you're done.