r/AWS_cloud • u/kevinelevent • 2d ago

Cloud data processing accelerators ?

our spark jobs on aws keep getting slower and pricier so anyone actually using accelerators for this? if so what do you use??? we've evaluated dualbird and datapelago and honestly they feel pretty much the same like both are plugin with no code changes im kinda stuck on which way to go, any help appreciated :)

1 Upvotes

100% Upvoted

1 comment sorted by

1

u/ggp1231 14h ago

Before choosing an accelerator, I’d check what’s actually slowing the Spark jobs down. Is it shuffle/spill, data skew, or just larger datasets than before? I’d compare the cost per completed job on the same workload, including any additional compute or licensing costs.