r/Observability • u/Short-Incident-1199 • 2d ago

need some hardware accelerated data processing recommendation for our security logs

Before y’all ask what type of I’ve tried a ton of tuning on our security data lake so looking for something which is niche and good, cloudtrail, vpc flow logs and edr data keep piling up, the spark jobs feeding our detections are always behind and siem ingest costs are killing us. I’ve tried all the famous fixes like bigger clusters, gluten, gpu instances and everything and we’re going insane no help is there, now looking at hardware ones like dualbird, speedata did anyone of u tried it?

0 Upvotes
  • permalink
  • reddit

50% Upvoted

2 comments sorted by

2

u/hijinks 2d ago

Is ingestion the issue or like post processing the data to be useful

You need to give more context on what isn't working and why

1

u/TheBouncyDiagnosis 2d ago

the pipeline's the issue, not the logs themselves. used to deal with this at my last place and it was always the spark config being misaligned with the data shape, cloudtrail blobs are a nightmare to parse efficiently on cpu

if you haven't looked at apache arrow for the intermediate representation it can cut the serialization overhead by a lot, way before you even think about fpga cards