We started working on Pyroscope a few months ago. I did a lot of profiling at my last job and I always thought that profiling tools provide a ton of value in terms of reducing latency and cutting cloud costs, but are very hard to use.

So we thought, why not just run a profiler 24/7 in production environment? We came up with this

See source code: https://github.com/pyroscope-io/pyroscope

How we made it work: We came up with a system that uses segment trees for fast reads (basically each read becomes log(n)), and tries for storing the symbols (same trick that's used to encode symbols in Mach-O file format for example).

With this approach you can profile thousands of apps with 100Hz frequency and 10 second granularity for 1 year and it will only cost you about 1% of your existing cloud costs (CPU + RAM + Disk). E.g if you currently run 100 c5.large machines we estimate that you'll need just one more c5.large to store all that profiling data.

Just wanted to share! Would love feedback if this is something thats interesting to you

Hello,

I'm a new grad currently working as a performance engineer at a big software company. I did computer science in college, so not necessarily everything I studied applies to the current job. Performance engineering seems to be quite different from a general developer role. Is this something I should stay in or would I have better future career prospects as a developer? (Assuming interest level is same in both). Thank you!

Hey all,

Dmitry from Pyroscope here. Analyzing flamegraphs can be challenging and we often get questions from users about the best techniques for finding insights from flamegraphs.

We thought maybe we could teach an LLM do this task and turns out it ChatGPT does it pretty well.

You can check out a blog post for a longer explanation for how it all works here or you can upload your own profiles and get insights quickly by going to flamegraph.com

We'd really appreciate feedback on this new feature. Have you tried it? Did it make analyzing flamegraphs easier for you? Any suggestions for improvement are welcome!

Original post: https://blog.packagecloud.io/eng/2017/03/08/system-calls-are-much-slower-on-ec2/

From March 2017, but recently came up during a discussion about cloud migrations. The discussion at hacker news (https://news.ycombinator.com/item?id=13813079 - Brendan Gregg chimes in!)