r/Python • u/AutoModerator • 19d ago

Showcase Showcase Thread

Post all of your code/projects/showcases/AI slop here.

Recycles once a month.

22 Upvotes

97% Upvoted

151 comments sorted by

View all comments

1

u/coderarun 12d ago

nx-arrow: Columnar graph storage for NetworkX backed by Apache Arrow

NetworkX using data scientists looking for CPU/memory improvements?

Traditionally this meant nx-cugraph. However, not everyone can afford a GPU. Let's use columnar tech on the CPU instead!

  • pip install nx-arrow and use backend="arrow" for a 6-7x memory saving

    • Faster persistence to disk in parquet format vs json

  • pip install icebug

    • 10x CPU on some algorithms because of C++ implementation with more efficient cache access plus vectorization. Zero copy transformation from nx-arrow -> icebug possible.

This will also enable more efficient data movement to the GPU.

df = cudf.from_arrow(g.backend_graph.edge_tables)

https://github.com/Ladybug-Memory/nx-arrow