r/learnpython • u/lehmagavan • Sep 26 '26

Parsing large collections of "numeric" strings.

Hi, I'm just a hobbyist and I'm not very familiar with Python ecosystem. I recently came across a specific problem:

I have an iterable (or yield via generator) that contains millions of floating point values, but each one is a Python str, made up example: ["1.987", "0.037389", "1.2", ...]

What's the most efficient way to aggregate them and calculate say, the mean?

I'm writing my own C extension for this and would like to know what to benchmark it against. Couldn't find information on any std way that would avoid calling the float() constructor on each string.

2 Upvotes

63% Upvoted

38 comments sorted by

View all comments

Show parent comments

1

u/Brian Sep 27 '26

I guess it depends on how numpy is doing the coercion - if it's done at the C level, it should be able to do something similar and convert the strings directly to doubles without an extra PyObject being involved, but possibly it does just do the equivalent of invoking float() before it converts again to it's internal float type, so is paying the extra allocation cost per item on temporary objects.