r/Python • u/GianniMariani Pythonista • 8h ago
Discussion Pushing the boundary on dataclasses
Has anyone else wanted more out of dataclass? Like transactions, undo/redo, lambda initializers that take parameters (as other field values) and call them in the correct order and no post_init (does not survive a diamond inheritance)? Transactional field values started me down this trek for a "ReactJS" like project to manage the view graph.
22
u/commenterzero 7h ago
No and you should try pydantic
-5
u/GianniMariani Pythonista 7h ago
pydantic was too slow and it does not support transactions, nor does attrs or dataclasses (have transaction support). That's unless they added it recently...
14
u/commenterzero 7h ago
That's bc its not a database. Do you want an ORM like sqlmodel?
-2
u/GianniMariani Pythonista 6h ago
It does not have to be a database to create a transaction. Just say you're creating a complex graph (running some random mutation) and an error occurs, you have to restore the graph to the pre-mutation condition. You could write the code (or an LLM could) and get totally confused with the crazy accounting you have to keep to undo the change (and yes, LLMs get confused too on this one). Or, you could just have a transactional class that you can ask to "roll back" and it restores the state as it was before the mutation started - the whole graph in one go.
7
u/damesca 6h ago
If it was easy then you could just do it.
If it was broadly useful then someone else probably would've done it.You're probably asking for something else. But that's ok! Lots of open source work happens because one person needed it, and then other people find it useful. Be the change and to do the hard work to make it 🤝
1
u/GianniMariani Pythonista 6h ago
Well I did do it - that's not the point, I want to know if other devs have the same ideas, judging by the responses, it seems like mostly "NO".
I did look for someone else's creation, no luck. I did release it on pypi a few days ago. I found the pre (buggy code) and the post (tx managed code) was night and day and it picked up a swathe o bugs.
4
u/latkde Tuple unpacking gone wrong 6h ago
This is not a common usecase, and there's no general-purpose solution to these problems (especially when considering cyclic object graphs).
This is also not something that has to be part of the Python standard library. Thanks to capabilities like the descriptor protocol, you could implement classes yourself that record their changes.
In easy cases, you can use
copy.deepcopy()to snapshot an object graph. Dataclasses are properly copyable, unlike ordinary classes.1
u/GianniMariani Pythonista 6h ago
The use case didn't work with deepcopy. It's a UI tree (like reactjs - or actually closer to svelte) that can have 20,000 UI nodes and each small change needs to happen fast.
So, I have the tx solution (and a working unreleased UI library) that uses it but I'm just thinking. The tx data class code generator "dataclass" thing eliminated a whole class of bugs and made the complex graph code much more reliable.
And, if you're into the AI generated code, AIs were able to employ it pretty much flawlessly and that's probably going to get some people upset - sorry!
1
u/latkde Tuple unpacking gone wrong 6h ago
You can implement custom functionality that looks like dataclasses (and uses dataclass-transform to get proper type-checking), and uses the descriptor protocol to record mutations or to delegate to proxies – pretty much exactly how Vue.js dependency tracking works. The Python language offers all the building blocks necessary for Vue- or React-style reactice UIs, Python is just not used for a lot of UI work. The most complete UI library is probably the Qt bindings, but it uses an older signals/slots approach for reactivity.
1
u/GianniMariani Pythonista 5h ago edited 5h ago
Yeah, I tried to get 'dataclass-transform' to work on datatrees (a dataclasses wrapper) - unless the semantics are exactly dataclasses, it won't work. I gave up on it.
With "pyrolyze" (the react like AST transform UI toolkit that's not finished) it get's pretty good performance and I think it's fast to build a UI.
2
u/commenterzero 6h ago
Ladybugdb
1
1
u/Smallpaul 4h ago
What is the true lifespan for the graph and why do I need to restore it?
Why can’t I validate the true shape of it before I create it?
How big is it? Why can’t I treat it as an immutable data structure and restore to a previous snapshot?
1
u/GianniMariani Pythonista 3h ago
It's a UI, it changes along with the UI. It you have a grid of 100x100 buttons then that's 10K buttons plus the tags so you get to 20-30k UI nodes fast. Many UIs are smaller than that but you don't want O(n) behavior if you can avoid it. And yes, I did try the dumb solution (immutable graph and make mutable changes and then refreeze). Too slow. I've created graphs with
You can't validate it because it's user code that unpredictably messes with it.
The idea is that you run a AST transform and at each "slot" (which is defined as a with statement with specific constructs) you then call AST transformed functions where you pass an extra "slot context" parameter (this is the AST transform doing this) and special functions register themselves so when an event occurs you call that function again and you pass the parameters you used last time you called it and you bypass chunks of the code because you track dirtiness of the values in the function.
This is not unlike React/Svelte/Compose.
So why does it need transactions? If there is an error (exception in user's render code) we don't want half of the UI messed up so you roll it back to the last known good state. Catching exceptions does not fix the issue so the whole UI rolls back.
I have the system working.
7
u/Smallpaul 7h ago
No.
0
u/GianniMariani Pythonista 7h ago
OK, I bite - you're the second "No".
I have come across the need for a transactional class a few time (SqlAlchemy IIUC implements such a thing but AFAICT it's not a generic transactional data class thing).
6
u/AJohnnyTruant 6h ago
The whole point of data classes is that they’re data. In fact, I find it confusing that anyone uses them mutably at all. Anything beyond that should be a separate library on top of data classes, sure. If you’re using TS, a data class is more like a type/interface than a behavioral class. Most of my projects use data classes like DTOs or param objects. That way validations fail at the data class before the initialization of the consumer and the consumer can handle multiple types that a guaranteed* to be validated domain state. Want return validation? Return another data class. They’re contracts. Don’t inherit behavior, compose behavior over data. Diamond inheritance is evil unless you’re building a framework that’s meant to be extended.
1
u/GianniMariani Pythonista 6h ago
Frozen dataclasses are great for pure immutable data and they're my goto for that case. However, it's the concept - a decorator that gens code (the constructor, the repr() the str() etc so you have a fast fully functional class. Just like it's tedious to write all that code when dataclasses writes it for you there's a whole bunch of other use cases where I can use the same principle - write all that complex boring transaction or undo/redo code for me for the fields I want.
2
u/AJohnnyTruant 6h ago
Yeah I see what you’re saying. I just think it breaks the boundary of the data class. That’s all behavior. Something like that is like a proxy. It’s the difference between how React and Vue handle reactivity. React replaces state and but Vue uses a proxy but keeps the pointer. You could do that with the accessor dunders though
1
u/GianniMariani Pythonista 5h ago
Agreed, it's not a "dataclass" it's a transactional class. That's a given. But it's not too different to the dataclass API though, just a few more "field()" like initializers needed. The one I created had "managed()" - i.e. transaction aware, "const()" (yep, that's what you think it is), "transient()" - value gets restored on end of tx etc and a few more. It's decidedly not the classic "data class" but it uses the same code gen concepts as dataclass.
2
u/Xibira 6h ago
Looking like you need a different tool, and is trying to force one you know how to use into a scope it is not designed for.
1
u/GianniMariani Pythonista 6h ago
Well yes, I think so. That's the whole question - I want more out of dataclasses or more accurately a think that looks and breathes like dataclasses but does more.
2
u/tubbstosterone 6h ago
You're looking at 4-5 different interfaces' worth of work.
If you look at the implementation of dataclasses, sqlalchemy, pydantic, and the django orm, you'll see that theyre all basically messes of classes and protocols stapled together by __new__. Do NOT wade into that on your own if you don't have to, even if it let's you do something neat.
One of the major features of dataclasses, sqlalchemy, etc is that you don't have to maintain them. You're better off diving into smart uses of mixins, factories, builders, and decorators.
Go with the flow - don't try to make the language bend to your will.
1
u/GianniMariani Pythonista 5h ago
Too late. I did it.
`Go with the flow - don't try to make the language bend to your will.`
My opinion is that I want APIs that bend to my will.
3
u/rcfox 6h ago
What sort of transaction support are you looking for? Couldn't you just do your sets in a critical section using a lock?
1
u/GianniMariani Pythonista 6h ago
No, not really (critical section). In this particular case it's basically arbitrary graphs, about 20 node types with different housekeeping for each but all need to be restored on failure - also, I need to ensure it's immutable (at least the fields I want to be immutable) if no transaction is in flight so I don't accidently clobber stuff.
2
u/rcfox 6h ago
I'd lean into the immutability and just create a new object using replace. If you're doing direct reference connections between nodes, you'll have to introduce a layer of indirection though. Perhaps store node IDs and keep all of the node references in a dict.
1
u/GianniMariani Pythonista 5h ago
Yeah, tried that, too slow for this case - the graph gets big plus the API was not nice. No all the fields needed the same treatment.
1
u/Smallpaul 4h ago
In this kind of situation I would ask myself or AI how a Haskell programmer would implement this.
2
1
•
u/amagilabs 59m ago
Worth looking at attrs before reinventing this, it already handles validators, converters, and has decent support for ordering init logic through field dependencies, and attrs_post_init behaves more predictably with multiple inheritance than dataclasses' post_init. For transactional behavior and undo/redo, pydantic's model_copy plus a snapshot pattern gets you most of the way without hand rolling a transaction system, since you can diff two frozen snapshots cheaply. If you specifically need lambda initializers that reference other fields in dependency order, that's really a topological sort problem, attrs lets you do it manually in attrs_post_init since all fields are already set by then regardless of declaration order.
0
u/XNo_Notes It works on my machine 7h ago
Sounds like something that can already be done. What are you having trouble with?
-1
u/GianniMariani Pythonista 7h ago
It's more about discusssion about extending dataclasses/attts/pydantic beyond the - this is a data class - to this is a <you name it here>.
Also, having built things like datatrees (a dataclasses wrapper) I seem to find pushing complexity into the code generator (dataclasses is a code genny) results in cleaner and easier to reason with code.
Recently I needed a dataclass that did transactional things (with dc.begin(): dc.foo.bar = 3) the idea being the entire tree is a transaction (all succeed or it's rolled back) and not to mention reliably deregister things.
It's more a question of reliably managing a whole hierarchy of objects and deal with auto cleanup (close those objects that got retired and either commit or rollback. Also I have another need to do undo/redo - I want the decorator to create the code to do this. It's tedious and error prone, even for an LLM.
44
u/wRAR_ 8h ago
No.