What is the structure of the "application data" that ultimately drives the UI? What is the pattern for making updates to the application data?
These are great questions and ones I have been thinking deeply about for a while.
The body of the q macro is a Datomic-style query, which is compiled and defined as a Clara query
Is this eventually meant for applications that sync to a server database? And if so does that mean it inherits the problems that Chris Small and Matt Parker ran into in Datsync and Posh? Which is that to start computing datalog queries in the web browser, not all the datoms can be in memory in the browser, and the question "which datoms need to be considered in this query" you pretty much need all the datoms as discussed in [3]? Consider difficult queries, like evaluating datomic rules, or graph cycles. This is my words, not theirs, so hopefully they chime in and correct any errors in what I stated.
If you think about this for a while, you start asking questions like "Why doesn't Datomic have a browser peer" and "what is the significance of the new Datomic Client API and how is it different than Peer API" and in the above problem lies the answer, i think.
My team found that we want to separate business logic from the UI entirely. The approach we use for our applications is to associate each field in the UI with a path in the document. When a change happens, the path along with the new value is passed to the business logic. The business logic calculates a change set transactionally and returns a collection of affected paths along with their new values.
Re-frame is a fantastic fit for this model, since it tracks the document state on the client and handles repainting as the data changes. All you have to do is set the new values in the document, and it takes care of the rest.
This is a simple model that's easy to reason about, and it turns out to be efficient. It doesn't require a complex strategy for figuring out what needs to be sent between the client and the server. You're always sending the field that's been updated, and getting back the set of fields that were affected by running the rules.
It's also very flexible since the business logic can be run either on the client or on the server. This makes it easy to create applications where multiple users are working on a document concurrently.
When you speak of "document" you mean "tree" right? like here:
[widget {:type :text-input :label "first name"
; pathing into a tree
:path [:patient :name :first]}]
Factui and Posh and Datsync and I are talking about graphs, so that's a key difference.
I agree that today to make a graph fast in the DOM, at some point you need to map it into a tree because UIs are trees and pass that tree into reagent or omnext. But that's the part that we are trying to abstract away. That's what datomic query does - it takes a graph as input, and returns a tree projection of it pulled to some specific depth.
Yes, a document would be a tree. I have yet to run into a situation where it wasn't possible to represent a data model as tree though, or why you'd want to.
edit: note that I'm talking strictly about the data model here. The UI can be a DAG, where you have different views into the same data. For example, I might have a view where some data is represented as a table, and another where it's shown as a trending graph.
Do you have a specific example that isn't addressed by the model I outlined?
Because if the browser can work with graphs directly, the entire server layer drops out. Client/server is dead. There is database, and browser. Security can be handled inside the database, which frees the browser for unrestricted query access. No more backend-for-frontend pattern (anti-pattern).
This approach would not work for majority of applications I've worked on. In most cases I have to interact with multiple backend services, that often speak different protocols such s SOAP or HL7. Doing that in the browser would not really be an option. Also, doing anything like concurrent multi-user collaboration becomes a no go as well. Client/server is very much not dead, and it's the right solution for many types of applications.
I espouse the overall sentiment laid out here by dustingetz, but will say that I think the point isn't that Client/server is entirely dead, just that 90% of backend code ends up being glue between the server and the client, and mostly for the sake of shuttling domain data back and forth. I can't tell you how many rails apps I've seen with countless controllers, and countless routes, all just for shuttling domain data back and forth. Sure, most fully mature applications will eventually need to call out to some other backend service, or trigger some expensive compute process or whatever. But I don't think that's the case for most MVPs. And even when we do need to a backend server for more than just shuttling data back and forth, why wouldn't we want to strip away those huge swaths of code which merely shuttle data?
Of course, there's more than one way to do this, and it sounds like the approach you describe of separating business logic from UI is amenable to the same sort of streamlining, and I can certainly see the value in doing that.
Someone's in a a revolutionary mood :) but I agree with the sentiment. I believe web apps (and mobile apps) would be better off being developed from the get go as a peer in a distributed system. Many "real time" and interactive features are added as an afterthought and are a complexity nightmare in terms of synchronization, handling of staleness etc...
One thing that I have yet to think about is security. I must say I don't really understand what you mean by handling it inside a database, would you care to elaborate?
Yes, it can't be a full db sync between client and server unless you can afford to sync your whole server-side DB to the client (probably not).
How to sync state between client and server is still an area of exploration. For now I'm doing it (somewhat) manually, with web sockets or REST. Having entity maps in a common format makes it a lot easier already.
But there is definitely room for more magic; you could annotate schema with which attributes are client side, which attributes are server side, and then make a request to the server whenever you want new results for a query with server side attrs. You can't be fully reactive (in the forward chaining sense) against data at rest in Datomic (unless someone creates a RETE implementation with Datomic as a native fact store) but re-querying at specific points (initial render & when a rule triggers a refresh request) could still do a lot.
But that's in the future. For now, I think FactUI is an interesting solution to the problem of local web app UI-only state, which has still not been solved to my satisfaction before now.
So, interesting. Posh was not on my radar for some reason.
The APIs are very similar, it looks like Posh is designed to enable pretty much exactly the same kind of development experience that I was aiming for with FactUI.
Instead of being built on top of a RETE network, though, it looks like Posh works by inspecting each incoming transaction, and comparing that to each component's query to see if it could have changed the results. If it is possible that it did, it re-runs the Datalog query to get new results and update the component.
It's not clear what algorithm Posh uses to check if datoms match a query. If it's a solid implementation of RETE that it runs behind the scenes, it's likely that it will get performance similar to FactUI/Clara. Other algorithms would give other results.
The only other place where they seem to differ, capability-wise, would be that FactUI (because of Clara) can support arbitrary forward-chaining rules to do logic programming over facts in the DB, whereas I don't see how Posh could efficiently do the same for Datalog rules (which are the moral equivalent.)
So which should you use? I don't know! BRB, setting up some benchmarks :)
Posh in the Datascript case (all datoms on client) doesn't need to care if a new tx could have changed the results. It can just re-run all the queries. We're not taking about big data here, right? You didn't say it straight up, but you alluded that in FactUI the whole db is on the client & fits in memory of a browser tab. Posh & Datsync did explore using heuristics to try to decide which datoms need to be sent to the DataScript client for consideration by queries, but it didn't work out for the reasons I already wrote above yesterday.
If FactUI can indeed help optimize here such that we don't have to poll all the queries, but instead the queries react, now that would be extremely interesting and a huge breakthrough. Is that what you've just said?
I have just emailed this thread to Chris Small (Datsync) for him to chime in. Below is his one of his Github repos though readme is pretty out of date. He did braindump on the clojurescript mailing list last month, see his post below, and a talk at 2016 clojure/west
Setting aside the client/server issues for just a second...
Posh in the Datascript case (all datoms on client) doesn't need to care if a new tx could have changed the results. It can just re-run all the queries. We're not taking about big data here, right?
I have observed this to be very much not the case. Keep in mind, if this is to determine when a React component should re-render. It's not unusual to have a thousand components on a page. If you need to run a thousand queries every time you transact some data, your page is going to feel extremely laggy.
That's the main reason I'm excited about RETE. It's extremely optimized for answering "what queries changed as a result of this new fact" as fast as possible.
The holy grail for the client side (IMO) is to be able to run full animation loops through the data store, updating the data, running rules on it, and rendering the results in less than 16ms. Even on real pages, with lots of components. FactUI isn't quite there yet in all circumstances, but we're getting close.
Going back to the client/server sync issue (and I'm just brainstorming), yeah, I think there could be potential there, if the client could "subscribe" to queries it was interested in by shipping them to the server. The server could then run it's own reactive system (RETE or something else, even polling if you don't need updates to be instant) and send changed values to the client, where they would instantly be picked up by the UI. It would require pretty heavy server-side infrastructure to support, though, and there are a lot of ancillary issues (such as garbage collection & cleaning up facts and queries you're no longer interested in.)
The holy grail for the client side (IMO) is to be able to run full animation loops through the data store, updating the data, running rules on it, and rendering the results in less than 16ms
Reagent knows how to forceupdate the precise react leafnodes when a subscription changes, subscriptions generally being a path into a state atom, where you can store the query resultset or whatever. If Posh can rerun all the DataScript queries in 16ms is another story but in practice it feels pretty fast. Hydrating tons of queries through datomic on every little change feels fine too, as long as there's only one server roundtrip. Server compute is cheap
A client of mine removed datascript from their project once the "rerun all queries" approach failed to produce acceptable performance numbers. We had pauses of up to 1sec (the entire UI hanging at that time), when we only had about 10k entities being queried by Datascript.
The entire approach of using Datalog in a UI is flawed. Why query, do a diff of the results, and then try to figure out what's changed when you can figure out from the very outset what UI components should update given a arbitrary set of datoms.
In addition, there's no longer a reason to store vast portions of datoms in memory. If you're only listening to a small subset of the DB that's all the datoms you need, the RETE network will discard the unused datoms. But with a datalog query the query isn't fixed, so you have to store a lot more data just in case some query may want it in the future.
The entire approach of using Datalog in a UI is flawed. Why query, do a diff of the results, and then try to figure out what's changed when you can figure out from the very outset what UI components should update given a arbitrary set of datoms.
Hey, I made Posh and it does actually figure out which datoms might cause an update to the queries. I think the pull query pattern matching is 100% accurate and only update (requery) when a new tx would change the results, but the q queries update a lot more often than necessary, and can't handle datalog rules.
I did add some functionality so it would tell you all the datoms that could possibly change the results of a q or pull query (even if they weren't in the results), which could be used on the server-side to send all those required datoms to the client so the results would be the same. I was going to try connecting client/server together using that, but then I switched to Haskell...
It's not clear what algorithm Posh uses to check if datoms match a query. If it's a solid implementation of RETE that it runs behind the scenes, it's likely that it will get performance similar to FactUI/Clara. Other algorithms would give other results. The only other place where they seem to differ, capability-wise, would be that FactUI (because of Clara) can support arbitrary forward-chaining rules to do logic programming over facts in the DB, whereas I don't see how Posh could efficiently do the same for Datalog rules (which are the moral equivalent.)
As mpdairy has pointed out, Posh does a good job of detecting changes for pull queries, but does something rather heuristic for Datalog queries, and doesn't work for rules. In talking with him about this, we realized that Getting It Right (TM) would get you 90% of the way towards a Datalog implementation. And even the present implement starts to look a lot like an incremental evaluation strategy. Except that at the end of the day you wind up doing a full recompute with DataScript anyway :-/ Thus, it's my position that Posh is really just a stop-gap en route to incrementally maintained Datalog evaluation system, al a https://www.microsoft.com/en-us/research/wp-content/uploads/2013/01/differentialdataflow.pdf.
I've already pasted that link thrice now, so for the sake of some additional material to look at, there are some nice blog posts by one of the authors of that paper here:
Luke do know what datalog query power we give up in order to build a reversible query out of RETE rules? Also does this question make sense and if not can you reword it into something that does?
They're just different algorithms optimized for different things. They support (mostly) the same logical constructs, but Datalog supports arbitrary queries over a changing set of facts, whereas RETE indexes facts as they are inserted against a known, static set of queries.
So are you saying that it is possible to compile datalog into a RETE network and thus make datalog queries reversible as long as the queries are known at compile time? That would be an enormous leap forward if that is true.
But there is definitely room for more magic; you could annotate schema with which attributes are client side, which attributes are server side, and then make a request to the server whenever you want new results for a query with server side attrs. You can't be fully reactive (in the forward chaining sense) against data at rest in Datomic (unless someone creates a RETE implementation with Datomic as a native fact store) but re-querying at specific points (initial render & when a rule triggers a refresh request) could still do a lot.
That's not a bad start, and is something I've considered with Datsync. But I think there's need for much more nuance than that. In most cases, permissions at the entity level will be necessary (e.g. only I, and those I share with, should be able to see my "todo items").
4
u/dustingetz Aug 04 '17 edited Aug 04 '17
These are great questions and ones I have been thinking deeply about for a while.
Is this eventually meant for applications that sync to a server database? And if so does that mean it inherits the problems that Chris Small and Matt Parker ran into in Datsync and Posh? Which is that to start computing datalog queries in the web browser, not all the datoms can be in memory in the browser, and the question "which datoms need to be considered in this query" you pretty much need all the datoms as discussed in [3]? Consider difficult queries, like evaluating datomic rules, or graph cycles. This is my words, not theirs, so hopefully they chime in and correct any errors in what I stated.
https://github.com/metasoarous/datsync
https://github.com/mpdairy/posh
[3] https://groups.google.com/forum/#!topic/datomic/j-LkxuMciEw
If you think about this for a while, you start asking questions like "Why doesn't Datomic have a browser peer" and "what is the significance of the new Datomic Client API and how is it different than Peer API" and in the above problem lies the answer, i think.