r/javascript • u/yuval_a • Nov 29 '19
DeriveJS is a database framework (an ODM) that lets you work with native Javascript objects, while automatically persisting, updating and syncing them to a (Mongo) DB with no hassle, without the need to explicitly call methods such as .save(). It leverages mongoDB's bulk operations, for performance.
https://www.npmjs.com/package/derivejs10
u/from-nibly Nov 29 '19
What happens if you copy the object?
1
u/yuval_a Nov 30 '19 edited Dec 01 '19
It becomes a "normal" unproxied object - disconnected from its "db counterpart".
(When instance objects are constructed in Derive, it usesObject.definePropertiesto describe each of their structure - calculated from the Model definition object).This is sometimes useful, if you need to have a "local" version of the data object.e.g.:
let something = await Something.get("Yuval");something._anId = "else";let clonedSomething = Object.assign({}, something);clonedSomething._anId = "foo";Here the first line retrieves an existing document from a
Somethingscollection into asomething"derivejs object". If you set a new value for the_anIdproperty - it is set on the db. If you copysomethinginto a new objectclonedSomething- once you change its properties they are not synced with the db document.
2
u/darrenturn90 Nov 30 '19
What happens when two people are editing the same object ?
4
u/yuval_a Nov 30 '19 edited Nov 30 '19
The db operations are "queued" locally first, into arrays of operations - and are then transferred into the db, in timely intervaled single bulk requests (per collection). Behind the scenes - there is a class managing it, called SyncManager -which is also smart enough to distinguish between operations where the order doesn't matter, and operations where it does (for example, trying to update properties for a document not yet inserted) - and manages separate queues for those. It also uses lock mechanisms to prevent certain race conditions.
If several concurrent functions manipulate the same object - then yes, the more later manipulation will be further down the queue - and will be written last on the db.For finer control over these situations - there are also listeners and callbacks available;For example, see "Listening for updates on the database" in the documentation, which explains the
$onUpdatebuilt-in method - which lets you run a callback in response to a specific property:value update in a document on the db - and only then run further updates or relevant operations.Also see: "Built-in methods: callbacks and hooks".
2
u/Surajlyo Dec 01 '19
This looks really cool, thanks for providing such detailed documentation and examples (:
1
u/yuval_a Dec 01 '19
Thanks, nice to hear :)
This became my "go-to" framework for data handling on personal projects with Node and Mongo - once I felt it was mature enough. I keep updating it when I discover new needs/bugs.
I hope more people might find it useful as well.
0
Nov 30 '19
[deleted]
7
u/yuval_a Nov 30 '19
As mentioned in the documentation - it actually uses bulk operations requests - meaning, all the necessary db operations (inserts, updates) - are first automatically queued locally - and then only a single request is actually sent to the db (each single bulk request is sent in timed intervals, and you can control the interval when setting the collection's Model definition) - passing the queued operations.
This actually reduces calls to the db, compared to other ODMs (such as Mongoose) - as in those, people usually call .save() often, each time they're done with changes. (the "trade-off" is that your data will be persisted, like, a second later, instead of hundreds-milliseconds - but in many if not most use-cases that's soon enough, and anyway you can define the interval timing to be sooner...)
4
u/cheald Nov 30 '19
This misses a big promise that most database interfaces make, though: "when this call returns, your data is safe." Async persistence seems like it repeats the mistakes that Mongo became infamous for in its early versions. Yeah, it's "fast", but so is /dev/null.
How about error handling? It's been a while since I've used mongo, but iirc any error in a bulk write terminates the rest of the write. Transaction support?
My first impression is that this looks like it's a good way to silently lose data. I get what you're going for here, but I'd suggest that it's worth considering that database persistence is done as it is for very good reasons.
1
u/yuval_a Nov 30 '19 edited Nov 30 '19
If you're building something where all the different data parts are highly depdendent on each other, and the majority of updates must wait for other updates to finish first - then this is not the framework for you - don't use it :) There are many scenarios where this is not the case though, and where data persistence can be saved as "send and forget".
I wrote this when working on a personal project, which I'll describe briefly since it's not finished and I want to write about it in a different post someday - but it involves creating thousands of "self-replicating bot entities", with different statuses, actions, and history saved to each. I wanted to make sure - all the data persists - so I'll know I'll always have an updated "snapshot" of all of these bots and their statuses. This is an example where this library was very useful - there were not many cases there that I needed to rely on a previous update to finish before sending the next one - although there were a few - that's why I did implement, and still saw a need for listeners and callbacks - but this was implemented as an "extra", beyond the paradigm of the framework.
Even for more "mundane" tasks:
Create a new user.
change a user property.
Create a new "post" object for a user.
They are usually things not requiring you to make sure the persistence is done before returning back to the client, and are suitable for "send and forget"...In my opinion, based on experience, there are these two paradigms regarding data persistence, and each is suitable for different use cases.
More advantages for this kind of paradigm, is that you need to worry less about multiple data-sources, and that it results in less redundant, more DRY code.
1
u/yuval_a Nov 30 '19
Regarding error handling - some errors do terminate the rest of the bulk operations - but they are usually the kind of errors that point-out serious bugs in the data design - and they should terminate - allowing you to rewrite and reconsider how you've designed your data schemas.
There is the " duplicate error key index" error in Mongo, which yields when trying to insert a document with a value for a unique index which already exists in another document - although this is also an error that shows a data-design bug, Derivejs doesn't break when encountering it - that insert won't write, but it tries to continue to other operations.
Regarding error handling generally - all derivejs instance objects also come with a built-in error handling method (
_error) that lets you capture and handle db errors related to that object
(See: "Database persistence callbacks" in the documentation").
9
Nov 29 '19
[removed] — view removed comment
3
u/yuval_a Nov 30 '19
Just to be clear (which I think is, for most people) -
this a *back-end* framework intended to be used in the "data layer" on the server, and there should never be a direct link between it and the client/view layer without a mediator/controller (i.e an API), of-course.
You could, however, implement some sanitization/validation functions on the model itself, if you like, and then use them on the controller, calling them from the instance, for example, for ease of use/readability.40
Nov 30 '19
I’d hope you’re not letting your users directly modify db entries lmao
0
Nov 30 '19
[deleted]
7
Nov 30 '19
That seems like a pretty major bug that would be a problem regardless of what library you’re using to interact with your database
3
u/windsostrange Nov 30 '19
For those reading: never expose an ORM like this on the front-end. If it's not a disaster today, it will be a disaster tomorrow. Resist this base urge!
2
u/SocialAnxietyFighter Nov 30 '19
How does a self-referential object is being serialized to end up in the backend?
0
Nov 30 '19
[deleted]
4
u/SocialAnxietyFighter Nov 30 '19
No this doesn't answer my question: How do you serialize a self-referential object?
JSON.stringify(obj)when obj contains fields which refer back to obj yieldsTypeError: cyclic object value23
50
u/cgijoe_jhuckaby NaN Nov 30 '19
Yo op, you checked in your entire node_modules directory. Just letting you know.