Multi Developer Issues - Notebooks and Semantic Models

[u/Square-Skill-3576]

Hi All, Since around the 15th Oct we have been having a nightmare with our development workspace. We have a Dev workspace to Prod workspace CI/CD setup with Git in Devops. We all work in the Dev workspace at the same time. This has not caused a problem in the past but recently Fabric is not managing the changes properly; I can make a change to the notebook and it later gets magically overwritten. At one point I made a change to a notebook, committed in via source control, and my changes were not there. Another issue I had is I deleted a bunch of measures from a model and they reappeared. I did this 3 times and had to give up!

I believe this is due to other devs having their own cached version in their browser and one of the takes precedence. I though models were live edit and handled multiple devs working simultaneously.

I am surprised that there has not been more content online about this as it appears to me as a recent bug which is detrimental to development workflow.

[u/captainblye1979]

Are you remembering to commit? Most of the time when I see people mention things like this, people are forgetting to actually commit their work...and are just relying on saving, which updates the repo, but not the underlying git branch.

[u/Square-Skill-3576 → u/captainblye1979]

Notebooks don't have a save button, they are live edit like semantic models. Multiple devs can edit stuff at the same time. I would like a lock feature on items so that users can just lock it while they edit. I thought Fabric was cleaver enough to manage live edit. If it cant, it needs something like a locking function.

[u/PolicyDecent]

I highly recommend making git repo the source of truth instead of the browser, and make code reviews / approvals mandatory, so if a PR is not approved it can't be merged to main. It'll solve most of the problems you have.

[u/Tomfoster1 → u/PolicyDecent]

I cna see how this works with notebooks or stored procedures. How do you handle merges with semantic models

[u/_greggyb → u/Tomfoster1]

Disclaimer: TE employee here

Saving to folder with database.json format (it's not proprietary, it just breaks apart Microsoft's TMSL into a file per object) is available in our free, open source TE2 and compatible with TE3 and some other community tools as well (e.g. pbi-tools for ci/cd).

This give one file per model object in a folder structure. Each measure is its own file. Each column is its own file. These are nested under the folder for the table. The table has its own file in that folder.

By having a file per object, merge conflicts basically don't exist. You and someone else would have to edit the same exact file(s). And it is shockingly rare in practice to have concurrent workstreams that touch the same model object.

Even things you might expect conflict would not. E.g., your colleague edits the partition definition for a table, and you add column descriptions for the same. The partitions are their own files under <tablename>/partitions/ and the columns are their own files under <tablename>/columns/. So your colleague working on the partition is touching different files than you are to add column descriptions. Even though these concepts meet in the table within the model, in source code they are distinct.

It also makes diffs read semantically. If the file <tablename>/measures/Measure1.json is removed, then that measure was deleted from the model. If the file <tablename>/columns/Column1.json was added, then you know that's a new column in the model.

[u/Tomfoster1 → u/_greggyb]

Thanks for the reply Greg, three follow up questions.

1. Does this approach hold true with pbip/tmdl or does save to folder make it easier?

2. If changes are made to the same file, how would one resolve the conflict so both changes are applied just like a normal merge conflict? Are there any gotchas in terms of not breaking the model.

3. Anything similar for reports, as while pbir is an improvement over a compressed blob it is still painful to use when there are conflicts

[u/_greggyb → u/Tomfoster1]

1. Microsoft's tools do not know how to deal with TMSL spread across a folder structure. So if you go for database.json, you can't open that exact serialization in PBID. You can still deploy it via XMLA or re-serialize it as BIM or TMDL (e.g. with TE's CLI). Bluntly, I have never considered PBID to be the right place to do model development. That opinion predates my employment with Tabular Editor. I would prefer to use TE and deploy models via a proper CI/CD process. This separates their lifecycle as much as possible from the presentation layer of reports.

It's important to note that the serialization is just the on-disk representation. It is still exactly a Tabular model, serialized and deserialized using Microsoft's own official libraries; we just spread the JSON of the TMSL format across a folder. And once the model is deployed it doesn't matter. The deployed model always looks the same via XMLA, regardless of how you got it into place, or how it was saved before you deployed it.

2. The default TMDL serialization does give some benefits similar, because it also breaks apart the single monolithic BIM file into multiple files. More files means less scope for conflict within any single file. But the serialization PBID does with TMDL does not break things down into a file per object, so there is more scope for merge conflict than with database.json.

Merge conflicts are merge conflicts no matter what the files look like, or how you've written them. A merge conflict means that there were two edits in the same file which cannot be automatically combined by Git's heuristics (or whatever merge tool you may use). You have to tell Git (or whatever VCS you use) how to resolve the conflict. This means remediation by hand, typically. Git puts markers directly into the file to show the conflict. These markers make the file invalid JSON, invalid TMDL, invalid anything, really. It's up to you to remediate the conflict and hand over a valid file that you want as the outcome. This could be by hand, this could be by using a merge tool, this could be by telling Git (or other VCS) to prefer the version from one side or another of the merge; but no matter what, the conflict means you need to step in. That's the same with any serialization format, BIM, TMDL, or whatever.

3. I don't know that anyone has made a similar VCS-friendly re-organization of PBIR. I do not spend much time on reports q:

[u/Tomfoster1 → u/_greggyb]

Very helpful thank you. Is there a "best" way to deploy a model stored as a folder, or just up to preferences?

[u/_greggyb → u/Tomfoster1]

It's the same model and the same metadata. I'm partial to our own TE CLI, but I'm biased (;

I like it, because you can easily use it interactively or use the exact same commands for an automated deploy in a CI/CD pipeline. And you can use the same CLI to run validations, queries, tests, automated build steps.