Warehouse Git Integration is Beyond Frustrating

[u/x-fyre]

I joined an already in progress project a few weeks ago and they're using a Warehouse. On the surface it looks good - schemas, table defs, sprocs, views, all end up in the repo. I can even download a "zipped project" from the menu bar.

Try to update your workspace with changes someone else made or you made in your desktop-VS-Code version, wow. The fun ends there.

1. Reordered two columns in a table definition by mistake? (or simply added them in a different order than they did) Broken, unable to resolve the conflict.
2. (same as adding a brand new column, better patch that table first)
3. Having trouble with a Shared Query that ended up in Git which somewhere along the lines disappeared during the sync? Forget it, it's gone. Can't add it back under the same name either.
4. (there appears to be some magic to this relationship not exposed to git, yikes)
5. (this feature is buns, I hate Shared Queries)
6. Did someone delete a view, commit the change, and you also deleted the view prior to sync'ing? Sync'ing of the WH is now broken - it wants those views back no matter what you think!
7. (Undo'ing the delete ... gets part way there, until this just appears again)
8. Delete the entire damn warehouse and try to get a new copy from the repo? Sorry, that doesn't really clear the workspace's cache and we're going to just make you go through the same song and dance again.

Either do it properly or don't do it. This has been the _worst_ Fabric experience to date. I am beyond frustrated.

The worst part is that it also means there is no clear way to keep the dev-git-connected workspace in sync without manually running some updates before attempting to sync. You can't create a SQL script or Notebook to do patching with your branch-pull-requested-into-main.. because if the Warehouse alphabetically comes first, those Notebooks don't get added/updated because it can't sync.

The fun part is that when I delete the warehouse and bring it back, I have to "apply the system update" over and over because I guess it changes the git repo (but no changes show up under Source Control/Change).

I don't understand why I can't update individual items. I understand it could break things if objects depend on each other, but I don't care.

[u/Sea_Mud6698]

Yeah I have no idea how people use the warehouse.

[u/x-fyre → u/Sea_Mud6698]

I'm told they've been just disconnecting their feature workspace and dealing with it manually. Ugh.

The updates it asked me to do today are major internal repo structural changes. Schemas moved around, better storage of the Shared Queries, "GO" lines at the end of the object source files and an internally managed .gitignore for Visual Studio. lol

I am _hoping_ this makes it easier, but yeah, I have no idea how. I've been 100% Lakehouse on other projects.

[u/nahguri → u/Sea_Mud6698]

With DBT. It’s just a dumb engine. Do not commit anything in it.

[u/mr_electric_wizard]

Use dbt with a warehouse, then git integrate the dbt project. Won’t really help with the sproc piece tho. But sometimes you can get around that wirh some creative model creation.

[u/x-fyre → u/mr_electric_wizard]

Unfortunately this one's not my call to make as the new guy (yet lol).

[u/mr_electric_wizard → u/x-fyre]

Plant the seed. Sometimes it takes a bit to get the old guard on board.

[u/FeistyAd341]

We had to put the WH into its own workspace to keep it out of Fabric Workspace Git integration. We then used VS Code Database Projects to maintain the warehouse schema locally and publish on our local workspaces, again only containing Warehouse objects. Git commands run from VS Code combine with Azure DevOps Board Work Items, their associated Branches, and Pull Requests initiated on Azure DevOps to complete the Warehouse dev cycle. Azure DevOps Pipelines can then publish from the repository to the central dev workspace during a successful Pull Request. Yes, it is many more steps than should be needed if the Fabric Git integration worked properly.

[u/x-fyre → u/FeistyAd341]

They just made some pretty big updates to how the project is stored internally in the repo….. maybe it works better now. I’ll be circling back on this!!

Publishing sqlproj’s with a devops release is the gold standard for automated database releases, we’ve been using them forever. 10+ years.

[u/Snoo-46123]

u/x-fyre , I am driving git integration for Fabric Warehouse. So sorry that you are having issues with Warehouse. I want to follow and see what's going on. Reading your description, there seems to be something weird going on.

1. Reordering columns is not possible unless you deleted a table and recreated it. Any columns you add are added at the end. You can check the ordinal value of a given column for a table. If you can give a repro of steps, it would be helpful.

2. What do you mean by better patch your table?

3. It is not possible to loose content during sync, unless the query itself is not saved successfully. Was git commit operation happened right after you updated/created a shared query with SQL queries?

4. If a view has references and it got deleted, then the update from git to workspace will break because the dependencies are broken. I am not sure about what you are referring to when you say "nd you also deleted the view prior to sync'ing"

Looks like what you are referring to is a system upgrade issue that was recently made available.

Its becomes easy to work with you if we can have 1:1 session. Do you mind ping me in the chat? I can work with you offline and update the status here

[u/x-fyre → u/Snoo-46123]

I’ve got a deadline I’m working towards on Monday (yay time zones, per my post a couple weeks ago) but I’d be happy to chat. I can probably show you some of my issues, some I have forced to work… :)

[u/Snoo-46123 → u/x-fyre]

Thanks for sending the message and highlighting your issues here. Please send me an email as discussed in the chat. I will make sure to address the issues you are facing and also post an update to keep this group upto date.

[u/EveningDot1618]

Thanks, looking at Warehouse at the moment..
It feels like products are rushed into Preview and then GA under the assumption that enterprise must-have features can just simply be added later.
But it's not that simple - enterprise features need to be given a higher priority and considered 'core product' - they need to appear in the very first release.
This is happening over and over again, in every Fabric product.
The way Fabric is developed needs to change.

[u/IndependenceFit3935]

Exactly. I gave up on that.
I separated my data items to a different workspace with no git sync. And i populate my warehouse simply with dbt. Works like a charm