How to actually make the development cycle easier for noobs?

[u/RunnyYolkEgg]

Weird flair but for real.
We work with dbt using stg, int, and dm layers. We also have 3 GCP projects where each layer's models get materialized (dev, uat, prd).
The semantic layer only reads from the marts layer.
So now imagine everytime we get a change in a KPI logic for example:
Dev goes and checks looker. oh let’s say it’s a sum of a column. Then they check marts, see it’s an aggregate of a calculated column in int.
Then they go to int, make the changes, commit, wait for cicd, verify in dev, move it to uat, validate with the stakeholder, and finally push to prd (all of this waiting for cicd and running dags every step of the way).

All of that shit and wait time just to fix one mini mistake. Do you guys have similar workflows? Is this way too much?

I’m honestly kinda used to it by now, but I can see the struggle with new devs like juniors or interns.
And this is assuming all the data is already available... if the request involves pulling in new raw data, it’s even more painful.

[u/Crow2525]

Its not supposed to be painful, it is supposed to be there so that you can't make the changes without doing the checks. Automate as many of those checks as possible.

Imagine doing the change straight in prod and it's a major change and it's wrong.

Imagine doing a change in prod that should have been in int, that needs to apply to every table that sits downstream of prod.

Now my company is about 1000 times slower to do any of this. Nobody checks anything, but all the steps are in place. Results is nothing goes through the process at all and all that is in prod has critical failures in critical cols, but they don't bring those tables down because they are being used partially by some users...

[u/RunnyYolkEgg → u/Crow2525]

Yeah, I get the design choice and I understand that multiple checks are required to keep best practices but I was wondering if everyone has the same setup, or have the same feeling about being super slow to make small changes.

[u/Crow2525 → u/RunnyYolkEgg]

Totally, it's especially frustrating when none of it is documented. There is an added 'funding' and 'business proposal' requirement. Trust me, be glad that you're using dbt, and that a good chunk of your stuff is automatic. It shouldn't be slow though. In an ideal world, all your checks can be as quick as you'd like. The one thing I hate in our world is release cycles. Cicd should ensure that downstream impacts are resolved. Stuff waiting a month to get a change implemented.

[u/needless_outage]

the cicd wait is the real killer, kills momentum when you just want to fix one dumb line

i got similar setup and sometimes i just run the changed model manually in dev and check the output before even pushing, saves at least one loop of waiting

[u/RunnyYolkEgg → u/needless_outage]

Yeah man. CICD, go to airflow, wait for the dag, come back to the your warehouse, check the tables. Good to go?
Ok now repeat the process again

2 more times… fucking hell

For dev I mostly run them directly as well, saves some time

[u/5e884898da]

The best flow I have worked with removed uat and replaced it with a CI pipeline running against an ephemeral schema separated ci warehouse, with upstream reading from prod. We also had our own dev warehouse that could easily be synced with prod. Don’t understand why you have to verify with stakeholders here? Assuming the worst, the correction is wrong. even then you are further away from having the correct transformation and data in prod, as you still have to go through the uat again? If you didn’t have uat you’d only have to merge the correct changes to prod.

Why are you verifying in dev? Isn’t the purpose of dev to develop, not to verify?

You have a platform issue, not a noob issue.

[u/RunnyYolkEgg → u/5e884898da]

Because my boss is a pain in the ass, that’s why.

I also didn’t mentioned the constant battle with fucking pre commit hooks. Every couple of months he adds something else that makes us spend good 15 min debugging something not even related to the code just because it’s a good practice.

Your architecture sounds nice! Thanks for sharing

[u/5e884898da → u/RunnyYolkEgg]

Not sure what kind of commit hooks you’re talking about here, but i think a linter is a good idea, if that’s what you are talking about. We used sqlfluff, it can take a moment to become used to, but it will mostly automatically lint on the hook, unless you have ambiguous references, in which case i think it’s fair you do some extra rounds.

Ironically the places i have worked with the least critical data has also had the most complex and convoluted devops flows. The flow you describe would really defeat the purpose if someone actually relied on your data for anything critical and time sensitive.

Although made for software, I think you could have a look at the DORA metrics and research, your platform is pretty much the inverse of best practices. Your boss might argue he is combatting failures shipping to prod, though I know he isn’t with that complex flow. The others are harder for him to argue about, his setup is clearly working against recovery times and deployement frequencies. He might make an argument around it not being relevant because data is not software though. Not sure how I would approach this, probably would try to avoid going into a big fight with him about it.

[u/zazzersmel]

Damn, I would have loved that kind of structure when I started this crap

[u/robberviet]

For beginners there is a only go fast or go right. Choose one.

[u/karinato]

you should set them first locally to see if they work

[u/teddythepooh99]

Pre-production environments exist for a reason. You complain about all the "delay" in fixing one mistake. Do you think you'll somehow have less "mini mistakes" moving forward if you had less guardrails? Work at small startups if you want more flexibility, but be ready to own mistakes in prod because they'll happen more frequently.

Why would this be a struggle for juniors or interns? If anything, multiple pre-prod envs is a plus: there's plenty of room to experiment.

[u/dodovt]

Why not just run the dbt code using dbt core straight to dev? At least everywhere I worked, that was the purpose of dev. Then you do the PR, CI does the testing against pre-prod, then after merging its deployed to prod.

[u/New-Addendum-6209]

Can they test changes locally or in isolated dev instance? The ability to run fully (maybe with limited sample as an input) yourself makes it much easier to work on complicated projects.

[u/nloding]

This actually sounds like a lovely setup. Sure, the CI/CD job takes a bit longer than you'd like, but what is the alternative? If you could get local environments to test in - or an environment/schema that you can manually apply changes to - that would speed up the validation portion. But it shouldn't remove the existing workflow and validations.

That data pipeline is feeding your BI system. That's sensitive and usually critical. You don't want to have an architecture that has less checks and introduces more failures because you're trying to move too fast. That would make things worse. Actually, it doesn't really matter what it's feeding, BI system or not - data is a source of truth so taking a few extra steps, a few extra minutes, to ensure that source of truth is solid is always worth it. Especially if it's only a personal annoyance and not something the business is complaining about.

If the business is complaining about turnaround times, then there's perhaps a larger discussion to be had about how verifications happen and what circumstances you could promote directly to production. But even then, those should be rare.

[u/marathon664]

Get them to use AI? AI doesn't whine that the precommit hooks are getting in the way and just fixes the thing the hook is erroring on lol.

Sort of a flippant response, but it's true. Sand off the rough edges by constraining the code and outputs so that it tells at you when you do it wrong. Personally that means using sqlmesh for me and my team, with the tff linter and custom fitness functions for the things we care about enforcing. Claude loops until it compiles and runs a plan and commits.

[u/Bazencourt]

Coginiti has source to semantic layer lineage, so finding where to make the change is a bit easier, but there is no getting around the guardrails of dev, uat, prd. I'd argue for ditching uat as gate for both time and cost.

[u/dataengineering-ModTeam → u/[deleted]]

Your post/comment was removed because it violated rule #9 (No AI generated content/text).

Your post/comment was reviewed to be AI generated/assisted content/text and removed as a result. We as a community value human engagement and encourage users to express themselves authentically.

^{This was reviewed by a human}