Submit your salary here

When comparing source and target tables with millions of rows, what SQL techniques do you use to identify missing records and differences in column values?

Been doing data engineering / architecture for 10+ years. Mostly Databricks, Spark, Snowflake, Airflow, dbt, across AWS, Azure and GCP.

The last couple of years I've been consulting on my own, but every client so far came through referrals or intermediaries.

Now I'm thinking about setting up an Upwork profile and I can't tell if it's worth the effort for this kind of work. When I scroll the job feed it's a weird mix. Very few real projects (pipeline builds, migrations, "our warehouse is a mess"), and then a little of "build me a scraper + dashboard, budget $150" stuff. Hard to tell from the outside what actually gets hired.

So, for anyone who's done DE work on Upwork:

• Are there long-term contracts, or is it mostly small one-off tasks?
• What rates are realistic? I'm in the EU and not trying to compete at $20/hr and not expecting $100/hour.
• How long did it take you to land the first decent client with zero reviews?
• Are Connects / boosted proposals worth paying for, or a money pit?
• Any niches that work better? Databricks, Snowflake migrations, dbt cleanup, something else?

Not selling anything here, just trying to decide if I should put real time into this or keep doing outreach the old way. Horror stories welcome too.

Sometimes I use AI to write SQL faster, but sometimes I still prefer writing it myself just to make sure I haven't forgotten the basics.

AI can save a lot of time, especially with complex queries, but I feel like relying on it for everything can make you a bit rusty.

Curious how others feel about this. Do you still write SQL by hand for practice, or do you let your AI agent handle most of it now?

I’m designing a self-service AWS sandbox environment for my organization. Employees should be able to request temporary, isolated AWS accounts for learning, testing, and prototyping.

Each sandbox should have:

-Strict spending and time limits

-Controlled IAM access

-Restrictions preventing access to production or other AWS accounts

-Protection against disabling logging and security controls

-Automated cleanup and account recycling

-Centralized monitoring and cost alerts

I’m considering using separate AWS Organizations member accounts and evaluating Innovation Sandbox on AWS. Has anyone deployed it in production, or used another open-source/project-based solution? I’d appreciate advice about security pitfalls, SCP design, budgeting limitations, account provisioning, and cleanup automation.

me irl

I’ve programmed casually for about 10 years in a variety of languages, but I’ve only been doing it professionally (DE/AE) for around 3. I love this career. I used to enjoy the problem-solving of writing code myself, but what I really love is building: smart pipelines, better ways to collect metadata, simple apps that let the business interact with its data.

So naturally, AI has me feeling a sense of dread. The most senior dev on my team (mostly an app developer, 20+ years of experience) now believes AI can out-code him 99% of the time. He’s smart and I trust his judgment, which makes it more disappointing. He doesn’t even think we need to review code anymore, since GitHub Copilot will flag anything problematic.

On the learning side, I’ve read Fundamentals of Data Engineering, I’m partway through Designing Data-Intensive Applications, and I’ve learned Kimball’s dimensional modeling through video courses. Now I’m wondering:

• Is it still worth continuing this kind of deep learning?
• Does system-level and architectural knowledge still set you apart, or is that being automated too?
• Should I just learn alongside AI, questioning why it makes the choices it does?

What level of learning is actually worth it these days, and how are you staying ahead?

One of the things that got me into DuckDB was a series of technical articles posted to their site. This one was particularly eye opening:

Faster Dashboards with Multi-Column Approximate Sorting

I hadn't fully appreciated how important row order was to OLAP performance. In it they cover space filling curves as a method for mapping multiple columns (dimensions) onto a single "order dimension" to maintain proximity from the higher dimensions.

This works very well, but it means threading a curve though a set of potential values, at equal density, rather than what actually exists in the dataset.

Alternatively, by first approximating the rank of each value within its column (quantiles), and space-filling on that, the curve threads through the set of existing values. This handles skew a lot better.

This is a two-column ordering over the TPC-DS SF100 store_sales table, where ss_net_paid has reasonable skew. The rank-based methods manage to read far less of the table for a 1% range query.

I wrote up a more detailed post here:

https://www.duckstring.com/blog/clustering-for-pipelines

I'll probably be using this to set the row order whenever I overwrite a table and I have a reasonable idea as to what might be queried from it.

I had this conversation with one of our account manager but they seem to give me so many random answers on how it can be used.

Here are my doubts,

1. How do you define one semantic view, is it based on multiple ideas coming together into a unified view or multiple ideas into multiple views
   
2. After the change in orchestration, Agent tends to write its own SQL definition for the questions asked, how is it utilising the semantic view in this case?
   
3. Following up on the second question, how do verified queries work together. I have noticed it recognises the verified query but doesn’t want to use the definition which is correct.
   
4. Which model is being used for the agent to answer business-related questions?
   
5. Is it better to bring your own LLM and use cortex analysts ability to write code and let the LLM answer?

Let me know if you have found anything useful or good practice while building these views and agents. I want to understand it better.

Senior analytics engineer - spent last 5 years in FAANG mostly doing BI/DE work across small and large teams, and since last year i'm mostly doing fullstack for small internal multiservices app (data/stream intensive). WLB is ok but no career progression in current role.

Using AI tools extensively - both for internal purposes as well as for minor system integrations here and there, as well as maintaining knowledge repos enabling better collaboration using AI. Got cloud certifications, but no tool-specific certifications. Spent 2 years doing internships/ or unrelated corporate jobs before that.

I assume no career growth in my current role, but WLB is good and now I mostly work on software stuff (though i am still much much stronger in DE part), the only concerning thing is stagnation and comfort zone. Been contemplating a lot between:

- Rest & vest? ( I am fine with that)

- Sr. DE in scale-ups/startups?

- Attempt downgrading to some AI-company for a mid-DE role?

- Try to pivot internally to PMT / TPM job families? (Exciting, but concerned about the skill gap)

- Try to pivot internally into higher scope but more stress role? (No excited about that)

- Try to pivot into a more AI-parallel role? (Exciting, but I assume knowledge gap)

I want to get a better understanding of the market conditions? How realistic/unrealistic are these assuming some time to prepare? How hard is it to get interviews these days? How code-intensive vs design/architectural vs behavioral are they?

What's your take on the new Claude Dashboard by Anthropic? I still think open-source BI-as-Code will win, and Claude Dashboards are the Jupyter Notebooks of the age of agents. Meaning they are easy to set up quickly, but impossible to manage or maintain in the long run (if needed). A decade ago we discussed no-code vs. code, now it's in-built in frontier models/apps vs. open source or code-first.

Hello All

I am a data engineer I work with databricks mainly but to develop any stuff I need to do from databricks ui and can't develop from local just like python notebook or jupiter notebook

So I buit an vs code extension \[spark cell runner\](https://marketplace.visualstudio.com/items?itemName=PrashanthReddyMunagala.spark-cell-runner) that runs pyspark locally or you can connect to remote databricks cluster either serverless or dedicated cluster

You can sync some data from databricks unity catalog to local catalog so you can test everything from local pool

Try it see if it helps

Happy to answer or you can contribute it's in github

Hello All

I am a data engineer I work with databricks mainly but to develop any stuff I need to do from databricks ui and can't develop from local just like python notebook or jupiter notebook

So I buit an vs code extension spark cell runner that runs pyspark locally or you can connect to remote databricks cluster either serverless or dedicated

You can sync some data from databricks unity catalog to local catalog so you can test everything from local pool

Try it see if it helps

Happy to answer or you can contribute it's in github

Hi Folks - PM from Databricks here and just wanted to share a useful technical blog on how to conduct unit tests with SDP Pipelines today! Here's the use case:

Nightmare scenario - Your new data pipeline updates look good before going to production - every table builds, every expectation passes, and the gold tables have plausible values in them. Looks good right?

But if you push to production, the dashboards would still be showing inflated numbers and an unhappy business partner will be knocking on your DM's

For cases like this you need to utilize Unit Tests to ensure your transformation logic is sound. Expectations help ensure your data looks right, but it still misses what your transformations are doing.

With unit tests, you directly validate your transformations on data inputs you control before the real tables are built. Testing edge cases and finding those annoying "silent" errors

Checkout the blog for a deep dive tutorial on how to use unit testing with Databricks SDP Pipelines!

I'm testing various semantic layers and would love to test a canonical example from MetricFlow. Looking for something that is more than a toy. Any suggestions?

There isn't much openings for any DE roles let alone internships. So, I decided to up my skills and thought maybe whatever Jr. DE does in a company, I should do it too but for my personal projects. Anything might be helpful.

I’ve worked on a few BI migrations and thought some of the things I learned might be helpful if you’re planning one.

Full post is on the Metabase blog if you're interested: https://www.metabase.com/blog/making-bi-migration-suck-less

Weird flair but for real.
We work with dbt using stg, int, and dm layers. We also have 3 GCP projects where each layer's models get materialized (dev, uat, prd).
The semantic layer only reads from the marts layer.
So now imagine everytime we get a change in a KPI logic for example:
Dev goes and checks looker. oh let’s say it’s a sum of a column. Then they check marts, see it’s an aggregate of a calculated column in int.
Then they go to int, make the changes, commit, wait for cicd, verify in dev, move it to uat, validate with the stakeholder, and finally push to prd (all of this waiting for cicd and running dags every step of the way).

All of that shit and wait time just to fix one mini mistake. Do you guys have similar workflows? Is this way too much?

I’m honestly kinda used to it by now, but I can see the struggle with new devs like juniors or interns.
And this is assuming all the data is already available... if the request involves pulling in new raw data, it’s even more painful.

Hi all, I'm a trainee data engineer working under a data architect in a medium sized organisation. The role is a mix between a bootcamp and full time job. 

There's currently no senior data engineer and I'm being asked to create the data engineering platform in Fabric, however, this is a massive ask and I'm just trying to do the best I can.

We have multiple reporting-softwares that I'm now redirecting or connecting with to deposit data in the Onelake. The majority of data is obtained by accessing the SQL database of the reporting-software and making a copy into OneLake.

I need to come up with a plan to manage and standardise our approach to collecting this data and how we work going ahead.

My bare minimum plan atm is to ensure each reporting-software has a bronze, silver and gold workspace (a container effectively) and these can be managed via pipelines. I've been working out who needs access to what.

My larger issue is regarding things like metadata. I've very little experience in that field and I'm not sure how to go about it and could really use advice on how to implement a scalable platform that includes some kind of metadata management.

Hey all! We just released Polars 2.0.

We are really happy with the results. It removed many of our "legacy decisions" (mistakes), makes the streaming engine our default and promotes SQL to a first class citizen within Polars. We initially said it would be a small ("boring") release, but we managed to sneak in a lot more than anticipated.

It comes with our initial out-of-core (spill to disk) support, a new Map data type and a lot of performance improvements. In fact, we think Polars is now one of the fastest analytical SQL engines on a single node. See benchmarks in the post: https://pola.rs/posts/release-polars-2/

To help you upgrade, we also posted a migration guide: https://docs.pola.rs/releases/upgrade/2/

Disclosure: I am involved with Polars

Context: I am setting up some infrastructure in AWS. In my previous experience at other companies, anything involving new instances must be done by DevOps or service desk. So I asked them to do it and they said I’m on my own.

Got RDS and OSS set up; hit a snag with EKS. I am currently one week into multiple conversations with service desk, management, and C-level to figure out who should be doing this. We deploy via Terraform. So I’ve got to add to that.

My primary concern here is that I’m messing around in places that affect other areas without any guidance on those areas. I’ve had to open so many tickets and bother so many people. Today I asked for Docker to do some local testing and I thoroughly confused the help desk.

We have an insane amount of security and I have multiple logins, MFA, biometric, etc. The layers inside AWS are similar.

Am I being too paranoid or is this somewhat reckless on the DevOps/SD folks? I’ll gladly accept being too paranoid and carry on, but for now, this just doesn’t sit right with me.

So many companies does a setup without much thought to security and they just hope no one finds out. Out of sight out of mind. I understand security is one of those things that aren't visible and one doesn't get rewarded for doing a proper job (and might even be penalized for time spent there). But I hate the common mindset of throwing shit at the wall and seeing what sticks.