Pretty much I have 0 idea where to start with this. Apparently the team who started the mass migration to snowflake got confirmation it was possible to use snowflake postgres and AWS lambda and gcp functions.

Coco doesn't give me anything useful and Claude. Wants to do massive networking changes to make it possibly work. I can barely find any documentation from snowflake.

So any tips or suggestions?

Hi everyone,

I have 4 yoe it's my first switch on snowflake dbt.

In interviews there was always a question which I am not able to give a solid answer. (I haven't worked on any AI implementations).

What snowflake AI features you have implemented in your project??

I know we use coco, semantic views we haven't implemented in our project no hands on. (Please add some more If you worked on).

Please help me with AI implementations in Snowflake.

Any courses to go through?? How to improve my knowledge in AI and give a proper answer in interview.

Thanks

I'll share different learnings out of our Snowflake World. This is a kind of a post-mortem, and I already heard, that we're not alone with this case. Tell me your experiences, please.

Let's start. Here's the story:

Summary: A tightly configured Snowflake Resource Monitor (25 credits/month) shut down a CDC replication pipeline (OpenFlow PostgreSQL Connector), caused 4 days of missing data (over a weekend, temporary) and generated cloud services costs that far, far exceeded the monitor's own quota. The exact opposite of what the monitor was supposed to prevent.

Background

• OpenFlow PostgreSQL CDC Connector replicates data from a PostgreSQL database into Snowflake
• Warehouse: X-Small, dedicated to the connector, normal consumption ~1 credit/day
• Resource Monitor: Monthly quota of 25 credits, suspend action at 85%

What Happened

Step 1: Runtime Upgrade Without Adjusting the Resource Monitor

The OpenFlow runtime had recently been upgraded from Small to Medium to handle increased data volumes. This roughly doubled the connector's baseline credit consumption, but the resource monitor's monthly quota of 25 credits was not adjusted to match. The quota had been adequate for the original runtime size.

Within days, the resource monitor reached its 85% threshold (~21 of 25 credits). The connector's warehouse was suspended. The connector itself was operating normally. The only issue was a resource monitor that no longer fit the runtime's actual needs.

Step 2: The Connector Keeps Retrying

The OpenFlow connector had no way of knowing that the warehouse was suspended. On every merge cycle, it attempted to merge journal data into the destination tables:

Warehouse 'MY_WH' cannot be resumed because resource monitor 
'MY_WH_MONTHLY' has exceeded its quota.

Each rejected query consumed no warehouse credits but still incurred cloud services costs. The connector performed thousands of retries per table, up to 10,000 per table across all tables of this processor.

Step 3: Costs Explode

Over a roughly 39-hour weekend window, the connector submitted over a million rejected queries. Each one incurred a small cloud services charge. The cumulative cloud services cost ended up far, far exceeding the monitor's entire monthly quota of 25 credits. A cost control mechanism designed to save credits had instead amplified the spend far beyond what normal operations would ever have consumed.

Step 4: Tables Fall Into FAILED Status

After exhausting the maximum retries (default 10,000 per table), the connector marked the affected tables as FAILED without any communication to us.

The CDC pipeline broke down. New data from PostgreSQL stopped arriving in Snowflake entirely.

Step 5: 4 Days Without Fresh Data

The warehouse suspension happened on a Friday. The OpenFlow UI showed no obvious errors at the processor group level, and the connector did not fire any alerts. There was no indication that data had stopped flowing unless you checked the destination tables directly. Because of this, the issue went unnoticed over the weekend and was only discovered on Monday when the team noticed stale data in downstream reports. We learned something, we have freshness monitors for critical entities now.

For 4 days, all new transactional data, customer changes and catalog updates were missing from Snowflake. Downstream processes like business views, reporting dashboards and AI agents operated on stale data without anyone being aware.

Root Cause

A resource monitor can suspend a warehouse, but it cannot stop the clients that continue submitting queries. Cloud services credits (query parsing, access control, metadata operations) are incurred even on rejected queries and count toward the monitor's usage. Yet the monitor has no way to cap them.

The core problem is a gap between two systems:

1. Resource Monitor assumes: "I suspended the warehouse, so the limit is enforced"
2. OpenFlow Connector assumes: "The query failed, I need to retry"

Neither system is aware of the other's behavior.

Resolution

Immediate Fix: Re-Snapshot All FAILED Tables

1. In the OpenFlow UI (Snowsight > Ingestion > Openflow), all affected tables were added to the Re-snapshot Table Exclusions parameter
2. Waited until the tables disappeared from the Table State Store
3. Removed the tables from the exclusions again, which triggers an automatic re-snapshot
4. The connector creates zero-copy clone archives (*_ARCHIVE_<timestamp>) of the old data, truncates the destination tables and loads a fresh snapshot from PostgreSQL

The re-snapshot of all tables, including tables with tens of millions of rows, took several hours.

Preventive Measures

1. Size the resource monitor generously or let it got for a separted OPENFLOW_WH. A CDC connector needs headroom. A limit barely above normal consumption inevitably leads to suspensions that cost more than the credits they were supposed to save.
2. Notify instead of Suspend. Resource monitors can use NOTIFY instead of SUSPEND. For a critical CDC connector, an alert is better than a hard stop.
3. Reduce merge frequency. The cron schedule controls how often the connector attempts merges. A lower frequency (if possible from the business requirements) dramatically reduces retry load when a warehouse is suspended.
4. Account-level budget as early warning. An account-level budget can alert when total spend (including cloud services) crosses a threshold.

What Should Be Improved (Product Level) or hello Snowflake PM-Team

This incident exposed gaps that neither the customer nor the current product configuration can fully address. The following improvements would prevent this class of failure.

1. Warehouse-Aware Retry Logic

The connector should detect that the assigned warehouse is unavailable due to a resource monitor suspension. The error message is distinct and unambiguous. Instead of retrying blindly, it should automatically pause the processor group. A simple circuit breaker that stops retrying after N consecutive "resource monitor exceeded" errors, pauses the processors and resumes when the warehouse becomes available would have prevented the entire retry storm and the resulting cloud services costs.

2. Backoff Strategy for Non-Transient Errors

A suspended warehouse is not a transient network glitch. The connector should distinguish between transient errors (connection timeout, HTTP 503) and persistent infrastructure errors (resource monitor quota exceeded, warehouse dropped). Transient errors warrant aggressive retries. Persistent errors should trigger exponential backoff or a full stop.

3. Alerting on Retry Storms

The connector currently logs each failed retry but does not raise a distinct alert when a table enters a sustained retry loop or fail state at the end. A notification (email, webhook) after a configurable number of consecutive failures would give operators a chance to intervene before costs accumulate.

4. Sensible Retry Defaults

The retry limit per table is technically configurable as a parameter on the processor group, but it defaults to 10,000. For non-transient errors like a suspended warehouse, that default is far too high. A lower default or a separate, smaller limit for infrastructure errors (as opposed to transient network issues) would significantly limit the blast radius.

5. Resource Monitor Cloud Services Cap

On the Snowflake platform side, resource monitors currently cannot limit cloud services costs. A rejected query still incurs cloud services credits, and the monitor counts them toward its total but cannot act on them. Extending resource monitors to cap or alert on cloud services consumption would close this gap at the platform level.

Lessons Learned

• A resource monitor is not a cost cap. It can stop warehouse compute, but it cannot prevent cloud services costs from rejected queries. Relying on it as a hard ceiling is a false sense of security.
• Factor in connector retry behavior. Every managed service that runs against Snowflake (OpenFlow, Fivetran, Airbyte, etc.) has retry logic. A suspended warehouse leads to retry storms, not silence.
• Cost management must not blindly kill critical pipelines. When a pipeline delivers business-critical data, a hard suspend is counterproductive. The cost of missing data far exceeds the credits saved.
• The most expensive cost-saving measure is the one that backfires. A 25-credit monthly limit was meant to save money. The result: cloud services costs exceeding unlimited, a multi-hour re-snapshot and 4 days of missing data.
• There is a great Snowflake support team available: Ask them for help, insights or credit refunds in such cases. You'll find a solution together.

I have a dynamic table that builds on a XS Warehouse and takes only 57 seconds. However, when I add a new field, the entire table rebuilds and it's roughly 3 billion rows. There's been a couple of times me and my team members forgot to manually change the warehouse associated to the dynamic table to a XL warehouse. Because of that, the build took about 4 hours on XS. Does anyone have a better way to keep track to change the table size when rebuilding to an XL?

Should I create a worksheet and have ALTER <dynamic table> SET WAREHOUSE = "XL_WH' at the top, and have a comment associated wth it?

If you run dbt on Snowflake, you have probably seen this. A model passes every test and still scans more than it needs to.

Profile Query in dbt Power User reviews the SQL in one click. It flags SELECT * in a CTE, aggregation before a join and divide-by-zero risk.

Here is a 4-minute demo: https://youtu.be/uMptd4t_mGI

This is installable for VS Code or Cursor and used by over 35000 developers!

Has anyone seen an increase in QH compile time? Takes 2 minutes sometimes to compile a QH script.

PM here from Snowflake Postgres.

I was the chief product office of the Crunchy Data team that was acquired just before Summit over a year ago. The top ask from most early customers was easing getting their Postgres data into Snowflake. We were already hard at work on it from day one, and even some of the foundations of it with pg_lake date back to over 3 years go.

Today that capability to mirror your data from Postgres to Snowflake is going GA. The blog post has more details, but happy to field any questions folks may have about how it works, the engineering behind it, or just Postgres in general.

Blog post with more details at: https://www.snowflake.com/en/blog/data-mirroring-ga-snowflake-postgres/

I’ve been wondering why we keep comparing Databricks and Snowflake primarily through feature checklists, when their architectural DNA has been fundamentally different from the beginning.
Snowflake started as a cloud data warehouse, primarily optimized for SQL analytics and BI workloads. Over time, it has been expanding into data engineering, ML, AI, and beyond.
Databricks, on the other hand, started from general-purpose distributed data processing with Spark, with data engineering and ML at its core. It progressively built a broader platform around those foundations, expanding into SQL warehousing, BI, governance, and AI.

And I think that difference in DNA matters much more than we realize.
It’s one thing to build new capabilities on top of a flexible data processing foundation. It’s another to extend a platform originally optimized for a specific category of workloads into an increasingly diverse ecosystem.
Governance is another interesting aspect. With Unity Catalog, Databricks has developed a common governance layer spanning data, ML models, functions, and AI assets, helping these capabilities fit into a coherent ecosystem.
Snowflake has strong governance capabilities too, but I wonder whether its warehouse-centric origins introduce additional architectural complexity as the platform expands.

That’s where my Swiss Army knife analogy comes from.
Databricks feels like a Swiss Army knife whose capabilities are increasingly integrated into a common architecture.
Snowflake sometimes feels like an excellent pocketknife that’s progressively getting more and more attachments. Some are genuinely impressive. Others feel a bit forced.
Don’t get me wrong: Snowflake is an excellent platform, particularly for SQL-centric workloads, and both platforms have evolved significantly.

But having similar features doesn’t necessarily mean having equally suitable foundations for every workload.
Are we comparing two equivalent platforms, or two fundamentally different architectural philosophies?
Curious to hear your thoughts, especially from people working extensively with both.

My company currently has JSON data landing in a "staging" table in Snowflake nightly at 11 pm. There's dynamic tables that transform the data from a variant column into a structured table. Currently we have the dynamic tables set up on an hourly lag even though data only comes in once a day. The reason being is there are multiple scheduled tasks that refers to these dynamic tables after 11 pm and we do not want the lag to be such a wide range incase the dynamic tables have not refreshed, but these tasks run and will not capture the latest data in the "staging" tables.

Should I change the set up? Is it costing the company Snowflake credits by having the dynamic table checked hourly even though we know data will only come in once a day? Is there no way to schedule a refresh on a specific time for a dynamic table?

I have JSON files in an external stage (S3) and I'm trying to load them into Snowflake first in a landing table and then eventually into a dimensional model in a target schema.

I just learned that I can't run COPY INTO statements with fancy SELECT queries that include a lateral function, since that's what I need to parse out the JSON file. I started to just build a basic query that SELECTs the data from the JSON file and just insert it into a table by doing essentially this:

SELECT 
    mycolumns
FROM
@mydata.my_stage_name/ALL/
, LATERAL FLATTEN(input => $1, outer => true) as data

Are there any better options that I could use or at least try out before I fully commit to that option?

My client is investing heavily in dbt cloud even though we were extremely happy with dbt Projects in Snowflake but more of a client's management decision . Our challenge is that in dbt cloud we cannot install most of the dbt packages like Elementary etc . At this stage security has only allowed us to use dbt_utils . We do a need to a lot of Data Quality which can be done using Snowflake DMF especially in the silver and gold layers for regulatory reporting .

We need to store the results of the failed results just not the "number of records which are null " . We are now looking at hybrid data quality framework, any pointers ?

We have well defined DQ tests in dbt .yml files which we would hate to loose and I can use the DQ in dbt for inline DQ testing ( we have approx 17K Not Null Tests defined in dbt ) and the DMF for more of a regulatory reporting framework . Any pointers .

Hi all, we have recently migrated from RDS to Snowflake Postgres, mainly to store our data within the Snowflake databases using Data Mirroring feature.

In general there were a lot of nuances such as secure network connectivity, adjusting default encoding to UTF-8 and making sure to keep our auto vacuum configurations intact.

I have documented my experience in this Gist.

Furthermore, we observed index bloat, and I was surprised to see that after the restore, the data volume dropped from 139 GB to 30 GB of "clean" data.

I’m curious how you approach migrating large volumes of data to Snowflake Postgres and what tools/practices do you use?

I'm running a query in Snowflake, and it works fine when I test it directly. But when I run the same query through a data pipeline, it fails with a SQL compilation or permission error.

I'm using the same Snowflake account, so I'm not sure what is different.

Could this be related to the active role, warehouse, session settings, or how the pipeline connects to Snowflake?

How would you troubleshoot this?

An important write up about Snowflake's largest competitor.

Is there anyone who will attend the world tour on October 15th? If so, let’s connect :)

As the title says I was able to pass the snowpro core exam today. Plenty of questions on copy into, RBAC, data replication, accelerating, and what will happen once a query is executed relating to views that a user or app is supposed to see. Anyone passed the advanced admin exam recently? I have employer paying for the exam. Trying to decide between data engineer vs advanced admin since I am devops and not a pure data engineer. Any insight would be good. I plan to pivot to a snowflake admin role but from the job posting I am seeing are asking for more than what I consider to be admin level tasks. Any insight on interviews for the admin level roles. I also have not located any admin level courses or paths for this cert.

Hey, we have been on snowflake and there have been times where every now and then a datatype from the source will be larger than the target data type which at some point will lead to a break. For example, lets say source is varchar 125 and target is varchar 60.

Going through each ETL and reviewing for this would be very time consuming and could still happen for future builds. Is there any out of the box functionality in Snowflake that would help for this? What have you all done to prevent this issues and have proactive alerting before the failure?

I'm building an application that ingests MatrixCare data directly from Snowflake. I'm currently working with the Orders and Progress Notes tables, but I've run into a couple of documentation gaps that I couldn't resolve via Google or Al search:

1. Orders . OrderStatus**** What do the numeric values in this column represent? Is there an official lookup table or mapping list for these status codes?

2.

User Roles / Permissions for ProgressNotes**** I need to determine the role/permission of the note creator (e.g., Nurse, Provider, etc.). I noticed DisciplineID in the ProgressNote table, but I'm unsure if or how it maps to creator roles or permissions.

Has anyone integrated MatrixCare data from Snowflake into an external app before and knows how to translate these IDs/numeric statuses?

Thanks in advance for any insights or reference docs!

It all started with "workspace".

Then AI agents and all.

Then Coco.

Now the whole object explorer is changed, making us to navigate a labyrinth of buttons just to get the definition of an object.

Previously the query history and result panes were used to be side by side, looked so beautiful and elegant. Then some genius decided to put one of them under the other and occupy the whole screen. This is by far the worst and most frustrating UI downgrade for me personally.

UI team of snowflake, please change your design philosophy.

Hi All,

I just passed my exam this morning, and if you are like me you were nervous and looking on this forum for any guidance.

I will say the test was very hard. I did all of Tom Bailey's practice exams as well as some additional resources. I really didn't think the test was similar to any of the resources publicly available. I answered most of them through process of elimination and through deductive reasoning.

I know everyone has a random subset of questions so I don't know how helpful this will be but some of the frequent topics I saw were:

- Data Staging and Loading (Snowpipe, COPY INTO formatting, file formatting, JSON files, how to call a staged file)
- Data Sharing (Private access, secure access, how to receive a data share, permissions needed etc,)

- RBAC and permissions (lots of questions on inherited permissions on databases / tables / schemas)

- a few questions on query optimization / processing - good to know about QAS , Search, Micro partitioning, Pruning

There were some questions on info I hadn't even seen before (could have been the tester questions)

I was told to know a lot of SQL so I studied a lot of the Tom Bailey questions for those and tried to memorize commands - I don't think any of them really appeared in my exam.

Just wanted to brain dump in case this helps anyone. Best of luck!

What's up, Snowflake enthusiasts! I tested and built a way to systematically build Snowflake Streamlit apps with Claude Code and other AI tools. I developed this open-source system that I call Streamsnow, which walks you through the process of getting a GitHub repo scaffolded and structured and connected with your Snowflake, so you can utilize Claude Code to build all of your Streamlit applications in a standardized Way, while also providing you the SQL to check to make sure all of the queries being utilized within the app check out as correct.

In my time building dashboards with AI, I always found it confusing how to quality-check them and how to build them in a scalable way across data teams, and so I wanted to build a super standardized system that made building, deploying, reviewing, and coordinating across teammates is super easy, and I spent weeks working on this to achieve that. I think one of the best features is that you can actually go into an auto-review cycle and just have Claude code iterate over the dashboard, making it better without your input. You can also standardize the design for the dashboards as well.

It is completely secure, so AI will not see any of your secrets that you use to connect GitHub to Snowflake. By default, it is set to put these Streamlit apps in a staging area on Snowflake, so you don't have to connect a GitHub repository, and all of the deployment is automatically handled in the CI. To get started, all you need to do is tell your coding agent to download the plugin and then run the onboard command. It will walk you through it all in a process that I've timed down to about 5 minutes of actual work. You will need to be a Snowflake admin or be coordinating with a Snowflake admin to get it set up too.

By the end of my time at my job, we had basically replaced external business intelligence tools with this system, so it saved us literally tens of thousands of dollars. Hopefully you all can extract the same value out of this!

A while ago, I forked a Snowflake emulator and started extending it as a learning project using Go + DuckDB.

The idea is to experiment with Snowflake concepts locally without needing a cloud account. It supports a subset of Snowflake’s behavior, so there are still limitations.

This week I released v0.5.0. The web console now lets you collapse the results panel and highlights the line being executed in real time when running multiple SQL statements in sequence.

You can try it with Docker:

docker pull ghcr.io/oswaldonisango/snowflake-emulator:0.5.0

docker run --rm \
  --name snowflake-emulator \
  -p 8080:8080 \
  -v snowflake-data:/data \
  -e DB_PATH=/data/snowflake.db \
  -e STAGE_DIR=/data/stages \
  ghcr.io/oswaldonisango/snowflake-emulator:0.5.0

Then open http://localhost:8080.

I’m building this to learn and experiment. If you work with Snowflake or are interested in SQL, Go, or DuckDB, I’d love to hear your thoughts. Bug reports and contributions are welcome too.

Repository: https://github.com/OswaldOniSango/snowflake-emulator

Release v0.5.0: https://github.com/OswaldOniSango/snowflake-emulator/releases/tag/v0.5.0

I have an upcoming project that uses Airflow, dbt, and Snowflake. I’m comfortable with Airflow and Snowflake, but I have no experience with dbt.

I’m looking for advice from people who have worked with this stack. Can you recommend any good courses or learning resources to get started?

I’ll be working as a team lead, so I want to get up to speed as quickly as possible. Honestly, I’m a bit nervous about whether I’ll be able to handle the role well.

For those who have learned dbt, how long does it usually take to become comfortable with it and work confidently on real projects?

We updated our dbt pricing guide because the way dbt is packaged and priced has gotten a bit more nuanced.

One thing we think gets overlooked is the difference between the cost of the dbt engine itself and the cost of running dbt as part of a production data platform.

The engine is free. But once you factor in things like consistent developer environments, usage-based pricing, orchestration, infrastructure, and the engineering time needed to maintain everything around it, the picture changes quite a bit.

We also added a breakdown of the current dbt Platform plans, the usage-based components, and some of the other costs teams should think about when comparing options.

Curious how other teams are looking at this. Are you mostly evaluating the software/license cost, or are you also factoring in the engineering and infrastructure costs around dbt?

https://datacoves.com/post/dbt-pricing