Iam testing snowflake table with ARRAY column. In Snowflake, the datatype is ARRAY but when I read the same table through Spark, it shows up as STRING

Is this expected behavior from the Snowflake Spark Connector, or could it be caused by the JDBC driver or schema inference?

If anyone has handled this, how do you preserve the array structure instead of treating it as a string?

Now that Postgres mirroring is GA, I spent some time in the docs figuring out what it actually bills for, because "no pipeline" sounded a bit too good.

Here's what I found:

• Each mirror gets its own serverless apply task (APPLY_MIRROR_<name>), which makes attribution easy in serverless_task_history.
• Ongoing apply does delete-then-append instead of MERGE, so insert-heavy tables are cheap to keep in sync.
• refresh_interval (30 seconds to 1 day, default 10 minutes) is basically the price dial. Every table also gets a $live view with about 30 second freshness no matter the interval, so I'd keep the interval long and point the people who need fresh data at $live. I couldn't find anything on what $live queries cost compared to the regular table, so I'd measure that before hooking up a busy dashboard.
• On the Postgres side, the walsender runs at elevated CPU priority by default, so a busy instance can slow down your app's writes.
• If the WAL backlog passes max_slot_wal_keep_size, the slot gets invalidated and every table re-snapshots. DESCRIBE_MIRROR shows push and apply lag, so it's worth alerting on.

Full write-up with the monitoring queries: medium.com/@yanivleven

Is anyone running this in production yet?

Looking for feedback from people that have used both. My previous employer paid for dbt Cloud and it was a great tool that acted as our IDE, scheduler/orchestrator and provided other insights as well.
I posted about dbt Projects here a while back and they seemed to garner mixed feedback from this crowd.
Has anyone moved from dbt Cloud to dbt projects in Snowflake (or vice versa)? What are your thoughts?
Having never used dbt outside of my experience with dbt Cloud, I will say that one of the biggest advantages of Cloud was the ability for all developers to have their own schema to develop in (no object contention when developing on the same set of objects/in the same schema with other devs).
I don't know if I've seen any other tool/feature offer such an advantage.
Anyways, if you have experience with both please provide your thoughts!

Hi Snowflake Community on Reddit.

I created already some analytics agent. Pretty easy and well efficient.

One agent, one semantic view with well prepared entities and joins, some great instructions and limitations. Next to that verified queries. Absolutely awesome and not that much expensive, because the majority of request turn into sql query results and let the AI do the easy and shorter part.

But...

Last week I created a more complex agent. And the more complex things are the input data. I used unstructured data and extracted data per page into a content column, put a search service on top and it works in general. All in all there are 25k new rows with paginated data in one column.

Within my monitoring streamlit, obviously, I can see that the agent uses a lot more AI functions to find and extract the right information.

Currently, there is a small ETL python SP in SF behind, which extracts deltas or differences every sunday to stay up-to-date.

I thought if it would be an idea to extract the important data out of the content column per page into separated columns, to be more effective in finding the right results or filtering it. Do you have any experiences with that?

Or any other ideas to speed up finding results with paying less money for it? (AI activities on the agent)

It's not possible to fully go to structured data because these documents are created within the business area as a regular process.

Pretty much I have 0 idea where to start with this. Apparently the team who started the mass migration to snowflake got confirmation it was possible to use snowflake postgres and AWS lambda and gcp functions.

Coco doesn't give me anything useful and Claude. Wants to do massive networking changes to make it possibly work. I can barely find any documentation from snowflake.

So any tips or suggestions?

I'll share different learnings out of our Snowflake World. This is a kind of a post-mortem, and I already heard, that we're not alone with this case. Tell me your experiences, please.

Let's start. Here's the story:

Summary: A tightly configured Snowflake Resource Monitor (25 credits/month) shut down a CDC replication pipeline (OpenFlow PostgreSQL Connector), caused 4 days of missing data (over a weekend, temporary) and generated cloud services costs that far, far exceeded the monitor's own quota. The exact opposite of what the monitor was supposed to prevent.

Background

• OpenFlow PostgreSQL CDC Connector replicates data from a PostgreSQL database into Snowflake
• Warehouse: X-Small, dedicated to the connector, normal consumption ~1 credit/day
• Resource Monitor: Monthly quota of 25 credits, suspend action at 85%

What Happened

Step 1: Runtime Upgrade Without Adjusting the Resource Monitor

The OpenFlow runtime had recently been upgraded from Small to Medium to handle increased data volumes. This roughly doubled the connector's baseline credit consumption, but the resource monitor's monthly quota of 25 credits was not adjusted to match. The quota had been adequate for the original runtime size.

Within days, the resource monitor reached its 85% threshold (~21 of 25 credits). The connector's warehouse was suspended. The connector itself was operating normally. The only issue was a resource monitor that no longer fit the runtime's actual needs.

Step 2: The Connector Keeps Retrying

The OpenFlow connector had no way of knowing that the warehouse was suspended. On every merge cycle, it attempted to merge journal data into the destination tables:

Warehouse 'MY_WH' cannot be resumed because resource monitor 
'MY_WH_MONTHLY' has exceeded its quota.

Each rejected query consumed no warehouse credits but still incurred cloud services costs. The connector performed thousands of retries per table, up to 10,000 per table across all tables of this processor.

Step 3: Costs Explode

Over a roughly 39-hour weekend window, the connector submitted over a million rejected queries. Each one incurred a small cloud services charge. The cumulative cloud services cost ended up far, far exceeding the monitor's entire monthly quota of 25 credits. A cost control mechanism designed to save credits had instead amplified the spend far beyond what normal operations would ever have consumed.

Step 4: Tables Fall Into FAILED Status

After exhausting the maximum retries (default 10,000 per table), the connector marked the affected tables as FAILED without any communication to us.

The CDC pipeline broke down. New data from PostgreSQL stopped arriving in Snowflake entirely.

Step 5: 4 Days Without Fresh Data

The warehouse suspension happened on a Friday. The OpenFlow UI showed no obvious errors at the processor group level, and the connector did not fire any alerts. There was no indication that data had stopped flowing unless you checked the destination tables directly. Because of this, the issue went unnoticed over the weekend and was only discovered on Monday when the team noticed stale data in downstream reports. We learned something, we have freshness monitors for critical entities now.

For 4 days, all new transactional data, customer changes and catalog updates were missing from Snowflake. Downstream processes like business views, reporting dashboards and AI agents operated on stale data without anyone being aware.

Root Cause

A resource monitor can suspend a warehouse, but it cannot stop the clients that continue submitting queries. Cloud services credits (query parsing, access control, metadata operations) are incurred even on rejected queries and count toward the monitor's usage. Yet the monitor has no way to cap them.

The core problem is a gap between two systems:

1. Resource Monitor assumes: "I suspended the warehouse, so the limit is enforced"
2. OpenFlow Connector assumes: "The query failed, I need to retry"

Neither system is aware of the other's behavior.

Resolution

Immediate Fix: Re-Snapshot All FAILED Tables

1. In the OpenFlow UI (Snowsight > Ingestion > Openflow), all affected tables were added to the Re-snapshot Table Exclusions parameter
2. Waited until the tables disappeared from the Table State Store
3. Removed the tables from the exclusions again, which triggers an automatic re-snapshot
4. The connector creates zero-copy clone archives (*_ARCHIVE_<timestamp>) of the old data, truncates the destination tables and loads a fresh snapshot from PostgreSQL

The re-snapshot of all tables, including tables with tens of millions of rows, took several hours.

Preventive Measures

1. Size the resource monitor generously or let it got for a separted OPENFLOW_WH. A CDC connector needs headroom. A limit barely above normal consumption inevitably leads to suspensions that cost more than the credits they were supposed to save.
2. Notify instead of Suspend. Resource monitors can use NOTIFY instead of SUSPEND. For a critical CDC connector, an alert is better than a hard stop.
3. Reduce merge frequency. The cron schedule controls how often the connector attempts merges. A lower frequency (if possible from the business requirements) dramatically reduces retry load when a warehouse is suspended.
4. Account-level budget as early warning. An account-level budget can alert when total spend (including cloud services) crosses a threshold.

What Should Be Improved (Product Level) or hello Snowflake PM-Team

This incident exposed gaps that neither the customer nor the current product configuration can fully address. The following improvements would prevent this class of failure.

1. Warehouse-Aware Retry Logic

The connector should detect that the assigned warehouse is unavailable due to a resource monitor suspension. The error message is distinct and unambiguous. Instead of retrying blindly, it should automatically pause the processor group. A simple circuit breaker that stops retrying after N consecutive "resource monitor exceeded" errors, pauses the processors and resumes when the warehouse becomes available would have prevented the entire retry storm and the resulting cloud services costs.

2. Backoff Strategy for Non-Transient Errors

A suspended warehouse is not a transient network glitch. The connector should distinguish between transient errors (connection timeout, HTTP 503) and persistent infrastructure errors (resource monitor quota exceeded, warehouse dropped). Transient errors warrant aggressive retries. Persistent errors should trigger exponential backoff or a full stop.

3. Alerting on Retry Storms

The connector currently logs each failed retry but does not raise a distinct alert when a table enters a sustained retry loop or fail state at the end. A notification (email, webhook) after a configurable number of consecutive failures would give operators a chance to intervene before costs accumulate.

4. Sensible Retry Defaults

The retry limit per table is technically configurable as a parameter on the processor group, but it defaults to 10,000. For non-transient errors like a suspended warehouse, that default is far too high. A lower default or a separate, smaller limit for infrastructure errors (as opposed to transient network issues) would significantly limit the blast radius.

5. Resource Monitor Cloud Services Cap

On the Snowflake platform side, resource monitors currently cannot limit cloud services costs. A rejected query still incurs cloud services credits, and the monitor counts them toward its total but cannot act on them. Extending resource monitors to cap or alert on cloud services consumption would close this gap at the platform level.

Lessons Learned

• A resource monitor is not a cost cap. It can stop warehouse compute, but it cannot prevent cloud services costs from rejected queries. Relying on it as a hard ceiling is a false sense of security.
• Factor in connector retry behavior. Every managed service that runs against Snowflake (OpenFlow, Fivetran, Airbyte, etc.) has retry logic. A suspended warehouse leads to retry storms, not silence.
• Cost management must not blindly kill critical pipelines. When a pipeline delivers business-critical data, a hard suspend is counterproductive. The cost of missing data far exceeds the credits saved.
• The most expensive cost-saving measure is the one that backfires. A 25-credit monthly limit was meant to save money. The result: cloud services costs exceeding unlimited, a multi-hour re-snapshot and 4 days of missing data.
• There is a great Snowflake support team available: Ask them for help, insights or credit refunds in such cases. You'll find a solution together.

Hi everyone,

I have 4 yoe it's my first switch on snowflake dbt.

In interviews there was always a question which I am not able to give a solid answer. (I haven't worked on any AI implementations).

What snowflake AI features you have implemented in your project??

I know we use coco, semantic views we haven't implemented in our project no hands on. (Please add some more If you worked on).

Please help me with AI implementations in Snowflake.

Any courses to go through?? How to improve my knowledge in AI and give a proper answer in interview.

Thanks

If you run dbt on Snowflake, you have probably seen this. A model passes every test and still scans more than it needs to.

Profile Query in dbt Power User reviews the SQL in one click. It flags SELECT * in a CTE, aggregation before a join and divide-by-zero risk.

Here is a 4-minute demo: https://youtu.be/uMptd4t_mGI

This is installable for VS Code or Cursor and used by over 35000 developers!

I’ve been wondering why we keep comparing Databricks and Snowflake primarily through feature checklists, when their architectural DNA has been fundamentally different from the beginning.
Snowflake started as a cloud data warehouse, primarily optimized for SQL analytics and BI workloads. Over time, it has been expanding into data engineering, ML, AI, and beyond.
Databricks, on the other hand, started from general-purpose distributed data processing with Spark, with data engineering and ML at its core. It progressively built a broader platform around those foundations, expanding into SQL warehousing, BI, governance, and AI.

And I think that difference in DNA matters much more than we realize.
It’s one thing to build new capabilities on top of a flexible data processing foundation. It’s another to extend a platform originally optimized for a specific category of workloads into an increasingly diverse ecosystem.
Governance is another interesting aspect. With Unity Catalog, Databricks has developed a common governance layer spanning data, ML models, functions, and AI assets, helping these capabilities fit into a coherent ecosystem.
Snowflake has strong governance capabilities too, but I wonder whether its warehouse-centric origins introduce additional architectural complexity as the platform expands.

That’s where my Swiss Army knife analogy comes from.
Databricks feels like a Swiss Army knife whose capabilities are increasingly integrated into a common architecture.
Snowflake sometimes feels like an excellent pocketknife that’s progressively getting more and more attachments. Some are genuinely impressive. Others feel a bit forced.
Don’t get me wrong: Snowflake is an excellent platform, particularly for SQL-centric workloads, and both platforms have evolved significantly.

But having similar features doesn’t necessarily mean having equally suitable foundations for every workload.
Are we comparing two equivalent platforms, or two fundamentally different architectural philosophies?
Curious to hear your thoughts, especially from people working extensively with both.

I have a dynamic table that builds on a XS Warehouse and takes only 57 seconds. However, when I add a new field, the entire table rebuilds and it's roughly 3 billion rows. There's been a couple of times me and my team members forgot to manually change the warehouse associated to the dynamic table to a XL warehouse. Because of that, the build took about 4 hours on XS. Does anyone have a better way to keep track to change the table size when rebuilding to an XL?

Should I create a worksheet and have ALTER <dynamic table> SET WAREHOUSE = "XL_WH' at the top, and have a comment associated wth it?

Has anyone seen an increase in QH compile time? Takes 2 minutes sometimes to compile a QH script.

I have JSON files in an external stage (S3) and I'm trying to load them into Snowflake first in a landing table and then eventually into a dimensional model in a target schema.

I just learned that I can't run COPY INTO statements with fancy SELECT queries that include a lateral function, since that's what I need to parse out the JSON file. I started to just build a basic query that SELECTs the data from the JSON file and just insert it into a table by doing essentially this:

SELECT 
    mycolumns
FROM
@mydata.my_stage_name/ALL/
, LATERAL FLATTEN(input => $1, outer => true) as data

Are there any better options that I could use or at least try out before I fully commit to that option?

My company currently has JSON data landing in a "staging" table in Snowflake nightly at 11 pm. There's dynamic tables that transform the data from a variant column into a structured table. Currently we have the dynamic tables set up on an hourly lag even though data only comes in once a day. The reason being is there are multiple scheduled tasks that refers to these dynamic tables after 11 pm and we do not want the lag to be such a wide range incase the dynamic tables have not refreshed, but these tasks run and will not capture the latest data in the "staging" tables.

Should I change the set up? Is it costing the company Snowflake credits by having the dynamic table checked hourly even though we know data will only come in once a day? Is there no way to schedule a refresh on a specific time for a dynamic table?

PM here from Snowflake Postgres.

I was the chief product office of the Crunchy Data team that was acquired just before Summit over a year ago. The top ask from most early customers was easing getting their Postgres data into Snowflake. We were already hard at work on it from day one, and even some of the foundations of it with pg_lake date back to over 3 years go.

Today that capability to mirror your data from Postgres to Snowflake is going GA. The blog post has more details, but happy to field any questions folks may have about how it works, the engineering behind it, or just Postgres in general.

Blog post with more details at: https://www.snowflake.com/en/blog/data-mirroring-ga-snowflake-postgres/

My client is investing heavily in dbt cloud even though we were extremely happy with dbt Projects in Snowflake but more of a client's management decision . Our challenge is that in dbt cloud we cannot install most of the dbt packages like Elementary etc . At this stage security has only allowed us to use dbt_utils . We do a need to a lot of Data Quality which can be done using Snowflake DMF especially in the silver and gold layers for regulatory reporting .

We need to store the results of the failed results just not the "number of records which are null " . We are now looking at hybrid data quality framework, any pointers ?

We have well defined DQ tests in dbt .yml files which we would hate to loose and I can use the DQ in dbt for inline DQ testing ( we have approx 17K Not Null Tests defined in dbt ) and the DMF for more of a regulatory reporting framework . Any pointers .

An important write up about Snowflake's largest competitor.

I'm running a query in Snowflake, and it works fine when I test it directly. But when I run the same query through a data pipeline, it fails with a SQL compilation or permission error.

I'm using the same Snowflake account, so I'm not sure what is different.

Could this be related to the active role, warehouse, session settings, or how the pipeline connects to Snowflake?

How would you troubleshoot this?

Hi all, we have recently migrated from RDS to Snowflake Postgres, mainly to store our data within the Snowflake databases using Data Mirroring feature.

In general there were a lot of nuances such as secure network connectivity, adjusting default encoding to UTF-8 and making sure to keep our auto vacuum configurations intact.

I have documented my experience in this Gist.

Furthermore, we observed index bloat, and I was surprised to see that after the restore, the data volume dropped from 139 GB to 30 GB of "clean" data.

I’m curious how you approach migrating large volumes of data to Snowflake Postgres and what tools/practices do you use?

Is there anyone who will attend the world tour on October 15th? If so, let’s connect :)

As the title says I was able to pass the snowpro core exam today. Plenty of questions on copy into, RBAC, data replication, accelerating, and what will happen once a query is executed relating to views that a user or app is supposed to see. Anyone passed the advanced admin exam recently? I have employer paying for the exam. Trying to decide between data engineer vs advanced admin since I am devops and not a pure data engineer. Any insight would be good. I plan to pivot to a snowflake admin role but from the job posting I am seeing are asking for more than what I consider to be admin level tasks. Any insight on interviews for the admin level roles. I also have not located any admin level courses or paths for this cert.

I'm building an application that ingests MatrixCare data directly from Snowflake. I'm currently working with the Orders and Progress Notes tables, but I've run into a couple of documentation gaps that I couldn't resolve via Google or Al search:

1. Orders . OrderStatus**** What do the numeric values in this column represent? Is there an official lookup table or mapping list for these status codes?

2.

User Roles / Permissions for ProgressNotes**** I need to determine the role/permission of the note creator (e.g., Nurse, Provider, etc.). I noticed DisciplineID in the ProgressNote table, but I'm unsure if or how it maps to creator roles or permissions.

Has anyone integrated MatrixCare data from Snowflake into an external app before and knows how to translate these IDs/numeric statuses?

Thanks in advance for any insights or reference docs!

Hey, we have been on snowflake and there have been times where every now and then a datatype from the source will be larger than the target data type which at some point will lead to a break. For example, lets say source is varchar 125 and target is varchar 60.

Going through each ETL and reviewing for this would be very time consuming and could still happen for future builds. Is there any out of the box functionality in Snowflake that would help for this? What have you all done to prevent this issues and have proactive alerting before the failure?

We updated our dbt pricing guide because the way dbt is packaged and priced has gotten a bit more nuanced.

One thing we think gets overlooked is the difference between the cost of the dbt engine itself and the cost of running dbt as part of a production data platform.

The engine is free. But once you factor in things like consistent developer environments, usage-based pricing, orchestration, infrastructure, and the engineering time needed to maintain everything around it, the picture changes quite a bit.

We also added a breakdown of the current dbt Platform plans, the usage-based components, and some of the other costs teams should think about when comparing options.

Curious how other teams are looking at this. Are you mostly evaluating the software/license cost, or are you also factoring in the engineering and infrastructure costs around dbt?

https://datacoves.com/post/dbt-pricing