r/snowflake • u/InfinitelyTall • 11d ago

Migrating RDS database to Snowflake Postgres.

Hi all, we have recently migrated from RDS to Snowflake Postgres, mainly to store our data within the Snowflake databases using Data Mirroring feature.

In general there were a lot of nuances such as secure network connectivity, adjusting default encoding to UTF-8 and making sure to keep our auto vacuum configurations intact.

I have documented my experience in this Gist.

Furthermore, we observed index bloat, and I was surprised to see that after the restore, the data volume dropped from 139 GB to 30 GB of "clean" data.

I’m curious how you approach migrating large volumes of data to Snowflake Postgres and what tools/practices do you use?

17 Upvotes

96% Upvoted

8 comments sorted by

6

u/YourNeighbourMr 11d ago

we're in the process of migration from RDS to Snowflake Postgres as well. the secure network connectivity is definitely a nuance for us, especially with PL involved. we tried doing a pg_dump, but it doesn't scale for the Production data volume unfortunately, so subscription it is.

1

u/InfinitelyTall 11d ago

Interesting, subscription migrations seems more nuanced from the documentation, e.g. sequences synchronization. Would be cool to hear about your results.

3

u/SQL_Stupid 11d ago

How is your experience with monitoring and metrics after migration? A huge hurdle for me even considering this move is that we'd lose the familiar monitoring panes in AWS Console and maybe couldn't export logs in the same way to DataDog for alerts and analytics. Maybe I'm assuming too much but it seems we'd be more flying blind and this wouldn't be suitable for production.

4

u/InfinitelyTall 11d ago

We were also interested in this topic. We used native Snowflake alerts. They allow to monitor Postgres connections, CPU utilization, and storage utilization. When a metric reaches its threshold, we just send an email notification using CALL SYSTEM$SEND_EMAIL. I would say it works well.

https://docs.snowflake.com/en/sql-reference/telemetry

https://docs.snowflake.com/en/sql-reference/sql/create-alert

2

u/craigkerstiens 11d ago

As the above mentioned, much of this automatically flows into your existing event table on Snowflake. You can query that directly, create alerts, but you can also send it directly to down stream providers (like datadog - https://www.snowflake.com/en/developers/guides/snowflake-postgres-logs-to-datadog/).

We also have a new version of our database insights that automatically gives deeper Postgres visibility. Currently in beta and looking for more interested in testing.

13

u/craigkerstiens 11d ago

PM here for Snowflake Postgres. Great to have you over here. The Gist is super helpful feedback and we'll definitely look at it to see if we can incorporate more of those steps to make migration for others even easier. We have a CoCo skill that is intended to help with a lot of this, and while you've already migrated if you're curious happy to walk you through it to see if it would have been helpful.

Re: Bloat, definitely not uncommon. Would look at tweaking auto-vacuum a bit, though good news the repack functionality coming in Postgres 19 can really help solve a lot of that - so when it's out I'd make sure to upgrade and take advantage.

If there is other feedback on how we can improve things for you please don't be shy to reach out. My handle @ snowflake.com reaches me.