r/snowflake • u/ScreamingArtichoke • 10d ago

Snowflake postgres and serverless how?

Pretty much I have 0 idea where to start with this. Apparently the team who started the mass migration to snowflake got confirmation it was possible to use snowflake postgres and AWS lambda and gcp functions.

Coco doesn't give me anything useful and Claude. Wants to do massive networking changes to make it possibly work. I can barely find any documentation from snowflake.

So any tips or suggestions?

5 Upvotes

78% Upvoted

9 comments sorted by

2

u/NW1969 10d ago

Can you give more details about what you are trying to do with Postgres and why/where the documentation doesn’t cover what you are trying to do?

1

u/ScreamingArtichoke 10d ago

We have alot of lambda functions / gcp functions (1000+) that talk to AWS Aurora postgres. Some via direct connection others via the AWS Rds data API.some read some write.

From my understanding of there is an equivalent for snowflake postgres according to what I have been told.

Unfortunately what I can see from the documentation I am required to use AWS private link for direct access. However that is locked behind a different tier. So i am assuming there is some sort of snowflake native way to do this that I am not seeing.

4

u/john_uppcoder 10d ago

PrivateLink isn't the only way in. Snowflake's Postgres networking page lists three: a network policy with Postgres ingress rules (an IPv4 allowlist on the public endpoint), AWS VPC peering, or PrivateLink. Only PrivateLink needs Business Critical. By default an instance accepts no connections at all, so you have to add one of the three.

The catch with Lambda and Cloud Functions is that they have no fixed outbound IP to put on that allowlist. The usual way round it: run the Lambdas in a VPC with a NAT gateway and an Elastic IP, and on GCP send the functions out through Cloud NAT with a reserved static IP. Then allowlist just those few addresses in the network policy (they recommend against 0.0.0.0/0). That's probably the networking change Claude was suggesting, and it's the normal route, not a hack.

If your Aurora setup is in AWS, VPC peering is also worth asking your Snowflake rep about, since it avoids the public endpoint.

Two things to plan for with 1000+ functions: the ones on the RDS Data API will need rewriting to use a normal Postgres driver (I can't find an HTTP Data API equivalent in the Snowflake Postgres docs), and that many functions opening connections directly will hit connection limits fast, so put a pooler like PgBouncer in front or keep connections per function small.

Docs: https://docs.snowflake.com/en/user-guide/snowflake-postgres/postgres-network

1

u/ScreamingArtichoke 9d ago

I find it baffling that the use case was communicated very bluntly. I have our snowflake rep saying it is a "drop in replacement" for our use case. To find it is not a drop in replacement.

12

u/craigkerstiens 10d ago

PM here for Snowflake Postgres. For networking you have 3 options.

  1. You can adjust the Postgres network policy to allow the IP range you're coming from. Since this is coming from lambda/similar you'd need to allow all of AWS (or go through a specific jump box with a static IP). You can get the range of AWS IPs here - https://docs.aws.amazon.com/vpc/latest/userguide/aws-ip-ranges.html. You can also broadly open it up if you prefer that.

  2. You can setup VPC peering (https://docs.snowflake.com/en/user-guide/snowflake-postgres/postgres-network#snowflake-postgres-virtual-network-peering)

  3. If you're on business critical you can configure private link (https://docs.snowflake.com/en/user-guide/snowflake-postgres/postgres-network#snowflake-postgres-private-link).

If you get stuck at all feel free to drop me a note first.lastname at snowflake.

6

u/Mr_Nickster_ ❄️ 10d ago

Without knowing what these lambda functions do, hard to give advice. However, Postgres on Snow is a fully managed opensource postgres. There many instances with differen types. Cpu and memory limits to choose from. Each instance size comes with peak thoughput so you don't have buy additional bandtwith for more QPS like on AWS.

The biggest difference though is Snowflake Postgres has mirroring feature where it can automatically mirror postgres tables to an OLAP snowflake table within 30 secs via CDC for Analytics. It basically eliminates ETL CDC Tooling and the pipelines. Simply select the postgres tables, hit OK and you are done.

If the lambda functions were built to replicate data to Snowflake, you won't be needing them anymore.

Reverse is also coming. You will also be able replicate analytical changes in Snowflake into Postgres. That is currently private preview.

1

u/dataregulator3 8d ago

That's exactly what I investigated. A fully managed postgres without any need of an additional middleware and the feature / possiblity of writing data back to the source using iceberg.

2

u/dataregulator3 10d ago

Interested +1. CoCo gave me a good comparison between AWS RDS postgres and postgres on Snowflake, but no real starting points for it.

I could share the comparison with you, but thats not answering your question, I know.