Ever wondered what powers Snowflake’s seamless cloud data operations?
The answer lies in its Cloud Services Layer – the mastermind controlling access, authentication, query compilation, and load balancing.
It’s like having a backstage pass to an exclusive data concert.
But this VIP treatment comes at a cost, calculated daily based on usage.
The Cloud Services Layer ensures you only pay when exceeding certain thresholds.
And we’ll explore tricks to optimize costs, making your Snowflake experience both high-performing and budget-friendly.
Keen to unlock the magic of the Cloud Services Layer and maximize your data investment? Let’s dive in!
The Inside Scoop on Snowflake’s Cloud Services Layer
The Cloud Services Layer is a key part of Snowflake’s architecture, acting as the “coordinator” or “control plane”.
This layer handles the following responsibilities:
- Access Control (managing access)
- Authentication (verifying users)
- Query Compilation (preparing queries for execution)
- Caching (storing data for faster retrieval)
- Query Placement and Load Balancing (distributing queries across availability zones)
- Replication (ensuring data redundancy)
Want to learn more about Snowflake Architecture? Check out this one
Cloud Services Costs – Formula
The usage of Cloud Services is charged only when your daily consumption exceeds 10% of the daily usage of your Virtual Warehouse.
The charge is calculated Daily.
Two key takeaways from this:
- Because Cloud Services are calculated daily, the monthly sum of Cloud Services credits charged could be less than 10% of the entire monthly Virtual Warehouse bill.
- Serverless functions don’t factor into the 10% adjustment for Cloud Services.
Every day, there is a calculation of credit adjustment for Cloud Services, using the following formula:
Virtual Warehouse Cost (Daily) + Cloud Services Charge (Daily) – Cloud Services Adjustment (Daily)
Examples:
The monthly total of Cloud Services credits is the sum of the daily aggregations
How to Reduce Cloud Services Costs
Refactor Complex Queries
One of the things that utilizes Cloud Services is query compilation time. Long and complex queries result in high compilation time, which increases Cloud Services usage.
Solution
One recommendation is to periodically review and refactor old queries to ensure they’re doing precisely what you need them to do. This methodology also helps keep your queries up-to-date and gives you a chance to boost performance from time to time.
Use a Shared Schema
Using a separate schema per customer generates a lot of metadata operations, which affects Cloud Services resources.
Solution
One recommendation is to use a shared schema for all customers and utilize Secure Views. Additionally, for performance improvement, cluster the tables on the customer_id column.
“Innocent Queries”
Queries against the INFORMATION_SCHEMA might seem “innocent” as they aren’t triggering Virtual Warehouses.
However, the truth is that although they aren’t using Virtual Warehouses, they still consume Cloud Services.
Solution
In this case, try to oversee and monitor these “innocent” queries and consider querying the ACCOUNT_USAGE schema instead of INFORMATION_SCHEMA when possible. Additionally, use already up-and-running Virtual Warehouses.
Metadata Queries
Simple metadata queries such as SHOW commands might result in an explosion in Cloud Services credits
Take a look at this discussion 🔗LINK where the Metabase service keeps spamming SHOW commands without any reason, causing an explosion in Cloud Services credits.
Solution
Closely monitoring these metadata operations is crucial.
Another suggested strategy is to implement a caching mechanism in your applications or scripts to store metadata locally after the first retrieval.
By doing this, you reduce the number of redundant requests to Snowflake, thus lowering the Cloud Services costs associated with these metadata operations.
Conclusion
Managing and optimizing Cloud Services costs in Snowflake is crucial for controlling overall expenses and ensuring efficient resource utilization. By implementing the strategies discussed in this blog post, you can significantly reduce your Cloud Services charges and maximize the value you get from your Snowflake investment.
Remember, refactoring complex queries, using shared schemas, minimizing “innocent” queries against INFORMATION_SCHEMA, and implementing caching mechanisms for metadata operations are all effective ways to lower your Cloud Services costs.
Additionally, it’s essential to regularly monitor and analyze your Snowflake usage patterns, query performance, and costs to identify potential areas for improvement.
By staying proactive and making necessary adjustments, you can ensure that your Snowflake environment remains cost-effective while delivering the performance and scalability you need.
Embracing cloud cost optimization is an ongoing process, and the strategies outlined in this blog post can serve as a starting point for your organization’s journey towards more efficient and cost-effective Cloud Services utilization in Snowflake.