Cloud

How Capital One improves visibility into Snowflake costs


Were you unable to attend Transform 2022? Check out all of the summit sessions in our on-demand library now! Watch here.


[Ed. note: Corrected 8/3/22 from “It also helped reduce Snowflake costs by 27%, improve the cost per query by 45% and save over 55,000 hours of work.” to say “It also helped reduce Snowflake costs by 27%, improve the cost per query by 45% and save over 50,000 hours of work.”]

Banking giant Capital One has packaged its extensive work to migrate to the cloud into a new software business called Capital One Software. Its first product, Slingshot, helps enterprises manage cloud costs and automate governance on top of the Snowflake platform. 

This is big news for both companies. For Snowflake, this demonstrates the ability of highly regulated industries to go all in on cloud data migration. For Capital One, it reflects a way to monetize its extensive work in data management to unlock lucrative new opportunities in the technology industry. 

The new service helps manage costs through intelligent cost savings recommendations. It also improves insight and visibility into Snowflake costs and automated governance using custom workflows, dynamic warehouse provisioning and self-service capabilities. These capabilities were built by necessity as part of Capital One’s early transition to the cloud. 

All in on cloud data

Capital One was one of the first banks to go all in on the AWS cloud in 2015. Salim Syed, head of engineering at Capital One Software, told VentureBeat they believed the cloud could help scale the number of queries its 6,000 business analysts could run concurrently and make it easier to share data across the business. 

In addition, the cloud promised a way to enable workload isolation so queries from one team of analysts did not impact others. This had been a big challenge with their on-premise database, particularly at busy times of the month.

The cloud also suggested a way to scale and change directions quickly when needed. “We were betting on the fact that the cloud would become this place where apps could be developed quickly,” Syed explained.

In 2017 Capital One began migrating all its data to Snowflake, which helped automate many aspects of data management and sharing. However, this introduced new data management challenges. The platform’s simplicity allowed analysts to develop new analytics as fast as they could combine datasets. But they soon discovered that many of these queries were inefficient and often resulted in provisioning excess cloud infrastructure, which increased costs.

“The challenge was how to make sure you are using Snowflake as efficiently as possible,” Syed said. So, his team worked on a new management tier that helped create and enforce best practices across Capital One. 

Developing a new model

They found that the old model of trying to centralize analytics was not feasible, since central teams don’t have the expertise and domain knowledge to process data across business units. So, they developed self-service tools that empowered business units, with data management, cost controls, and automated governance baked in. This allowed them to onboard almost 450 new use cases since migrating from the on-premise data infrastructure. 

It also helped reduce Snowflake costs by 27%, improve the cost per query by 45% and save over 50,000 hours of work. For example, it can alert analysts when a query is inefficient so they can turn it off and create a more efficient one. It also helps ensure that teams don’t over provision data warehouses. For example, they found that development and testing teams often previously provisioned much larger instances than required.  

Governance as a service

In the early days of the cloud, many regulated industries were concerned with some governance challenges of moving their processes off their internal servers. Capital One’s success suggests that even some of the largest and most regulated businesses can succeed in going all in on the cloud. 

Nick Kramer, leader of applied solutions at SSA & Company, a global consulting firm advising companies on strategic execution, told VentureBeat, “Any Snowflake customer would have to be intrigued by Capital One’s application, particularly for governance.” 

All banks face stringent regulatory requirements and comprehensive, measurable governance is a cost of doing business. He argues that Capital One’s experience in ensuring governance combined with high data volumes, velocity and privacy, presents a compelling data management value proposition, particularly for medium-sized businesses without the resources to build comparable capabilities.

“The most promising features we see are the focus on and deep partnership with Snowflake, the modular building blocks for customization and the governance workflow,” Kramer said. This combination should enable Slingshot to accelerate feature development in response to customer feedback, underpinned with governance-enabled data quality and usability. 

Easing last mile analytics

The second key aspect of this partnership demonstrates how Snowflake is building an ecosystem to build on its core competency in cloud data management. Priya Iragavarapu, VP in global management consultant AArete’s Center of Data Excellence, pointed to Snowflake’s lead in massively parallel processing, intelligent indexing and expedited querying as key differentiators. 

This enables fast analytics, but leaves it up to each enterprise to figure out how to implement these. Iragavarapu said the analytics user journey is disparate since it involves connecting to the right data source, submitting queries via the Snowflake user interface (UI) and then extracting the data to visualize in more powerful ways using tools like Tableau or Power BI. 

Capital One simplified this process by eliminating the need to switch apps by incorporating content management, project management and communication within a single tool. Last mile analytics refers to the enhancements required to complete the final leg of the analytics journey. This is where Iragavarapu sees most inefficiencies because of the need for customizations, limitations on automation and the need to cater to varied, complex team dynamics and communications. 

This partnership between Capital One and Snowflake addresses this gap.

“The analytics user journey will never be the same again,” she declared. 

A model for the banking transformation

Eventually, this suggests a way for banks to not only innovate, but also gain a leg up on new financial technology startups nipping at their heels. 

Ronak Doshi, partner at Everest Group, an advisory firm, said, “This FinTech revolution is putting technology as the growth driver for banks and the lead steers in the market like Capital One are now in an advantaged position to take these early bets and monetize them.”

Doshi observed that FinTech startups have market valuations of 20-30 times their revenue, which is three to four times higher than large banks. The pivot to a technology organization providing financial services will drive valuation for Capital One as they scale their software business. It could also provide an on-ramp to the payment and banking service business for Capital One. 

Banks and payment companies are embedding their financial services and products using APIs into the leading software -as-a-service (SaaS) products. 

“The next stage is banks offering these SaaS solutions and providing the banking and payments services APIs out of the box, creating stickiness with consumer relationship and owning the end-to-end experience,” Doshi said. “We see this future of banking as open finance or embedded finance to be a massive growth driver for the industry, and lead steers that have invested in these software businesses will have the capability to capture this opportunity.”  

It also suggests how Snowflake will continue to grow its lead in the data management space. Iragavarapu said, “As long as vendors are trying to address a particular pain point using data, there is no need for them to re-invent the wheel and can effectively leverage the framework Snowflake already has.”

The one caveat is that Snowflake is focused on online analytic processing and is not optimized for transactional application needs. She believes enterprises will turn to other databases for transactions depending on the size, structure of data and need for aggregation.

VentureBeat’s mission is to be a digital town square for technical decision-makers to gain knowledge about transformative enterprise technology and transact. Learn more about membership.



READ SOURCE

This website uses cookies. By continuing to use this site, you accept our use of cookies.