Serverless Data Architecture: The Modern Path to Digital Transformation

By Ed Anuff, Chief Product Officer, DataStax

When it comes to digital transformation, data architectures have gotten short shrift.

Many enterprises have focused on modernizing by moving applications to the cloud, or building ecommerce offerings (if they are retailers). But in many cases, data has been left out of the digital transformation story because data systems tend to be large, monolithic, fragile, and difficult to deal with—in other words, there are significant risks to the business if the modernization process doesn’t go as planned. Many companies have turned their attention to easier-to-manage projects, leaving data platform modernization as a challenge for a later time.

I’d like to discuss a couple of misconceptions that can hinder data architecture modernization efforts, and a key approach to enable this kind of transformation.

Two misconceptions about digital transformation

There is a misunderstanding that often crops up when enterprises are considering digital transformation: it is a monumental, all-or-nothing task that suddenly morphs a traditional company into a modern, digital enterprise.

But think about how, from a high-level viewpoint, retailers transformed themselves. A traditional brick-and-mortar retailer didn’t just reboot itself and, voila, it’s an ecommerce company. Rather, ecommerce started out accounting for, say, 2% of sales, then 10%, then 20%, and so on.

It’s the same for most enterprises across verticals. Transformation is a step-by-step process that usually starts small—one initiative at a time (from a project to a program, to, eventually, a platform).

There is another misconception that still exists: innovation is driven from the top down. Generally, however, technology leaders optimize and scale successful processes that have started within their organization, and create the platforms for innovation that arise from projects. But those projects are usually built by developers. Digital transformation bubbles up from experiments that start small and, if they’re successful, often need to scale fast. This presents a challenge, particularly when you think about modernizing an organization’s data estate.

Equipping developers for transformation

For digital transformation to succeed, developers require the ability to quickly start a modest project and be prepared for it to grow explosively when needed—without having to pause and perform scalability testing, or worry about how much funding the project will take, or how much latency it might introduce into a system. Spread these requirements over multiple new projects, and the demands for powerful, scalable, and easy-to-use data platforms become critical to success.

However, databases for a long period of time made this kind of work challenging. Scaling up and down took time and effort, which often led to costly overprovisioning.

A select group of cloud databases, including services provided by DataStax and MongoDB, is making it easier for developers to focus on their modernization projects–without the distractions of provisioning, scaling, and other aspects of data management–by offering “serverless,” or “pay-as-you-go,” data. By separating the compute and storage functions, scaling up or down becomes easier and faster.

A serverless architecture precisely matches data usage to workload peaks and valleys. The pay-as-you-go architecture eliminates the costly and labor-intensive task of estimating peak loads and enables developers to pay only for what they use—no matter how many database clusters they create and deploy.

Serverless data is making a real difference at Circle Media Labs, a provider of apps and devices to help parents manage their family’s time online. The company relies on Astra DB from DataStax, a serverless, multi-cloud database-as-a-service (DBaaS) built on Apache Cassandra^®.

Circle’s former principal engineer Nathan Bak used this pay-as-you-go version of Cassandra, the high-performance, open source, NoSQL database, to test multiple product and service ideas that make use of the company’s trove of data. He mentions that with a serverless database, the contention of who gets access to a database for building proofs of concept disappears; everyone can run a project on their own Cassandra cluster.

“I probably have half a dozen serverless databases with POCs running on them that might not go anywhere, but I can keep them running because it’s costing just pennies, and the data isn’t lost,” Bak says.

One of these projects blossomed into a successful new product for Circle, and so it needed to scale quickly.

“This project went from me working on it on and off—with maybe a megabyte or two of data. But then it pretty quickly multiplied 1,000-fold—and then 10,000-fold,” Bak says. “There were plenty of things to worry about as that project grew. The database wasn’t one of them.”

Cassandra without limits

Even as a cloud-native company, Circle benefited from the ability of a serverless database to enable the development of many projects at once, without the infrastructure concerns. Having this kind of access to Cassandra can be particularly empowering in digital transformation projects.

Cassandra has historically been a database that enterprises turned to when they became aware of a need to scale massively: if nothing else could handle a project that needed solid reliability at scale, Cassandra was the answer. It’s why Netflix, Best Buy, Bloomberg News, and many other enterprises bet their businesses on the database.

But with today’s serverless technologies, developers now have access to all of Cassandra’s perks without the costs and administration requirements.

It’s easy to get overwhelmed with any digital transformation efforts, whether you are modernizing your data architecture or any other part of your organization. A key strategy to work through this potentially paralyzing challenge is to understand that your developers are critical to the success of any transformation effort. To do their part, development teams need tools that enable them to attack projects that, if successful, are ready and able to grow to enterprise scale.

Learn more about DataStax here.

About Ed Anuff:

Ed is chief product officer at DataStax. He has over 25 years experience as a product and technology leader at companies such as Google, Apigee, Six Apart, Vignette, Epicentric, and Wired.