The partnership enabling Pegasystems to maximize open-source potential

Pega Infinity is Pegasystems’ low-code application development platform designed for workflow automation, customer engagement, AI decisioning, and RPA. But for it to work properly, it requires a wide range of open-source software and services.

In the past, the company, based in Waltham, Massachusetts, put all these different services into Infinity. “But because everything was embedded inside the software, each time we wanted to scale up a specific service, we were essentially scaling up the entire platform,” explains Ramzi Souri, Pega VP of cloud technologies. To do so, their internal teams were constantly managing different open-source data infrastructure components, including data technologies like Apache Cassandra, Apache Kafka, and OpenSearch, which meant they didn’t have time to focus their energy on moving the business forward.

As a result of this flawed strategy, the team decided to break everything up and have independent services available on the platform. “We had two options,” says Souri. “Either we needed to build everything ourselves and hire service teams and software development teams to focus purely on spinning up the different services, or we could look for a third-party vendor that met our security guidelines, operational style, and our deployment needs. We’re currently deployed on AWS and GCP, so this third party also had to make it possible for the service to run across multiple clouds and be able to offer the same level of service across all clouds to future-proof the solution.”

In search of a like-minded partner

Souri and his team certainly had their work cut out for them to find the right solution, or, more specifically, the right alliance. They ultimately went with NetApp Instaclustr, a managed data infrastructure platform that essentially functions as an extension of Pega’s operational arm for several reasons.

“This service was compelling for us because it isn’t a black box solution running in an environment that we can’t see or control, like many other services we considered,” he says. “This model means everything’s deployed in an account that we have full access to and own.”

Now the services from the Google and AWS accounts, which Pega owns and manages, are deployed via Instaclustr and linked back to Infinity. If a new version of Cassandra is released and Pega wants to migrate to that version, for example, they simply inform Instaclustr about the upgrade. The Instaclustr team will do all the work in the background and notify the Pega team when the service is ready.

“Similarly, if we’re looking to spin up a new Cassandra cluster for a client, we can send a request through an API that lives on the platform side to provision a new cluster, and that service gets provisioned on an account that we own,” Souri adds. “It’s truly an extension of our existing engineering and operational teams.” But having everything so closely intertwined wasn’t without complications.

Relinquishing control

One of the early challenges the team faced was making changes to existing infrastructure. “If we needed to scale up a cluster due to performance issues, we had to follow a strict approval process,” says Souri. “This meant opening a ticket and going back and forth for approval. Such a lengthy process delayed things so by the time approval came through, the performance issue may have already had a negative impact on the client’s environment.”

Realizing how inefficient this process was, Instaclustr was given more control to make decisions without seeking approval. The last thing Souri and his team wanted was to have a negative impact on clients because of lengthy wait times to get approvals. “We had multiple meetings with the team where we walked them through different scenarios, and agreed on when they’re authorized to act on our part,” he adds. “This was important because every time they did, it increased our infrastructure cost, so we needed to be clear about when it’s okay for them to do that.”

Now teams can concentrate their efforts on solving real business problems, without having to worry about managing external resources. Pega estimates the shift has saved the business over 60,000 hours annually, based on a team of 30 engineers working for 40 hours a week. The team also believes they’ve been able to avoid hiring more engineers by freeing time for existing teams.

Looking back on the project, Souri’s advice is to automate everything you can, from the testing, provisioning, and monitoring, to the validations, upgrades, and updates — and do it early. “This will save you a lot of time, not only today but also down the road when things get bigger and more complex,” he says.