CIOs listen up: either plan to manage fast-changing certificates, or fade away

It’s tempting to view certificates and machine identities as the nuts and bolts of your organization’s tech stack: critical but purely technical functions safely left to IT practitioners. But a string of recent disruptions and upcoming policy changes are forcing a more proactive, agile, and strategic approach from the top to signal the importance of getting these fundamentals right.

“PKI and cryptography have always been very low-level, in the weeds but foundational for security even though CIOs probably haven’t paid much attention to it,” says Christian Simko, VP of product marketing at low-code automation platform AppViewX. “Now it’s much more in the spotlight as you’ve got machine identity management, non-human identity management, and post quantum cryptography all becoming hot button items that are going to impact security and compliance across the organization. One you start to bring compliance in, the CIO starts to take a little bit more notice as well.”

Mail and machines

You may have noticed emails in your inbox being tagged as not just from outside your organization but from unverified senders. That’s part of a push to improve adoption of existing standards for email reputation, which rely on certificates for authentication, that the majority of organizations have simply ignored, even as email threats have increased.

“It should be best practice, but you’d be surprised how many organizations lag behind and haven’t fully and effectively implemented it,” says Simko.

Earlier this year Google and Yahoo, soon to be joined by Microsoft, began requiring any organization who’s ever sent out 5,000 messages in a single day to use SPF, DKIM, and DMARC email authentication, as this will affect password resets, shipping notifications, and purchase receipt emails going to consumer email addresses, as well as newsletter and marketing material sent directly or via email providers like Mailchimp. And while consumer email providers are starting with enforcement against bulk senders that still allow them to continue with lax DMARC policies, strong email authentication protections are relevant to every organization, and may well be enforced more stringently and more broadly in the future.

Not only do enterprises need to make sure DMARC is implemented properly across all their domains and subdomains — even if you don’t use them to send email, you need to protect against a hacker spoofing them as an email source — they need to monitor reports of rejected emails because of DMARC failures. And this is something CIOs will need to budget time and resources for, either internally or through a mail security provider. They should also consider tagging and rejecting incoming email that doesn’t meet the reputation standards. This can significantly reduce phishing emails, but the potential impact on legitimate communication from other organizations who haven’t yet adopted email reputation standards makes it a strategic rather than technical decision.

But certificates for email are only a small part of the problem. Thanks to the adoption of complex infrastructure like IIoT, JSON Web Tokens, and Kubernetes, among others, organizations already have in use hundreds of thousands of machine identities secured by SSL/TLS certificates, with lifespans ranging from years to minutes. A single physical device can run hundreds of ephemeral workloads. And certificates are often poorly managed and secured, even by organizations in regulated industries like finance. Erik Wahlstrom, VP analyst at Gartner calculates that machine identities typically outnumber human by an order of magnitude. That number is still growing, especially with the adoption of AI tools that need credentials for both the systems they access and the humans they’ll act on behalf of. Coleman Parkes research for automation vendor Venafi predicted that organizations with more than 10,000 employees will have as many as 1.3 million machine identities and certificates to deal with by 2025.

But manual scripts, spreadsheets and homegrown automations don’t scale to support those numbers, especially as most enterprises have poor visibility of how many certificates and machine identities they’re already using. “When people do fuller discovery and inventory, they’ll be shocked at how quickly that number grows,” says Geoff Cairns, principal analyst at Forrester Research.

It doesn’t help that workloads are digitally stored in divergent databases and user-stores. “They all need identifiers and credentials such as certificates, secrets, and keys, and organizations must manage it all,” Wahlstrom says. Typically, though, that’s not happening, and the scope of this challenge is underestimated, he adds, without a tool available to organizations to help them manage all of this across their hybrid and multi-cloud environments.

The industry lacks maturity, suggests Matt Caulfield, VP of product at Cisco security subsidiary, Duo, with “no standard machine identity directory and a confusing array of machine to machine (M2M) authentication and authorization protocols,” he says.

Another problem: the issues cross organizational boundaries and traditional identity, and access management teams who run existing public key infrastructure, like Active Directory, are often not involved in all the initiatives that require certificates and machine identities — or even the discussions about how to manage the problem.

“Cloud has exponentially made this problem worse,” warns AppViewX chief solutions officer Murali Palanisamy. “PKI is typically set up and managed by the internal Microsoft team, and they don’t have much say in cloud decisions and digital transformation. The DevOps team is more about speed and agility: they do what’s good to get things faster, not necessarily what would be strategic or the right thing to do or what’s secure.”

Even when organizations finally decide to set policies and standardize security for new deployments, mitigating the existing deployments is a huge effort, and in the modern stack, there’s no dedicated operations team, he says.

That makes it more important for CIOs to take ownership of the problem, Cairns points out. “Especially in larger, more complex and global organizations, the magnitude of trying to push these things through the organization is often underestimated,” he says. “Some of that is having a good handle on the culture and how to address these things in terms of messaging, communications, enforcement of the right policies and practices, and making sure you’ve got the proper stakeholder buy-in at the various points in this process — a lot of governance aspects.” And a spate of recent incidents and new proposals make this more urgent.

Faster rotation required

The combination of identity sprawl and poor identity hygiene creates vulnerabilities. Certificate issues already routinely cause outages and security problems inside organizations, often because expiry notifications go unnoticed, even inside technology providers as large as Microsoft. These issues also give attackers access to cloud infrastructure, enterprise workloads, and the software supply chain when credentials are poorly managed and secured.

Sometimes the results are merely embarrassing but an expired certificate breaking TLS traffic inspection at Equifax led to the massive data breach back in 2017. “Complexities brought in by cloud adoption and DevOps — all these things that are driving exponential growth of machine identities set organizations up for an incredible amount of identity sprawl and susceptibility to attackers who want to move within the organization,” Cairns says.

The combination of increasing scrutiny and commoditization of certificate authorities by cloud vendors and new providers like Let’s Encrypt is going to put more pressure on enterprises, making poor internal practices unsustainable. “The TLS certificate market became as much of a procurement and cost issue as anything else, but some of the recent incidents bring things back to more of a focus on the underlying trust and security,” he adds.

Timing is everything

Many large organizations will soon need to revoke and reprovision TLS certificates at scale. One in five Fortune 1000 companies use Entrust as their certificate authority, and from November 1, 2024, Chrome will follow Firefox in no longer trusting TLS certificates from Entrust because of a pattern of compliance failures, which the CA argues were, ironically, sometimes caused by enterprise customers asking for more time to deal with revocation. Browsers will show security warnings rather than completely block access to sites using Entrust certificates issued after October 31, 2024, but many organizations will want to change CA to avoid users losing trust in their sites.

This also sets a precedent for tougher enforcement of browser certificate security policies, after years of tension between browser vendors and CAs. Even without that, DigiCert customers, including those operating critical infrastructure like healthcare systems and telecoms networks, recently had to replace over 83,000 TLS certificates, mostly with only 24 hours’ notice, because for the last five years, its self-service customer portal didn’t create the records for DNS verification correctly. Those notifications arrived via email on a Monay morning, leaving many organizations scrambling to rotate certificates across multiple systems.

Currently few organizations are set up to handle these issues effectively, but this is an expertise they will need to develop because these issues will continue to happen, Palanisamy warns. “Lightning has struck many times,” he says. “This is not a rare thing. This is going to be the norm, so we have to prepare for it.”

He suggests that part of the problem is certificate authorities needing to maintain root certificates for key generation that are valid for decades. “In a digital world, it’s almost like dinosaur years,” he says. “You have to keep the key alive and kicking for 30 years. You need to maintain and manage it. This is critical infrastructure that needs to be maintained for a long time.” Staff turnover and decreasing margins make that more challenging, too.

Ironically, Google’s proposal to reduce the validity of TLS certificates to just 90 days might ease that burden, because the root validity requirements would drop to five to seven years. But if this becomes a reality, enterprises will have to rotate certificates so frequently, they’ll need automation to keep up. Public-facing TLS certificates used to be valid for up to three years, but that was reduced first to 825 days (two years plus the 30 days before expiration when they should be renewed), and now 398 days (one year plus 30 days). Google recently proposed shortening that to just 90 days, meaning certificates would need to be renewed every two to three months. It’s unclear if the CA/Browser Forum will support the change, but if Google chooses to implement this for Chrome anyway, its substantial market share could push certificate authorities to make 90 days the default, requiring up to six times as many certificate renewals.

That may be too aggressive a timescale for most organizations. Palanisamy called it a forcing function that would require better controls, practices, and security baselines. “It’s not practically possible to do this manually, for every device and every certificate in the organization,” he says.

Manually revoking and replacing certificates means using multiple systems to generate and distribute keys, typically taking one or two hours per certificate; homegrown automations might be able to do it in 15 to 30 minutes, though. Effective automated systems integrated with a range of CAs, which means working with a range of APIs, SDKs, agents, and the ACME protocol, as well as with enterprise systems and popular DevOps tools, can generate new keys in a few seconds or minutes if there’s a cost for the certificate that needs to be approved as part of the workflow. That should be efficient enough to do proactively shortly before expiry, getting the maximum use out of short-lived certificates.

Even without 90-day certificates, organizations need to plan ahead for adopting NIST’s newly approved set of post-quantum encryption standards for both general encryption and digital signatures. Certification authorities aren’t likely to be ready to issue them until 2026, but that still leaves a short time to prepare for the extra complexity it’ll add to certificates.

To avoid organizations needing to install duplicate TLS certificates — one for clients that understand new algorithms, and one for those that don’t — TLS is standardizing on hybrid encryption suites that use both traditional and post-quantum encryption algorithms. That’s intended to simplify the transition, but also reflects the fact that while they can actually be faster, many post-quantum algorithms are new and haven’t been scrutinized as deeply as current cryptography approaches, so they may need to be rolled back if vulnerabilities are discovered later.

As with the transition from TLS 1.2 to 1.3, there’ll likely be some issues as this rolls out. When Chrome and Edge 124 enabled Google’s hybrid postquantum TLS key exchange by default earlier this year, web servers that didn’t implement TLS correctly began rejecting connections. In fact, compatibility issues detected in earlier testing by Google and Cloudflare have already delayed browser rollout of post-quantum keys by several years.

Some cloud services already support hybrid post-quantum key exchange, and it’ll become standard soon. CIOs need to be ready for not just testing systems and services for compatibility, but the added load of managing certificates and machine identities for the post-quantum transition long before quantum computers that can break traditional encryption are a reality, because information that’s harvested today may still be sensitive when it can be decrypted.

Whether quantum computers or other changes are the cause, organizations need to be able to switch to new, more secure cryptographic algorithms when necessary. “The day will come when the organization’s secrets and certificate management technologies will experience an inevitable and massive transformation that obsoletes them,” Gartner’s Wahlstrom warns. “It’s only a matter of time. Crypto-agility is all about how fast you can get back up on the other side.”

Building your machine identity strategy

Enterprises are starting to pay attention, Wahlstrom says. Gartner’s IAM team took twice as many calls about managing machine identities as about multi-factor authentication (MFA) last year. “Not because MFA isn’t important, but managing machine identities is extremely urgent for organizations right now,” he says.

To regain control, he advises CIOs to establish a strategy for certificates and machine identities, starting by defining their own scope of what’s included, and ground it in technical needs to ensure they don’t get swayed by vendor-hype.

Consolidation is happening, with large identity solutions providers acquiring smaller, specialist machine identity vendors, and the business value of the tools increasing. But dealing with machine identity requires a lot of work. “There’s no single solution today that’s going to solve all aspects of this,” says Cairns. “It starts with a governance process that understands the organization’s business needs, as it relates to machine identities, and how they’re being used. Understand those use cases, and the specific and unique requirements of the organization, and then move into your technology selections.”

CIOs will need to build a cross-functional team that bridges the traditional tooling siloes in enterprise identity, and assign responsibility and accountability across the evolving identity fabric for machine identities, Wahlstrom recommends. “The things we see most organizations fail with is not establishing ownership across their many business units involved, and jumping straight into automating everything now,” he says. “Establish processes, do discovery, and set expectations on how much you can automate.”

Inventory needs to include continuous discovery and audit rather be a one-off. NIST’s 2020 certificate lifecycle framework (SP 1800-16) is a good starting point, covering the risks and best practices for large-scale TLS server certificate management, including automated issuing, renewal, and revocation processes, but there are still plenty of questions for enterprises to consider, from key lengths and algorithms to certificate procurement and ownership. If you decide to use multiple CAs to minimize the risks of revocation and availability, how will those be allocated? How are the configurations for certificates tracked to organizational structure, purpose, and risk?

Organizations that do any business with the US federal government are already encouraged to automate security processes, including identity and certificate management. But Cairns cautions against rushing to automate before understanding what certificates and machine identities you have and what your use cases are.

“Certificate, SSH key, secrets, and non-human identity strategies don’t all look the same,” he says. “For example, to better control the rotation of certificates, set up a certificate life-cycle management practice. Gain visibility into your certificates by inventorying them, maintain up-to-date information about them, and implement expiration alerts. Then, not before, ensure uninterrupted operation in high-volume and/or short-life-span certificate use cases by implementing automation and non-standard-based lifecycle management when needed.”

The recent issues may give CIOs a feel for how much work will be involved in creating and putting their strategy into play. Along with the Google proposal Cairns suggested, they are wakeup calls for enterprises. “These incidents highlight the business realities and the fact you need to take care of this for your own business purposes, agility, and resiliency.”