How to win at AI: think like a systems designer, not a tech shopper

I spoke with IDC’s Katie Norton, Research Manager, DevSecOps and Software Supply Chain Security, at IDC’s 60th annual Directions conference. As an industry analyst Katie spends her life working with and advising senior IT decision makers around technologies such as agentic AI, sharing her deep domain expertise.

You can watch our conversation here, or via the YouTube player below.

AI: Short term vs long term

I asked Katie about the state of play with AI and the IT buyers she works with – What’s top of mind for them?

She said the productivity promise is driving short-term decisions. Buyers are under pressure to boost developer productivity and reduce toil — so AI tools that can deliver quick wins are top of mind, even if the long-term questions such as integration, governance, security, and ROI are being deferred. (See also: Can AI solve your technical debt problem?)

AI inside vs AI outside

We spoke about the complaint we both hear often from CIOs: a sense of frustration that ‘AI’ and ‘AI projects’ are lumped together as one amorphas thing. In response Katie speaks about the concept of AI inside vs AI outside.

‘AI inside’ refers to AI embedded in the tools and platforms IT already uses — think copilots in dev tools, AI-powered observability, or smarter firewalls. ‘AI outside’ is where organizations are building or fine-tuning their own models, integrating AI into their own applications or business logic. The risks, benefits, and investment profiles of inside and outside are completely different — but they’re often discussed as if they’re the same. Helping CIOs separate those two tracks is often the first step toward a practical AI strategy.

AI inside has been a major focus for most organizations: applying AI to internal processes and functions to accelerate output and drive efficiency.

One challenge is demonstrating return on investment. Are organizations demonstrably seeing ROI today? How is it being measured (if at all)?

Katie feels this is relevant especially to tools such as coding assistants or DevOps AI agents that promise efficiency gains. ROI is more anecdotal than analytical right now. She gave the example of ‘developer productivity’. Developers may appreciate AI support, but organizations struggle even to define what productivity means in this context. Many of the AI vendors have produced studies but organizations struggle to measure material ROI themselves, lacking tools and the visibility into workflow and output.

So how about AI outside? Increasingly we are seeing organizations using AI to build transformative products and services, so what is the impact of AI on app development? What are some of the potential implications of baking AI into customer facing apps?

According to Katie the upside is huge, but so are the potential security implications. The upside of ‘AI Outside’ is enormous — differentiated experiences, smarter products, more engagement — but it demands a fundamentally new approach to secure development. AI has to be treated as an untrusted input, so specific AI security tools and tests need to be integrated into workflow and output.

The AI-ready organization

For any business to generate ROI in AI, inside or out, the organization needs to be AI ready. I know from my own conversations with IT buyers that there is some tension between what AI can do conceptually, and what AI can do for a living organization, warts and all. IT leaders need to get their organizations AI ready.

We also see lots of IT professionals seeking work with AI able to take on some functions previously done by humans, and yet we hear that there is a critical skills gap that IT leaders are struggling to fill – how should IT buyers build a winning team for an AI-infused organization?

Katie takes the view that AI is reshaping the workforce, but not in the binary way people fear. What we’re seeing is a shift from automation to autonomy — from AI that helps with a task, to agentic AI that can make decisions, trigger actions, and coordinate across systems. That raises the bar not just for technical execution, but for oversight, orchestration, and trust. So when we talk about building a winning team, it’s not just about hiring data scientists — it’s about building teams who can manage and design systems where humans and AI are co-workers. (See also: How agentic AI makes decisions and solves problems.)

And one piece of wisdom for the IT leaders Katie advises? AI in 2025 isn’t about chasing the next model — it’s about building the muscle to deploy it responsibly, repeatedly, and at scale. IT should think like a systems designer, not a tech shopper. It’s not about which LLM you choose or which copilot you deploy — it’s about how AI shows up across your architecture, workforce, and risk model.