Unleashing API-powered agents at Microsoft: Our internal learnings and a step-by-step guide

Agentic AI is the frontier of the AI landscape. These tools show enormous promise, but harnessing their power isn’t always as straightforward as prompting a model or accessing data from Microsoft 365 apps. To reach their full potential in the enterprise, agents sometimes need access to data beyond Microsoft Graph. But giving them access to that data relies on an extra layer of extensibility.

To meet these demands, many of our teams within Microsoft Digital, the company’s IT organization, have been experimenting with API-based agents. This approach combines the best of two worlds: accessing diverse apps and data repositories and eliminating the need to build an agent from the ground up.

We want to empower every organization to unlock the full power of agents through APIs. The lessons we’ve learned on our journey can help you get there.

The need for API-based agents

The vision for Microsoft 365 Copilot is to serve as the enterprise UX. Within that framework, agents serve as the background applications that streamline workflows and save our employees time.

For many users, the out-of-the-box access Copilot provides to Microsoft Graph is enough to support their work. It surfaces the data and content they need while providing a foundational orchestration layer with built-in capabilities around compliance, responsible AI, and more.

But there are plenty of scenarios that require access to other data sources.

“Copilot provides you with data that’s fairly static as it stands in Microsoft Graph,” says Shadab Beg, principal software engineering manager on our International Sovereign Cloud Expansion team. “If you need to query from a data store or want to make changes to the data, you’ll need an API layer.”

By using APIs to extend agents built on the Copilot orchestration layer, organizations can apply its reasoning capabilities to new data without the need to fine-tune their models or create new ones from scratch. The possibilities these capabilities unlock are driving a boom in API-based agents for key functions and processes.

“Cost is one of the most critical dimensions in how we design, deploy, and scale our solutions. Declarative API-driven agents in Microsoft 365 Copilot offer a path to unify agentic experiences while leveraging shared AI compute and infrastructure. By aligning with core architectural principles such as efficiency, scalability, and sustainability, we can ensure these agents not only drive intelligent outcomes but also maximize value across service areas with minimal overhead.”

Faisal Nasir, principal architect, Microsoft Digital

In many ways, IT organizations like ours are the ideal places to implement API-based agents. Our teams are adept at creating and deploying internal solutions to solve technical challenges, and IT work is often about enablement and efficiency—exactly what agents do best.

“Cost is one of the most critical dimensions in how we design, deploy, and scale our solutions,” says Faisal Nasir, principal architect with Microsoft Digital serving on the AI Center of Excellence’s leadership team. “Declarative API-driven agents in Microsoft 365 Copilot offer a path to unify agentic experiences while leveraging shared AI compute and infrastructure. By aligning with core architectural principles such as efficiency, scalability, and sustainability, we can ensure these agents not only drive intelligent outcomes but also maximize value across service areas with minimal overhead.”

{Learn more about our vision and strategy around deploying agents internally at Microsoft.}

The Azure FinOps Budget Agent

Our Azure FinOps Budget Agent is a perfect example of a scenario for API-based agents.

The team responsible for managing our Microsoft Azure budget for IT services was looking for ways to reduce costs by 10–20 percent. To do that effectively, service and finance managers needed the ability to track their spending quickly, accurately, and easily.

The conventional approach to solving this problem would be creating a dashboard with access to the relevant data. The problem with a UI-based approach is that it tends to cater to more specific personas by providing data only they need while oversaturating others with information that’s irrelevant to their work.

“Azure spend is basically the lifeline for our services,” says Faris Mango, principal software engineering manager for infrastructure and engineering services within Microsoft Digital. “Getting the information you need in a concise format that provides a nice, holistic view can be challenging.”

With the advent of generative AI and Microsoft 365 Copilot, the team knew that a natural language interface would be much more intuitive. The result was the Azure FinOps Budget Agent.

The team created the agent and the necessary APIs using Microsoft Visual Studio Code. Its tables and functions run on Azure Data Explorer, allowing the APIs and their consumers to access data almost instantaneously, thanks to its low latency and rapid read speeds.

The tool retrieves data by running Azure Data Factory pipelines that pull and transform data from three sources:

• Our SQL Server for service budget and forecast data
• Azure Spend for the actual spending amounts
• Projected spending, a separate service stored in other Azure Data Explorer tables

Processing the information relies on our business logic’s join operations, followed by aggregations by fiscal year and service tree levels. These summarize the data per service, team group, service group, and organization.

After the back end processes the day’s data, it ingests the information into our Azure Data Explorer tables, which the agent accesses by calling via Kusto functions (the query language for Azure Data Explorer). The outcome is very low latency. Typically, the agent returns results in under 500 milliseconds.

For users, the tool is stunningly simple. They simply access Copilot and navigate to the Azure FinOps Budget Agent.

The agent provides three core prompts at the very top of the interface: “My budgets,” “Service budget information,” and “Service group budget information.” Clicking on one of these pre-loaded prompts returns role-specific information around budget, forecasts, actuals, projections, and variance, all at a single glance. The interface even includes graphs to help people track spending visually.

If users are looking for more specific information, they can input their own queries. For example:

• “Get me the monthly breakdown of service Azure Optimization Assessment analytics.”
• “Find me the service in this tree with the highest budget.”
• “Show me the Azure budget for our facilities reporting portal.”
• “Which service deviates most from its budget forecasts?”

The Azure FinOps Budget Agent primarily serves two groups: service managers who directly oversee spend for Azure-based services and FinOps managers responsible for larger budget silos.

Mango is responsible for the internal UI that helps network employees access parts of the Microsoft network. With 18–20K users per month, budgeting and forecasting are highly dynamic due to traffic fluctuations and the resourcing that supports them. He also oversees the internal portal that helps service engineers manage our networks. The tool is growing rapidly as we onboard more teams, so forecasting is anything but linear.

For both of these services, keeping close track of spending is essential. Mango finds himself checking the Azure FinOps Budget Agent about twice a month to gauge how his services are trending.

“It’s taking me less time to do analysis and come up with accurate numbers. And the enhanced user experience just feels more natural, like you’re asking questions conversationally rather than engaging with a dashboard.”

Faris Mango, principal software engineering manager for infrastructure and engineering services, Microsoft Digital

For FinOps managers, the value is more high-level. They are responsible for overseeing tens of services featuring vast volumes of Azure usage across storage and compute while managing strict budgets. That requires constant vigilance.

Switching context from one dashboard to another to track different Azure management groups was a constant hassle for them. Now, they use the Azure FinOps Budget Agent to get an up-to-date view of the overall spend picture. It gives them a place to start. From there, they can drill down if he sees any abnormalities.

“It’s taking me less time to do analysis and come up with accurate numbers,” Mango says. “And the enhanced user experience just feels more natural, like you’re asking questions conversationally rather than engaging with a dashboard.”

The arrival of the Azure FinOps Budget Agent is just one example of how agents take your context and get your people the answers they care about faster at less cost.

Benefits like these are spreading across teams throughout Microsoft. Overall, we’ve been able to save 10–12 percent of our overall Azure cost footprint for Microsoft Digital, and individual users are thrilled at the amount of time and effort they’re saving.

“Now the info is at people’s fingertips. The advantage of an agent is that users don’t have to understand a complex UI, so they can get quick answers and get back to work.”

Shadab Beg, principal software engineering manager, International Sovereign Cloud Expansion

Five key strategies for building an API-based agent

After seeing what we’ve accomplished with API-based agents, you might be wondering how to put them into action at your organization. This step-by-step guide can help you get there.

Building an API-based agent needs to fulfill multiple requirements. It has to expose APIs, align with real user needs, integrate seamlessly with Microsoft 365 Copilot, and work reliably, efficiently, and scalably. Achieving those outcomes depends on five key strategies.

Start by asking a simple but powerful question: What will users actually ask your agent? Instead of designing the API first, flip the process:

• Gather real user queries to understand actual use cases.
• Refine the queries using prompt engineering techniques to align them with expected AI behavior.
• Design the API to provide structured responses to those refined queries.

By starting with user intent, you ensure your agent answers real user questions directly, avoids over-engineering unnecessary endpoints, and delivers meaningful results without excessive back-end processing.

“Now the info is at people’s fingertips,” Beg says. “The advantage of an agent is that users don’t have to understand a complex UI, so they can get quick answers and get back to work.”

The advantage of an agent is that users don’t have to understand a complex UI, so they can get quick answers and get back to work.”

It’s important to build an API schema that returns precise and structured data to make it easy for Copilot to consume. This ensures your APIs return data in a format that directly answers user queries. Copilot expects responses in under three seconds, so focus on optimizing API responses for low latency.

Once you have your list of key questions, design your API schema to return the exact data you need to answer those questions. Your goal should be to ensure every API response has a structure that makes it easy for Copilot to understand.

Copilot needs to know how to call your API. Manifests and OpenAPI descriptions accomplish that training.

Create detailed OpenAPI documentation and plugin manifests so Copilot knows what your API does, how to invoke it, and what responses to expect. You’ll likely need to adjust to these files through a process of trial and error.  

Once you have your schema and integration in place, it’s time to move on to the primary engineering challenge: making your API scalable, efficient, and reliable.

Prioritize the following goals:

• Fast response times: Copilot expects quick answers.
• High scalability: This ensures seamless performance at scale.
• Reliable uptime: The system needs to remain robust.

We recommend setting a very strict latency limit while implementing your API to retrieve data, since Copilot needs time to generate its response. Existing API endpoints often involve complex data joins rather than simply returning rows from data tables. This complexity can lead to longer processing times, particularly with intricate queries that involve multiple data stores.

To address these potential delays, pre-cache results to significantly enhance performance. This can help overcome the latency requirements imposed by Copilot.

At this point, you’ll see why starting with user intent and iteratively refining API design is important. By grounding your work in user behaviors, you’ll align with the following best practices:

• Structure your response to directly address user queries.
  Instead of just returning raw data, the API should provide meaningful insights Copilot can interpret. Prompt engineering marries user intent with the most understandable API schema.
• Keep your API flexible enough to adapt to evolving business needs.
  Real-world workflows change over time, and an API should be able to support those changes without massive refactoring.
• Avoid performance bottlenecks caused by unnecessary complexity.
  Understanding the exact data requirements up front prevents heavy joins, excessive filtering, and inefficient data retrieval logic.
• Optimize for Copilot’s real-time response constraints.
  With a strict limit on latency, consider pre-optimization techniques like pre-caching results and simplifying query logic from the very beginning of your API implementation.

If you attempt to build a scalable, reliable API without first understanding how users will interact with your agent, you’ll spend months reworking the schema, debugging inefficiencies, and struggling with integration challenges.

Unlike custom agents or OpenAI API integrations, knowledge-only agents require far less effort to meet Microsoft’s Responsible AI Standard. Microsoft tools’ built-in compliance capabilities handle much of the complexity. As a result, you can focus on efficiency and optimization rather than regulatory hurdles.

“Agent-based automation must balance speed with responsibility,” Nasir says. “We embed compliance, cost control, and telemetry from the start, so our systems don’t just scale, they mature.”

APIs and the agentic future

Building API-based agents is more than just an integration exercise. It’s about creating scalable, intelligent, and compliant AI-driven workflows. By aligning your API design with user intent, you set Microsoft 365 Copilot free to retrieve and interpret information accurately. That leads to a seamless AI experience for your employees.

Thanks to Copilot’s built-in security and compliance features, API-based Copilot agents are some of the most efficient, compliant, and enterprise-ready ways to deploy AI solutions. They represent another step into an AI-first future tailored to your employees’ and organization’s needs.

Tools like API-based agents democratize the information we all need to do our jobs better, because we’re all getting the same data from the same place. This is why an AI-first mindset is actually human-first.