Eight steps for managing your support team content with AI tools

Like many large organizations, our support team develops thousands of troubleshooting guides, self-help resources, knowledge base articles, and process documents to guide our support engineers in helping customers with technical issues.

As we have grown over the years here at Microsoft, and with the continuous release of new products and features, these document repositories have become incredibly large, sometimes unmanageable; they also occasionally contain outdated information.

Back in July 2022, our Modern Solutions & Support (MSS) Supportability team had an idea for semantic search. This tool would potentially allow our support engineers to go to a single location and enter a search query that would scan the vast support document repositories, returning the most relevant results.  

Over the course of the year, team members explored different solutions and had conversations that eventually led them to the world of generative AI. The team knew they were on the cutting edge of something exciting with lots of possibilities. Then, a few months later, ChatGPT was announced and flipped the world on its head.

“It was fascinating how fast things were changing,” says DJ Ball, a senior escalation engineer on the MSS Supportability team. “OpenAI was so new, and its features were growing so fast—from GPT 2.0 to 3.0 to ChatGPT to 4.0 almost overnight. Keeping up with the technology is a challenge and big opportunity.”

Joining the generative AI revolution

The team quickly shifted gears, secured subscriptions to Microsoft Azure OpenAI, and stumbled upon an internal GPT playground focused on handling enterprise content. This playground happened to be very similar to what the Supportability Team was designing, which made joining forces much easier.

“People thought we planned it, which we didn’t, but for once we felt we were ahead of the game,” says Sam Larson, a senior supportability PM on the MSS Supportability team.

Then the development of an AI-based solution really picked up speed. No one had done what they were attempting to do with this ChatGPT technology, so they had to learn by digging in, playing around, and seeing what would happen. The MSS Supportability team provided continual feedback to the engineering team about what worked and what didn’t. This helped shape the product that became  Microsoft Azure AI Studio, a solution that makes integrating external data sources into the Microsoft Azure OpenAI Service simple.

With this development, the team was allowed to create their own private chat workspace, which they called Modern Work GPT (MWGPT). The Modern Solutions & Support Supportability team started by curating content from different sources for the Teams product and injecting that into the large language model (LLM).

By taking advantage of Azure Cognitive Search to help inject and chunk the documentation into smaller components, they were able to test the results with the help of subject matter experts (SMEs) across the Teams support business. The team has since expanded the tool to include all Modern Work Technology solutions—estimated to be more than 300,000 pieces of content for 34 products. Along the way, they’ve learned a lot about content curation, prompts, use case scenarios, and how LLMs work.

The team quickly realized that they needed more people to help them test the tool. Today, they continue to refine the content, updating its capabilities and testing it for accuracy.

One of the early volunteers was Mayte Cubino, now an AI strategy and programs director for the MSS organization. An engineer at heart, Cubino was excited by the rumblings she was hearing across the business about the possibilities to use ChatGPT in supporting customers.

From a support delivery standpoint, two questions stood out for Cubino:

• How could we ensure a successful deployment of this new technology across all our support engineers?
• How could this technology be the most helpful, without creating extra work for anyone?

Cubino started to document the content process. She helped the team see that some things were non-negotiable and outlined the steps they needed to focus on to ensure accuracy and responsible engagement with the model.

“There’s no question that there are a lot of variables that come into play that we are exposing our engineers to, and quality is a non-negotiable factor,” Cubino says. “We owe it to our customers, who turn to our support engineers to help them solve their most challenging technical problems.”

Unpacking our ‘8 Ds’ framework

This documentation process led to what became our “8 Ds” framework. These steps are designed to help develop agentic AI systems that don’t simply respond to inquiries, but also observe, decide, adapt, and learn.

Agentic AI that supports everyone must be built by—and for—everyone. That means training on linguistically and culturally rich data, designing with accessibility in mind, and continuously checking for exclusionary patterns.

When developing AI agentic models, begin by securing a breadth of viewpoints from different disciplines, demographics, and experiences. Diversity at the outset shapes every later choice, from dataset curation to interface tone, and is the surest guardrail against hidden bias and narrow problem framing.

Diversity is not a feature—it’s a foundation. Neurodiverse, multilingual, and underrepresented voices must be heard and embedded into the system. From pronoun recognition to language variety, a truly inclusive AI model builds trust not through perfection, but through recognition.

With a diversity lens in place, gather the data the agent will learn from. In the landscape of AI-powered support, everything begins with data. But not just any data. It must be rich, diverse, and contextual—support tickets, chat logs, voice transcripts, and knowledge base entries all serve as the lifeblood for AI model training.

Data isn’t merely an input—it’s a reflection of the lived customer experience. By curating high-quality, current, and representative datasets, organizations can shape AI models that understand nuanced intent, avoid bias, and respect privacy. Quality, representativeness, and clear provenance are critical here; rush this step and every downstream phase inherits the flaws.

In the design phase, translate the problem and the data into an architecture, user journey, and success criteria. Good design turns raw inputs into a coherent blueprint, aligning technical possibilities with real-world user needs. Ideation, testing, and prototyping are activities that govern this phase.

When preparing for AI implementation, you should identify how you can best integrate AI with your existing technology and services. It’s particularly useful to consider the following topics:

Content creation

Determining how you collect and store your data is vital. Among the key considerations are:

• Data collection pipelines that support reliable model performance and a clean input for modelling, such as batch upload or continuous upload.
• Storing your data in databases; the type of database you choose may change, depending on the complexity of the project and the different data sources required.
• Data mining and data analysis of the results.

“Inputting all of your content into an AI model is like taking a flashlight and shining it into every dark corner of your content,” says Jason Weum, director of supportability on the MSS Supportability team. “You quickly realize what’s outdated.”

Responsible AI review

Plan for security before you start: Make sure you design your system to keep the data secure, designing for responsible AI and compliance with General Data Protection Regulation (GDPR) and other policies and standards. For example, Microsoft’s Responsible AI principles include fairness, reliability and safety, privacy and security, inclusiveness, and transparency and accountability.

Other design elements

Be sure to think about how to use technology efficiently and plan for how to train your AI model:

• Evaluate the platforms your team uses to collate the technology used across the AI project to help speed up AI deployment.
• Consider the network and memory resources your team needs to train your model, in addition to the ongoing costs. Writing and training algorithms can take a lot of time and computational power.

It’s time to build—to code the pipelines, train the models, and craft the UX. Since design decisions are settled, developers can iterate quickly, without constantly revisiting fundamentals. During the development phase, you need to create and test grounding data sets.

This is a highly iterative process requiring substantial amounts of data. The following steps are critical during this phase:

• Content preparation: This includes content preparation and data quality assessment using a combination of accuracy, bias, completeness, uniqueness, timeliness, validity, and consistency.
• Content ingestion: This involves ingestion of curated data content, using the required formatting for the model. For example, larger documents should be chunked into smaller sections before ingestion.
• Fine-tuning: We fine-tune the model to distinguish between (for example) a refund request and a product bug, while embedding empathy, precision, and flexibility into its responses. At this stage, we also harness tools like reinforcement learning and simulation environments to elicit a desired response from the model or to prevent it from generating certain types of output. The prompt can also be appended with grounding data from the curated content. This is an iterative process and may require several rounds of testing to achieve the desired results.

Rigorous and uncompromised testing and training are crucial before you proceed towards deployment, but this can be a time-intensive process. Even the best-built models can drift. Here are some best practices for dealing with this challenge:

• Systematically test the models, trace any errors, and harden performance. Catching edge-case failures before the agent starts making real decisions is far cheaper—and safer—than patching a live system.
• Debugging an agentic AI model is a continual discipline, not a one-time task. It requires end-to-end monitoring to check for hallucinations, regressions, and unintended behaviors. We rely on telemetry, customer satisfactions scores, and human-in-the-loop feedback to expose blind spots in the model.
• Fixing the bug isn’t enough: trace its roots, learn from the breakdown, and harden the system for what’s next.

Before you deploy and use your model, you need to understand whether it’s actually delivering the kind of results you want. Check if the results are accurate and see if the data you’re loading into the model will keep these models consistent and relevant over time. Weak, old data can create model drift, leading to inaccurate outcomes.

In this phase, consider these elements:

• Responsible development and diagnosis: This is an important stage in building responsible AI systems. Key factors include developing proper data collection and handling practices, ensuring fairness in performance and representation, boosting transparency through validating citations, monitoring security and privacy, increasing accountability by including author contact info, and emphasizing inclusiveness by incorporating a feedback process into pre-deployment validation.
• Validate the model deployment:­ After the AI model has been trained, it should be tested to ensure that it provides accurate responses and not hallucinations. The testing should be conducted in a controlled environment, and the model’s responses should be compared to the approved documents and data.

Testing the AI model throughout the process is critical to mitigate against any problems—such as overfitting and underfitting—that could undermine your model’s effectiveness once deployed.

• Overfitting refers to an AI model that models the training data too well. This happens when a model learns the detail and noise in the training data to the extent that it negatively impacts its performance on new data—the noise or random fluctuations in the training data are picked up and erroneously adopted as concepts.
• Underfitting refers to a model that can neither model the training data nor generalize to new data. An underfit machine learning model is not a suitable model; this will be obvious, as it will have poor performance on the training data. The remedy is to move on and try alternate machine learning algorithms.

Agentic AI isn’t just defined by its ability to answer questions, but also by the ability to decide.

Once you have a stable core, focus on the agent’s reasoning layer: policies, guardrails, evaluation metrics, and AI explainability (explaining how the model works and its expected output). This is where you confirm that the agent not only works, but that it behaves responsibly under the different types of scenarios it might face.

From interpreting tone to recognizing when to escalate, AI models must make real-time choices that echo human judgment. This requires context awareness, confidence scoring, and ethical constraint modeling. The model learns to prioritize factors such as fairness, customer sentiment, and organizational policies. For example, if a customer calls with a question regarding their bill, the model, might choose to respond first with compassion rather than procedure. In doing so, it doesn’t just act on behalf of your brand, it embodies it.

Deployment is defined as the process through which you integrate your AI model into an existing production environment to obtain effective business decisions based on data. It’s one of the last steps in the framework and should be preceded by a SME and validation-team sign-off process.

Rolling out agentic AI across support channels requires orchestration: integrating with CRMs, safeguarding compliance, conducting responsible AI reviews, configuring escalation protocols, and preparing for edge cases. We begin with pilots and cohorts, gradually expanding while listening closely to real-world signals.

This phase involves integrating the AI model into the service’s decision-making process and using the live data for the AI model to make predictions. When launched, it’s very important to continuously evaluate the AI model to ensure it still meets the business objectives, and the performance is at the level required.

This ensures the AI model’s performance is in line with the modelling phase and helps you identify when to retrain the model. It can also help you feel confident in using, interpreting, and challenging any outputs or insights generated by the AI model.

After completing the typical workflow with steps like data ingestion, pre-processing, model building and evaluation, and deployment, it’s time to document.

Documentation is the connective tissue that links creators, regulators, and users. Internally, it guides engineering, legal, and operations teams through the model’s capabilities, boundaries, and known issues. Externally, it reassures customers—clarifying when AI is in use, offering transparency in how decisions are made, and explaining how they can opt out.

Think of documentation not as paperwork, but as a living record that ensures every decision made by the AI can be understood, audited, and improved upon. Close the loop with thorough public-facing and internal documentation. While documents should evolve throughout the project, finalizing them ensures they capture the solution as actually shipped, which is vital for maintainability, audits, and future work.

Other considerations

Because agentic AI models are so new, the team is aware that they are doing important, pioneering work that will hopefully spread across the industry.

“As we enter the agentic era of AI, we recognize the opportunity and importance of doing this responsibly,” says Ross Smith, a support leader at Microsoft. “We hope others can build on our lessons learned and improve their own support processes using AI tools.”

Starting to think about content support in this way led Smith to have conversations across Microsoft, exploring what other teams were doing, learning how to integrate best practices into the support solutions, and drilling down on a strategy for building responsible AI models from the start.

The team wanted to create a model that was responsible-AI ready. This required understanding its potential impacts—both beneficial and harmful—on people and society, and then taking the appropriate measures to mitigate anticipated harms and prepare responses for unanticipated ones.

Another consideration was how we measured success, not just of the model itself, but also how it impacted the business based on different use-case scenarios. Knowing that support engineers would be using the model to assist with administrative-type tasks (such as email and auto-summarization) and technical tasks (such as troubleshooting, learning, and debugging assistance) helped the team develop a set of metrics they could track to see how the model assisted the engineers, impacted their productivity, and contributed to our customer’s experience.

The application of the “8 Ds” framework can easily reach well beyond the technical support use case scenario to include a host of other company disciplines, including human resources, finance, sales, and legal.

“Humans and machines are more powerful working together than either one alone,” Smith says. “Those that really embrace and explore this new technology will be ready for the new roles that will be needed to make it successful.”

Qualified and experienced prompt engineers, strong content curators, and responsible AI experts will soon be in high demand as more companies employ AI technologies in their own operations.

What’s next

The team is looking forward to many new opportunities in the AI space, including:

• Using Copilot in Dynamics 365 Customer Service and applying what they learn directly to the product. Increased use of Copilot by all Microsoft support engineers will help us continue to improve the Copilot experience and product performance.
• Carrying on the work with Modern Work GPT to help improve the experiences our customers can have using the new Microsoft Azure AI Studio.
• Continuing to learn, refine, and build new AI models, embracing improvements in the underlying AI technology to deliver better results to support engineers; this will also facilitate excellent customer support experiences.

The world is entering a new era of human and machine collaboration. It’s an exciting time in technology, as AI helps power monumental changes in how companies serve their customers.

“In this new era of AI-powered support, agents will play a critical role not just in resolving issues, but in shaping how support evolves,” says Diego Silva, a senior technical adviser and AI support specialist at Microsoft. “Empowering them with AI fosters an agentic mindset, where they become proactive problem-solvers and process improvers. With the right insights, we’re transforming support from reactive to strategic.”