One of the world’s largest data teams, Microsoft IDEAS uses Fabric to usher in new era of AI

In today’s era of AI, brands that don’t invest in cloud and data infrastructures designed to support robust analytics may hinder their innovation and growth. Microsoft itself took action to fortify its future growth and innovation.

Microsoft formed IDEAS (Insights, Data, Engineering, Analytics, Systems) to build a comprehensive data analytics platform to reduce data barriers and accelerate productivity, transitioning its data to Fabric.

The IDEAS and Fabric analytics platform solution projects a 50% efficiency boost from consolidating assets in OneLake, enhanced productivity using modern analytics tools and Direct Lake in Power BI, and AI-assisted coding via IDEAS Copilot.

To be competitive in today’s era of AI, established brands are feeling pressure to prioritize investments in foundational cloud—along with data and analytics infrastructures—to support fast, secure generative AI adoption. Without a durable analytics platform, the ability for companies to innovate quickly to stay competitive is becoming more challenging, especially in today’s changing consumer mindset environment. Even Microsoft, a global leader in AI-powered platforms and tools, discovered six years ago that inconsistent and inaccessible data issues were creating barriers to its own product and service growth goals.

Seeing the risk, Microsoft took action. It formed IDEAS (Insights, Data, Engineering, Analytics, Systems), an internal Microsoft engineering organization to build, maintain, and deliver a comprehensive data analytics platform. The platform encompasses 27,000 data sources; 420 petabytes of data volume; 35,000 pipelines; 38,000 datasets; and supports more than 600 separate Microsoft teams today.

An ocean of data: The challenge and the solution

“Microsoft IDEAS was formed as an executive imperative to be the single source of truth, a sort of ‘United Nations’ for data, capable of reducing prohibitive data barriers to accelerate team productivity and AI adoption strategies for us and our customers,” says Ravi Vedula, Corporate Vice President of Engineering and leader of the IDEAS Team at Microsoft.

Microsoft IDEAS was formed as an executive imperative to be the single source of truth, a sort of ‘United Nations’ for data, capable of reducing prohibitive data barriers to accelerate team productivity and AI adoption strategies for us and our customers.

Ravi Vedula, Corporate VP of Engineering, Leader of IDEAS Team, Microsoft

“We are more than a business intelligence team,” says Gerardo Bodegas Martinez, Partner Director of Engineering at Microsoft. “Our goal is to make data into an essential service to power multiple experiences and business functions from one investment.”

Building a durable platform for the era of AI

“When we embarked on the IDEAS team journey, we quickly realized that becoming the single source of truth requires an immense amount of focused effort and continuous data monitoring in key areas: building trust, treating data as a product and service while maintaining quality, and developing a center of data gravity to build a durable data platform,” says Vedula.

The IDEAS team chose to transition its data from Azure Synapse to Microsoft Fabric, including adoption of IDEAS Copilot. This approach to data was strategic: it enabled preparing for a cookieless future, faster email platform management, and the powerful harnessing of AI and language models to generate tailored insights and recommendations.

IDEAS began with individual Microsoft Office products and quickly grew to encompass Microsoft 365 and Security. Today, IDEAS operates across Microsoft’s largest businesses. "Intelligence now flows to the Microsoft 365 apps people use daily, with IDEAS Copilot and other AI-powered assistants boosting productivity whether creating dataflows, writing SQL statements, or building reports,” says Bodegas Martinez.

Intelligence now flows to the Microsoft 365 apps people use daily, with IDEAS Copilot and other AI-powered assistants boosting productivity whether creating dataflows, writing SQL statements, or building reports.

Gerardo Bodegas Martinez, Partner Director of Engineering, Microsoft

Activating data for AI innovation

“One of the benefits of Fabric is that it comes from a company leading the AI era,” says Bodegas Martinez. “Fabric interoperates seamlessly with other Microsoft apps and tools, such as Office apps and Azure AI, facilitating the creation and fine-tuning of custom AI models and copilots.”

For example, Bodegas Martinez’s team is developing custom extensions within the Microsoft 365 Copilot agent, with the help of Azure services. The IDEAS Copilot agent is a custom extension that helps team members and users of the platform discover the right data seamlessly. It also interoperates with Copilot in Fabric to assist users as they dig deeper into the data. “IDEAS Copilot, along with our custom AI models, enhances productivity by making these technologies first-class citizens in the platform, eliminating the need to jump through hoops for integration,” notes Bodegas Martinez.

Empowering analytics through a single toolchain

By integrating a single toolchain for all roles, from engineers to data scientists, IDEAS is empowering everyone and leaving no one behind—an intentional mission with IDEAS. This advantage allows individuals with varying skill sets to work within the same framework, enhancing collaboration and efficiency. The inclusion of AI in the workflow further supports users in learning and using new tools and languages.

“By centralizing data into OneLake in Microsoft Fabric, it makes data visible and accessible in a consistent format for everyone in the organization,” says James Clark, Principal Machine Learning Science Manager at Microsoft. “This visibility helps us realize the wealth of data we have at our fingertips to drive further investment and innovation.”

Increasing collaboration and flexibility

Also driving innovation is the ability for users to collaborate across different data personas using the same dataset. With Fabric, users can choose the tools that work best for them. This flexibility is particularly important for data engineering, because it simplifies the process of working with data in various formats and locations.

“Managing tools for diverse teams with different skill sets can be a significant challenge for companies,” explains Clark. “Fabric offers a unified tool set that caters to various roles, from data engineers to business analysts and data scientists.” As an example, Clark explains that his team of 600 people includes 100 data scientists with varying skill levels. With Fabric, he can now provide the right tools for everyone without the need for multiple configurations. Non-coders can build semantic models or cubes easily and share them with business partners. This flexibility and ease of use makes Fabric an attractive solution for business decision makers looking to streamline their data operations and enhance user productivity.

Managing tools for diverse teams with different skill sets can be a significant challenge for companies. Fabric offers a unified tool set that caters to various roles, from data engineers to business analysts and data scientists.

James Clark, Principal Machine Learning Science Manager, Microsoft

Reducing cost and risk

Fabric uses a single, organized data lake with separate compute and storage, which significantly reduces the need to move data around. This not only cuts down on engineering and human resource costs but also minimizes compliance risks. By maintaining one copy of data in one location, teams can efficiently use it for multiple purposes without the added complexity and riskof data movement.

“Data scientists are experiencing a revelation with Fabric,” says Clark. “The cumbersome days of managing massive telemetry data and creating multiple copies are gone.” He adds, “Now, with reduced data exposure and enhanced compliance, we’ve significantly mitigated risk. It’s a win-win situation—improved performance, seamless integration of diverse data types, and fortified security.”

Gaining a substantial return on investment

The ability to drive substantial data efficiencies on Fabric is broadening horizons. The IDEAS team estimates a 50% efficiency boost from consolidating assets in Microsoft OneLake and increasing the productivity of analytics experts using modern tools such as Spark and Python. Data scientists no longer need to host or maintain cubes, thanks to Direct Lake in Power BI, and can now unleash AI-assisted coding through IDEAS Copilot.

The IDEAS team effort is set to culminate midyear 2025. Microsoft is already documenting significant boosts in its AI innovation, productivity efficiencies, and early Microsoft business wins—advantages it is passing on to its partners and customers.

Currently, the IDEAS team is creating bigger and more datasets that will be available through Fabric for data scientists and data analysts. The team is first ensuring Fabric SQL Endpoint and Fabric Spark have met performance criteria for interactive querying against those larger data tables. IDEAS is also rolling out more Microsoft Power BI reports to be used by Microsoft partners once Direct Lake mode meets query performance criteria.

As Vedula sums up, “As customer zero, we knew Microsoft IDEAS becoming a center of data gravity was a massive undertaking and investment—an all-in bet on a durable data platform that sets us, and our customers, up for incalculable innovation and accelerated business growth in the era of AI.”

As customer zero, we knew Microsoft IDEAS becoming a center of data gravity was a massive undertaking and investment—an all-in bet on a durable data platform that sets us, and our customers, up for incalculable innovation and accelerated business growth in the era of AI.

Ravi Vedula, Corporate VP of Engineering, Leader of IDEAS Team, Microsoft