Microsoft built three fake companies to test Copilot, and quality scores rose 39%

Microsoft has built fake companies, complete with staff, emails and meetings, to test whether Copilot does its job. The company describes the method in a post on its research blog.

The problem

Copilot helps people draft documents, summarize meetings and find information. To check it properly, testers need data that looks like a real organization. But real company data is full of private details and cannot be used for testing.

The fix

Microsoft's answer is a system it calls the Industry Evaluation Data Generation Platform. It creates whole simulated workplaces, called tenants. A tenant is an isolated company environment with its own users, data and settings.

Three were built:

• A pharmaceutical company, with regulatory submissions and clinical trial talks.
• A capital markets firm, with deal pipelines and compliance messages.
• An IT and software company, with sprint planning and product launches.

Each has hundreds of users and full communication histories.

Where the realism comes from

The fake companies are modeled on public sources. O*NET, a US Department of Labor database of jobs, shaped roles and daily tasks. Online job postings supplied titles and reporting lines. SEC filings guided company size and structure. Microsoft says its legal and policy teams approved all of these sources.

The companies also change over time. Email threads grow over weeks and teams reorganize. That matters because people ask questions like "What changed since the last quarterly review?"

Questions on autopilot

A six-stage automated pipeline writes the test questions. It starts from common workplace tasks, adapts them to an industry, grounds them in the fake company's data, rewrites them to sound human, and checks each for accuracy and uniqueness.

The numbers

Microsoft reports a 39% gain in its composite quality score against its first approach. Some environments scored above 0.9 out of 1.0. It estimates savings of about $100,000 per dataset and $300,000 for the project. Running the same configuration again gives identical results. The datasets are now the first standardized evaluation baseline for Copilot.

Imagine you test a meeting summarizer: you want the same fair exam every release. That is what this gives.

Limits

The figures are Microsoft's own, measured with its internal scoring system. The source shows no Copilot results from these tests, and only three industries are covered so far.