Microsoft combines Copilot AI with mixed reality for industrial workers

Microsoft plans to integrate its Copilot generative AI (genAI) assistant into a mixed-reality app that can provide guidance to factory technicians, aiming to reduce the time needed to manage and maintain industrial equipment.

At its Ignite conference Wednesday, Microsoft announced that Copilot will be integrated into Dynamics 365 Guides. Guides, launched in preview in 2019, is a mixed-reality app for HoloLens and mobile devices that provides step-by-step instructions to field workers via graphics overlaid onto physical objects, such as the tools and machine parts workers use.

The Copilot integration is designed to provide field workers with hands-free access to information that can help them in their daily tasks.

“Copilot and Microsoft Dynamics 365 Guides combines the power of generative AI with mixed reality to help frontline workers complete complex tasks and resolve issues faster, with less disruption to the flow of work,” Frank Shaw, Microsoft’s chief communications officer, said in a pre-recorded video ahead of the Ignite event.

“Frontline workers often struggle to quickly diagnose asset/equipment issues and navigate a series of work instructions to resolve the issue,” said Aly Pinder, research vice president at IDC. “Being able to leverage personalized insights from genAI can ensure more frontline workers will be experts at the point of service need.”

By using Guides, workers will be able to interact with the Copilot using natural language and physical gestures — specifically by looking and pointing at physical objects. For example, rather than relying on an experienced colleague, a new employee could simply point to a component and ask what it does or how it functions. The Copilot will then search files to provide relevant information, combing technical documentation, service records, and any other data available. Finally, the Copilot then summarizes and presents the information via mixed-reality “holograms” overlaid on the physical equipment.

The addition of Copilot will enable more conversational interactions. Guides has typically relied on pre-prepared instructions to support field workers, but the Copilot’s large language models (LLMs) open the door for wider ranging interactions with the genAI assistant.  

“Workers can have progressive conversations such as, ‘Walk me through the steps to disassemble this filtration unit,’ or ‘I disconnected the motor, what’s next?’” Microsoft said in a blog post. “Copilot follows along with the process and conversation using customer’s curated knowledge base to articulate answers with speech, text, or 3D holograms, eliminating ambiguity, and reducing the need for rework.”

Microsoft also said Copilot will assist in the creation of spatial models and 3D content — often one of the challenges with delivering enterprise mixed-reality projects.

“Copilot simplifies the creation, management and delivery of mixed-reality content using customer-defined data sources, enabling companies to adopt mixed reality much faster,” Microsoft said in a statement announcing the feature.

The addition of generative AI will help retain a company’s knowledge base as senior staff retire, said Pinder. “With a rapidly aging and retiring workforce, the need is growing for tools that can ensure less-tenured frontline workers will have the (on-demand) knowledge to resolve issues enabling organizations to scale to meet growing service demand,” he said.

Copilot in Dynamics 365 Guides is currently in private preview. The feature will be available first on Microsoft’s HoloLens 2 headset, with access on mobile devices “at a later date.”