- Microsoft has updated its Copilot Vision tool to let the AI assistant see your desktop
- The feature lets copilot respond to any content you see
- To give users control, Copilot Vision must be enabled so it will only look at what you want AI to see
Microsoft has opened his eyes on his Copilot AI on desktops to see the entire screen. The feature that allows copilot to look at and analyze things on your screen can see any window or browser you want, as a feature Microsoft calls “Desktop Share.”
When Copilot Vision was first launched, it was limited to just seeing what happened at the edge. You would ask a question about a web page, it would scan that page and AI would help explain what was there. With this update, it can now zoom out and see your entire desk or any specifically the app window you choose to share
You can choose what the AI Assistant is looking at by clicking the glass icon in the Copilot app. Then you choose the area you want AI to look at and you can discuss what’s on screen with AI without having to describe what you are looking for infinitely.
Let’s say you revise a resume and want it to sound more like a confident human and less like a panic -degree student. You can open the word document, share the copilot window and ask it to highlight specific types of experiences. Copilot sees the document itself and suggests edits.
The same goes for PowerPoint, Photoshop and even some games. You can ask Copilot for help with the Settings menu in a game you’ve never played before, and it will guide you through options with tips and suggestions for when you’re stuck.
Visionary planning
In the event that you are concerned about Copilot snooping on your secret romance novel or anything else you’d rather see, you can always turn it off. Currently, the update is only available for Windows Insiders that already have Copilot Vision. It will probably come to the General Windows 11 population soon.
It may not appear to be too big a deal compared to other generative AI features, but it is a subtle way for Microsoft to integrate its AI in quite literally everything on your computer. Instead of writing an e email in a tab and had to describe it to AI in another tab for help, it’s okay there. Copilot vision makes the process more intuitive. You point digitally to something you want to help with, ask a basic question and get an answer that makes sense in context.
It is difficult to exaggerate how much friction this can eliminate. Of course, it’s not perfect. There is a need for some specificity as AI cannot read your mind. But an extra set of eyes to help you over the humpkelet designing a PowerPoint or shortening a meeting is nothing.
The dream of AI as a personal assistant has always been about context, the idea of an AI that could know what we mean, not only what we are saying. Copilot Vision is Microsoft’s clearest step in that direction yet. Copilot Vision could calmly become one of the most useful AI tools that Microsoft has ever made. And unlike Clippy, it will only set aside judgment if you ask.
