Over a decade since Microsoft tried to make talking to Cortana on PCs a thing — and spectacularly failed in the process — the Windows giant is taking another swing at voice commands with its Copilot AI assistant in Windows 11. Starting today, the company is rolling out an upgrade to its existing Copilot Voice and Vision features which will let you say “Hey, Copilot” and then ask your PC a question based on what’s on the screen. If you’re looking at pictures of Hawaii, for example, you could ask your Windows 11 PC where exactly they were taken, have it plot you a flight plan and potentially even give you some budgeting tips to afford that island vacation.
Microsoft’s jaunty promotional videos for the Copilot features, set to Vampire Weekend’s almost two-decade-old “A Punk,” make the process look practically seamless. One user asks Copilot to show them how to stream their music in the “best possible quality,” and the AI proceeds to highlight the exact location of the streaming settings in Spotify, while suggesting they choose the lossless option. Another person asks Copilot to write up a short biography based on their photo portfolio. Now Copilot isn’t just about searching the web or generating novelty AI art, it’s making it easy for users to perform practical tasks without much effort.
Microsoft is clearly striving for the convenience of the Star Trek ship computer, a dream that also pushed Amazon to invest billions in its Echo devices and Alexa. The difference with Copilot is that you’re not just talking to a faceless speaker — Microsoft is also trying to make Windows 11 aware of what you’re doing on your screen. The “Hey Copilot” feature and all of the Copilot Vision are cloud-based, so you’ll have to live with image data of your desktop making its way to Microsoft’s servers. That involves a level of trust the company has lost with many users, especially after the messy debut of Recall, its first flagship AI-powered feature.
It doesn’t help that many people are still peeved about the death of Windows 10 support this week. Unsurprisingly, the company stresses that “Hey, Copilot” is a purely opt-in feature that’s buried in the Copilot app settings. (Of course, that can always change, especially if the company wants to juice AI engagement stats in a few years.)
Copilot Actions
(Microsoft)
I suspect it’ll be even harder for users to swallow where Microsoft wants to take Copilot: Giving it the ability to perform Windows tasks on its own. That’s the goal of the experimental Copilot Actions feature, which initially debuted as a tool that could perform tasks on websites. Once enabled, Copilot Actions can be prompted to handle manual tasks, like resizing and straightening an a folder of photos. If any questions pop up, it can prompt you to answer them within the Copilot app. And as Copilot Actions is handling its job in the background, you’re free to do anything else you’d like on your computer.
Conceptually, Copilot Actions sounds similar to handing off a task to a real life assistant — but just like a human assistant, there’s always a chance something could go wrong along the way. It’s also not hard to imagine the feature being coopted by nefarious malware down the line, since it’s basically a Windows script in a better interface. Microsoft says it’s tested Copilot Actions “extensively” internally, and it’s rolling out the feature slowly to gather feedback.
Just like “Hey, Copilot,” it’s entirely opt-in, and you can see everything Copilot Actions is doing step-by-step in the Copilot app. Microsoft says you’l be able to jump in and take control of a Copilot Action job at any point, as well as control the permissions of AI agents in Windows 11’s user settings. Copilot tasks are also performed in a contained environment, according to Microsoft, which allows for even more specific permissions controls as well as runtime isolation (so Copilot can’t affect the rest of your system beyond its specific task).
And as if we’re not already inundated with Copilot all over Windows 11 already, Microsoft also plans to add an “Ask Copilot” search function right on Windows 11’s taskbar. The company claims it’s part of a mission to make the taskbar “a dynamic hub” for accomplishing tasks, but personally I like to keep my taskbar clear so I can cram in more app windows. Like everything Microsoft is announcing today, the Ask Copilot bar will also be entirely opt-in.
As someone who’s been skeptical of Microsoft’s Copilot initiatives so far, I could actually see myself using “Hey Copilot” if it works as advertised. It sounds far more practical than the old Siri voice commands, which were limited by simplistic language models from a decade ago. Microsoft is also expanding AI actions built into Windows 11, including a new integration with Manus, an AI agent that can do things like turn several documents into a website, as well as Filmora, which lets you create AI videos right from the File Explorer.
The new “Hey Copilot” and Copilot Vision features are available today on all Windows 11 PCs that have access to Copilot. Microsoft is also making Copilot Vision broadly available around the world today where Copilot is available. Copilot Actions and the Ask Copilot taskbar feature will “gradually” become available to Windows 11 Insiders, according to Microsoft.
Credit: Source link