Anthropic’s Claude AI Can Now Control Your Computer Without Any Help

Anthropic has updated its Claude 3.5 Sonnet AI model and along with that, it has brought the ability to carry out tasks on the computer for you. The new feature, called a very obvious ‘computer use’, is designed to mimic human actions on a computer like clicking, typing, and moving the cursor. Anthropic Claude AI’s computer use is available on the API for developers, starting today in beta.

In the official release post for the feature, Anthropic gives us a good look at how the feature works. We see Sam Ringer, an Anthropic Researcher, testing out the tool. He prompts Claude to fill out a request form by extracting information from the spreadsheet and online portal that it’s seeing on the screen.

We then see the tool get to work by screenshotting the information on the screen. Additionally, we see the tool search from an online portal (as directed) to look for information that was unavailable on the spreadsheet. Side-by-side, Claude opened up the form and started filling it up.

Anthropic notes that Claude 3.5 Sonnet scored 14.9% in the screenshot-only category which is “notably better than the next-best AI system’s score of 7.8%.” Additionally, Anthropic states that, “when afforded more steps to complete the task, Claude scored 22.0%.”

Developers can further build on the computer use feature using Anthropic API, Vertex AI as well as Amazon Bedrock. Anthropic also notes that the updated Claude 3.5 Sonnet shows a range of improvements on industry-leading benchmarks.

Additionally, Anthropic has also released a new Claude 3.5 Haiku smaller model which “improves across every skill set and surpasses even Claude 3 Opus.” Claude 3.5 Haiku will be made available on the API later this month.

Teaching general computer skills to a computer certainly sounds interesting, and something I’m looking forward to. I mean, it has the potential to save a lot of time by automating tasks. What do you think about Claude AI’s computer use feature? Drop your thoughts in the comments down below!