What GUI tasks will Claude be able to do automatically before end of 2024?
➕
Plus
16
Ṁ1203
Jan 1
80%
See what tabs I have open
78%
Exit vim
68%
Send an email
53%
Resolve a Manifold question
48%
Find this market after being told its title and download the AI generated banner
34%
Transfer money from one bank account to another
29%
Drag a file from one folder to another

Anthropic has just released a new interesting update.

https://www.anthropic.com/news/3-5-models-and-computer-use

For this question, I will be testing all of the created answers myself when the market closes. I'll give Claude 5 tries. It needs to get it "right" (accomplish the task) at least 4 of those times.

Due to the tricky nature of testing and possible ambiguity, I will not be betting.

Get Ṁ1,000 play money
Sort by:
Transfer money from one bank account to another

@MagnusAnderson Can both accounts be owned by me, or no?

Like a checkings account and a savings account.

@singer yes. You can use a password manager with the credentials saved for the banks' page and include an instruction for how to use the password manager (bitwarden is ctrl+shift+L for example)

See what tabs I have open

@singer Equivalent requirements of this: /singer/will-ai-correctly-see-what-tabs-i-h

Drag a file from one folder to another

@singer Specifically it needs to click and drag to accomplish this.

Find this market after being told its title and download the AI generated banner
sold Ṁ50 Find this market aft... NO

@singer Does it need to download just the banner? If it just saves all images on the page or the whole page that will include the banner without identifying the banner specifically.

@LiamZ Good point. It has to return just the specific image to me, regardless of how it downloaded it.

Resolve a Manifold question

@singer Equivalent to the resolution criteria here: /singer/will-ai-automate-guis-by-end-of-202

Find this market after being told its title and download the AI generated banner

@singer It needs to open the browser itself. It will be told there's a "Manifold market" with the title of the market, but no other details of where to find it.

Send an email

@singer The inbox will be open onscreen to start with.

Exit vim

@singer The vim window will be open onscreen to start with.

bought Ṁ5 Resolve a Manifold q... NO

@singer So to be clear it doesn’t need to do any GUI interactions right? Just hit ZZ or :wq or :q! ? Does just closing parent terminal window count (can you run in tmux to exclude this)?

@LiamZ I'll run it in tmux, good idea. Yes, the intention is that it types ZZ or another vim command to exit.

@singer I hope it writes a bash command in :! to ps | grep | sigkill itself lol.