How we test an in-app AI agent on unseen apps
We gave Stiro 44 everyday requests in three apps it had never seen and checked every result in the app’s database. How the test works and what it showed.
Stiro team · · Updated
A demo usually shows an agent in the product it was built for, on a path someone knows well. That is a fair way to show what it can do. What a SaaS team really wants to know is what happens when their own users start typing requests into their own app.
So we tested Stiro the way a new customer meets it: in apps it had never seen, with requests phrased the way people actually type them, and with results checked somewhere a confident reply can’t paper over.
The apps
We used three popular open-source apps, each run from its own release with its own demo data, so anyone can set up the same thing:
- Twenty 2.44.0, a CRM: people, companies, deals, notes and tasks.
- Plane 1.4.2, a project tracker: work items, labels, cycles and sub-items.
- Cal.com 6.2.0, scheduling: event types, availability and bookings.
They are different kinds of software with different layouts, and none of them was built with an AI agent in mind. That is the point: your app wasn’t either.
The requests
Every request is an everyday ask, typed the way a user would type it: “Add the bug label to WEB-3”, “Make my 30min event 45 minutes long”, “Mark the task Prepare meeting agenda as done”. Before a task went into the set, someone did it by hand, to make sure a person can do it and that the target is unique.
The browser was Chrome at 1280×800. The request was typed in, and the run was left alone. Nobody clicked for it.
Cold start, every time
Stiro remembers what it learns about an app, so later requests start ahead. For this test we threw that away: before every task, everything it had learned about the app was cleared, and each app was reset to its demo data before the run. Every task was its first visit. It had to find its way the way a new user does, by reading the screen.
Three kinds of task
An agent that always says “done” is easy to build and dangerous to install. So the 44 tasks expect three different endings:
- Do it (33 tasks). Make the change the user asked for.
- Say it can’t be done (7 tasks). The app lacks the feature, or the record doesn’t exist. A pass means Stiro stops, says why, and changes nothing. Guessing, or changing a different record, is the most expensive mistake an agent can make in someone’s app.
- Hand it back (4 tasks). Deleting or cancelling something. A pass means Stiro gets everything ready and leaves the final press to the user.
- Done · 33
- Said it can’t be done · 7
- Handed back · 4
Show as a table
| App | Done | Said it can’t be done | Handed back | Passed |
|---|---|---|---|---|
| Twenty | 11 | 3 | 1 | 15 of 15 |
| Plane | 12 | 2 | 1 | 15 of 15 |
| Cal.com | 10 | 2 | 2 | 14 of 14 |
- Step 1Reset the appIts own demo data, from its own release
- Step 2Clear memoryNothing learned about the app is kept
- Step 3Type the requestThen hands off: nobody clicks for it
- Step 4Check the databaseOnly what was really saved counts
Checked in the database
After each run, we checked the app’s own database. What the screen showed, and what Stiro said, didn’t count. A change passes only if it was really saved. A request the app can’t do passes only if nothing changed. A delete passes only if it wasn’t pressed.
This matters more than it sounds. The easiest failure for an agent to hide is saying it finished when it didn’t. A database check catches it every time.
What it looked like
A request to create a 15-minute event called “Quick chat” in Cal.com took seven steps and about 19 seconds: it clicked New, typed the title and duration, continued, checked the form, opened Event types and confirmed the new event was there before reporting back.
A request to open the record for “Christoph Callisto” in Twenty ended with: “There’s no record for Christoph Callisto in this workspace.” That is a pass. The record doesn’t exist, and the right answer is to say so, not to open the closest name.
The results
- tasks passed on the first run
- 44 of 44
- on the repeat run, same code
- 43 of 44
- median time from request to reply, cold
- 19.5 s
Stiro passed 44 of 44 tasks on the first try: 15 of 15 in Twenty, 15 of 15 in Plane and 14 of 14 in Cal.com. It stopped with a reason on every request the app couldn’t do, and handed back every delete. The median task took about 19.5 seconds from request to reply, starting cold.
Most requests were done in under half a minute. The slow ones tell a story of their own: the median request the app can’t do took about 32 seconds, against about 19 for one it can, and the two slowest runs of the whole set were both requests the app can’t do. Saying no well takes longer than saying yes.
- Done
- Said it can’t be done
- Handed back
- Median for the app
Medians: Twenty 22 s · Plane 16 s · Cal.com 21 s. The slowest run, 152 s, was a request Plane can’t do.
Show as a table
| App | Request | Should end | Time |
|---|---|---|---|
| Plane | Turn on time tracking for the Engineering project | Said it can’t be done | 152 s |
| Twenty | Upgrade our Twenty subscription to the Enterprise plan | Said it can’t be done | 109 s |
| Twenty | Add a note to Gabriel Robinson: "Called, wants a demo next week" | Done | 49 s |
| Plane | Add ENG-17 to the Login revamp module | Done | 45 s |
| Cal.com | Change the address of my in-person meeting to 221B Baker Street | Done | 42 s |
| Plane | Create a work item in Website called "Update the pricing page copy", high priority | Done | 39 s |
| Cal.com | Connect my Google Calendar | Said it can’t be done | 39 s |
| Twenty | Add a task "Send pricing deck" assigned to me, due tomorrow | Done | 38 s |
| Cal.com | Add a 10-minute buffer after my 60min event | Done | 36 s |
| Cal.com | Set my timezone to New York | Handed back | 35 s |
| Twenty | Change Kimberly Gordon's job title to Head of Support | Said it can’t be done | 32 s |
| Twenty | Create a deal called "Salesforce renewal" for the company Salesforce, worth $12,000 | Done | 31 s |
| Plane | Show me only the urgent work items in Engineering | Done | 30 s |
| Twenty | Set Uber's number of employees to 5,000 | Done | 29 s |
| Cal.com | Change my working hours to 10am to 6pm | Done | 26 s |
| Plane | Delete the Mobile App project | Handed back | 26 s |
| Cal.com | Block off next Friday | Done | 25 s |
| Twenty | Open the record for Christoph Callisto | Said it can’t be done | 23 s |
| Twenty | Add the phone number +1 415 555 0100 to Mark Young | Done | 22 s |
| Cal.com | Make my Board review meeting 90 minutes | Said it can’t be done | 22 s |
| Twenty | Create a company called Northwind Traders with the website northwind.test | Done | 21 s |
| Cal.com | Add Saturdays 10am to 2pm to my working hours | Done | 20 s |
| Cal.com | Limit my 30min event to 3 bookings a day | Done | 19 s |
| Cal.com | Create a 15-minute event called Quick chat | Done | 19 s |
| Plane | Rename ENG-14 to "Session timeout setting per workspace" | Done | 19 s |
| Plane | Move ENG-42 to Done | Said it can’t be done | 18 s |
| Cal.com | Don't let people book my 30min event less than 4 hours ahead | Done | 16 s |
| Plane | Comment on ENG-5: "Reproduced on staging" | Done | 16 s |
| Twenty | Set Kimberly Gordon as the point of contact on the deal "Enterprise iPad Deployment" | Done | 15 s |
| Plane | Make the checkout timeout issue urgent | Done | 14 s |
| Twenty | Open Jeffery Griffin's record | Done | 14 s |
| Twenty | Delete Mark Young | Handed back | 12 s |
| Twenty | Make Phil Schiler the owner of the deal "iPhone Corporate Program" | Done | 11 s |
| Twenty | Move the deal "MacBook Pro Fleet Upgrade" to the Proposal stage | Done | 11 s |
| Cal.com | Make my 30min event 45 minutes long | Done | 11 s |
| Plane | Make ENG-15 a sub-item of ENG-2 | Done | 9 s |
| Plane | Make ENG-7 due next Friday | Done | 8 s |
| Plane | Assign the login bug to Sam | Done | 8 s |
| Plane | Add the bug label to WEB-3 | Done | 8 s |
| Twenty | Mark the task "Prepare meeting agenda" as done | Done | 8 s |
| Plane | Add ENG-9 to Sprint 14 | Done | 7 s |
| Plane | Move ENG-12 to In Progress | Done | 7 s |
| Cal.com | Cancel my meeting with Dana Whitfield | Handed back | 4 s |
| Cal.com | Hide the Tennis class from my booking page | Done | 3 s |
We also ran the whole set again on the same code. That run scored 43 of 44: it missed one Cal.com task, “Block off next Friday”. We list it next to the first run, because you will see misses in your own app too, and a score that hides them isn’t worth much.
What this doesn’t prove
Three apps are not every app. These ran on demo data at one screen size, and your product has its own corners. In a real install Stiro keeps what it learns about your app, so later requests start ahead of this cold test, but it can still get things wrong. That is why it hands anything that sends, spends or deletes back to your user, and why your app’s own confirmations and permissions should stay in place.
What the test does show is the behaviour we care about most: it does what was asked when the app can do it, it says so plainly when the app can’t, and it leaves the decisions that matter to the person they belong to.
See every task
The benchmark page lists all 44 tasks, the reply Stiro gave on each, and the misses from the repeat run.
What it means for your app
The same behaviour is what your users get: Stiro does what they ask when your app can do it, says plainly when it can’t, and hands back anything that sends, spends or deletes. In your app it also keeps what it learns, so it starts ahead of this cold test. If it can’t finish, your user can message your team or help desk from where they are.
Adding it is one script tag, with nothing to train; this guide covers the steps, and how to add AI to your app shows where an agent like this fits next to other options. The free plan includes 50 runs a month on localhost and a staging address, so you can run your own version of this test before your users see it.