agentfootprint
How do we humans solve a problem?
How do you solve a problem?
User
You
You · with a question: why did sales drop in March?
Tools
Your tools · a few books to find out
Answer
“Sales dropped because a campaign ended.”
fact, or hunch? which book?
No notes · you read, you answer… but where did it come from? It’s a guess.
Ledger
1 · Sales report
why: I’m sure it has the answer
A ledger · which book you opened, and why: you’re sure it has the answer
2 · Marketing plan
why: I assume it might
Not always sure · sometimes you open a book on a hunch
✓ has it
✗ not in it
What you found · it had the answer, or it didn’t
Your answer
FactSales fell 20% in the North
from: Sales report ✓
HypothesisA campaign ended there
not sure: no book says so
Your answer · each part with its reason: facts from the ledger, hypotheses marked
Trust
Honest · anyone can follow it and check it. That’s what makes us trust it.
Now, an LLM · its tools are its books. Where is its ledger?
agentfootprint
AgentFootprint · our open framework: it builds the ledger, so the LLM answers from it, not a guess
How do you solve a problem?
User
You
You · with a question: why did sales drop in March?
Tools
Your tools · a few books to find out
Answer
“Sales dropped because a campaign ended.”
fact, or hunch? which book?
No notes · you read, you answer… but where did it come from? It’s a guess.
Ledger
1 · Sales report
why: I’m sure it has the answer
A ledger · which book you opened, and why: you’re sure it has the answer
2 · Marketing plan
why: I assume it might
Not always sure · sometimes you open a book on a hunch
✓ has it
✗ not in it
What you found · it had the answer, or it didn’t
Your answer
FactSales fell 20% in the North
from: Sales report ✓
HypothesisA campaign ended there
not sure: no book says so
Your answer · each part with its reason: facts from the ledger, hypotheses marked
Trust
Honest · anyone can follow it and check it. That’s what makes us trust it.
Now, an LLM · its tools are its books. Where is its ledger?
agentfootprint
AgentFootprint · our open framework: it builds the ledger, so the LLM answers from it, not a guess
How do you solve a problem?
User
You
You · with a question: why did sales drop in March?
Tools
Your tools · a few books to find out
Answer
“Sales dropped because a campaign ended.”
fact, or hunch? which book?
No notes · you read, you answer… but where did it come from? It’s a guess.
Ledger
1 · Sales report
why: I’m sure it has the answer
A ledger · which book you opened, and why: you’re sure it has the answer
2 · Marketing plan
why: I assume it might
Not always sure · sometimes you open a book on a hunch
✓ has it
✗ not in it
What you found · it had the answer, or it didn’t
Your answer
FactSales fell 20% in the North
from: Sales report ✓
HypothesisA campaign ended there
not sure: no book says so
Your answer · each part with its reason: facts from the ledger, hypotheses marked
Trust
Honest · anyone can follow it and check it. That’s what makes us trust it.
Now, an LLM · its tools are its books. Where is its ledger?
agentfootprint
AgentFootprint · our open framework: it builds the ledger, so the LLM answers from it, not a guess
How do you solve a problem?
User
You
You · with a question: why did sales drop in March?
Tools
Your tools · a few books to find out
Answer
“Sales dropped because a campaign ended.”
fact, or hunch? which book?
No notes · you read, you answer… but where did it come from? It’s a guess.
Ledger
1 · Sales report
why: I’m sure it has the answer
A ledger · which book you opened, and why: you’re sure it has the answer
2 · Marketing plan
why: I assume it might
Not always sure · sometimes you open a book on a hunch
✓ has it
✗ not in it
What you found · it had the answer, or it didn’t
Your answer
FactSales fell 20% in the North
from: Sales report ✓
HypothesisA campaign ended there
not sure: no book says so
Your answer · each part with its reason: facts from the ledger, hypotheses marked
Trust
Honest · anyone can follow it and check it. That’s what makes us trust it.
Now, an LLM · its tools are its books. Where is its ledger?
agentfootprint
AgentFootprint · our open framework: it builds the ledger, so the LLM answers from it, not a guess
How do you solve a problem?
User
You
You · with a question: why did sales drop in March?
Tools
Your tools · a few books to find out
Answer
“Sales dropped because a campaign ended.”
fact, or hunch? which book?
No notes · you read, you answer… but where did it come from? It’s a guess.
Ledger
1 · Sales report
why: I’m sure it has the answer
A ledger · which book you opened, and why: you’re sure it has the answer
2 · Marketing plan
why: I assume it might
Not always sure · sometimes you open a book on a hunch
✓ has it
✗ not in it
What you found · it had the answer, or it didn’t
Your answer
FactSales fell 20% in the North
from: Sales report ✓
HypothesisA campaign ended there
not sure: no book says so
Your answer · each part with its reason: facts from the ledger, hypotheses marked
Trust
Honest · anyone can follow it and check it. That’s what makes us trust it.
Now, an LLM · its tools are its books. Where is its ledger?
agentfootprint
AgentFootprint · our open framework: it builds the ledger, so the LLM answers from it, not a guess
How do you solve a problem?
User
You
You · with a question: why did sales drop in March?
Tools
Your tools · a few books to find out
Answer
“Sales dropped because a campaign ended.”
fact, or hunch? which book?
No notes · you read, you answer… but where did it come from? It’s a guess.
Ledger
1 · Sales report
why: I’m sure it has the answer
A ledger · which book you opened, and why: you’re sure it has the answer
2 · Marketing plan
why: I assume it might
Not always sure · sometimes you open a book on a hunch
✓ has it
✗ not in it
What you found · it had the answer, or it didn’t
Your answer
FactSales fell 20% in the North
from: Sales report ✓
HypothesisA campaign ended there
not sure: no book says so
Your answer · each part with its reason: facts from the ledger, hypotheses marked
Trust
Honest · anyone can follow it and check it. That’s what makes us trust it.
Now, an LLM · its tools are its books. Where is its ledger?
agentfootprint
AgentFootprint · our open framework: it builds the ledger, so the LLM answers from it, not a guess
How do you solve a problem?
User
You
You · with a question: why did sales drop in March?
Tools
Your tools · a few books to find out
Answer
“Sales dropped because a campaign ended.”
fact, or hunch? which book?
No notes · you read, you answer… but where did it come from? It’s a guess.
Ledger
1 · Sales report
why: I’m sure it has the answer
A ledger · which book you opened, and why: you’re sure it has the answer
2 · Marketing plan
why: I assume it might
Not always sure · sometimes you open a book on a hunch
✓ has it
✗ not in it
What you found · it had the answer, or it didn’t
Your answer
FactSales fell 20% in the North
from: Sales report ✓
HypothesisA campaign ended there
not sure: no book says so
Your answer · each part with its reason: facts from the ledger, hypotheses marked
Trust
Honest · anyone can follow it and check it. That’s what makes us trust it.
Now, an LLM · its tools are its books. Where is its ledger?
agentfootprint
AgentFootprint · our open framework: it builds the ledger, so the LLM answers from it, not a guess
How do you solve a problem?
User
You
You · with a question: why did sales drop in March?
Tools
Your tools · a few books to find out
Answer
“Sales dropped because a campaign ended.”
fact, or hunch? which book?
No notes · you read, you answer… but where did it come from? It’s a guess.
Ledger
1 · Sales report
why: I’m sure it has the answer
A ledger · which book you opened, and why: you’re sure it has the answer
2 · Marketing plan
why: I assume it might
Not always sure · sometimes you open a book on a hunch
✓ has it
✗ not in it
What you found · it had the answer, or it didn’t
Your answer
FactSales fell 20% in the North
from: Sales report ✓
HypothesisA campaign ended there
not sure: no book says so
Your answer · each part with its reason: facts from the ledger, hypotheses marked
Trust
Honest · anyone can follow it and check it. That’s what makes us trust it.
Now, an LLM · its tools are its books. Where is its ledger?
agentfootprint
AgentFootprint · our open framework: it builds the ledger, so the LLM answers from it, not a guess
How do you solve a problem?
User
You
You · with a question: why did sales drop in March?
Tools
Your tools · a few books to find out
Answer
“Sales dropped because a campaign ended.”
fact, or hunch? which book?
No notes · you read, you answer… but where did it come from? It’s a guess.
Ledger
1 · Sales report
why: I’m sure it has the answer
A ledger · which book you opened, and why: you’re sure it has the answer
2 · Marketing plan
why: I assume it might
Not always sure · sometimes you open a book on a hunch
✓ has it
✗ not in it
What you found · it had the answer, or it didn’t
Your answer
FactSales fell 20% in the North
from: Sales report ✓
HypothesisA campaign ended there
not sure: no book says so
Your answer · each part with its reason: facts from the ledger, hypotheses marked
Trust
Honest · anyone can follow it and check it. That’s what makes us trust it.
Now, an LLM · its tools are its books. Where is its ledger?
agentfootprint
AgentFootprint · our open framework: it builds the ledger, so the LLM answers from it, not a guess
How do you solve a problem?
User
You
You · with a question: why did sales drop in March?
Tools
Your tools · a few books to find out
Answer
“Sales dropped because a campaign ended.”
fact, or hunch? which book?
No notes · you read, you answer… but where did it come from? It’s a guess.
Ledger
1 · Sales report
why: I’m sure it has the answer
A ledger · which book you opened, and why: you’re sure it has the answer
2 · Marketing plan
why: I assume it might
Not always sure · sometimes you open a book on a hunch
✓ has it
✗ not in it
What you found · it had the answer, or it didn’t
Your answer
FactSales fell 20% in the North
from: Sales report ✓
HypothesisA campaign ended there
not sure: no book says so
Your answer · each part with its reason: facts from the ledger, hypotheses marked
Trust
Honest · anyone can follow it and check it. That’s what makes us trust it.
Now, an LLM · its tools are its books. Where is its ledger?
agentfootprint
AgentFootprint · our open framework: it builds the ledger, so the LLM answers from it, not a guess
agentfootprint
And now, the frontend
Bring the same idea to the user’s side.
The frontend · the screens your users already use
FilterDate rangeApplyApprove
the user’s actions
filter()set_range()apply()approve()
tools for the LLM
User actions → tools · the same actions, exposed as tools for the LLM
HCI Footprint
This talk · HCI Footprint: how we did it, why, and how it helped our agent
agentfootprint
And now, the frontend
Bring the same idea to the user’s side.
The frontend · the screens your users already use
FilterDate rangeApplyApprove
the user’s actions
filter()set_range()apply()approve()
tools for the LLM
User actions → tools · the same actions, exposed as tools for the LLM
HCI Footprint
This talk · HCI Footprint: how we did it, why, and how it helped our agent
agentfootprint
And now, the frontend
Bring the same idea to the user’s side.
The frontend · the screens your users already use
FilterDate rangeApplyApprove
the user’s actions
filter()set_range()apply()approve()
tools for the LLM
User actions → tools · the same actions, exposed as tools for the LLM
HCI Footprint
This talk · HCI Footprint: how we did it, why, and how it helped our agent
agentfootprint
Welcome
Hi, I’m Sanjay Krishna Anbalagan.
Frontend engineer · AWS
Open source: AgentFootprint · HCI Footprint · FootPrint
Libraries · LinkedIn · writing
agentfootprint
HCI Footprint
Your app just got
a second user
— and it doesn’t need its own UI.
HCI Footprint · our library that lets an AI assistant
use your app’s existing screens, and keep a ledger of what it did
Your app
agentfootprint
01
Part 1 of 7
Two ways in
Two kinds of app, one backend.
1Two ways in2Try 1: UI in chat3Try 2: AI clicks4When is it done?5Action Binding6In practice7Two of us
agentfootprint
1 · Two ways in
It starts with data.
User
All services ▾Last 7 days ▾Slowest ▾
By service
Checkout
Payments
Cart
Step 1/3
Step 2/3
Step 3/3
Chat
Which services got slower this week, and why?
Checkout and Payments slowed down after Tuesday’s deploy. Checkout’s slowest requests went from 220 to 610 ms. Cart and Login are flat.
which function the click calls
Agent
words → what you want
Functions
Data
MCP
1
2
3
1
2
3
✗
no screen built for this question
Which services got slower this week, and why?
Update, then Notify
Checkout ▾
On the screen
1 · fill in
Checkout · Tue 14:00 – 15:00
Over $50
✓ checked
2 · review: nothing changed
18 of 42 orders · over $50
3 · confirm
Refund 18
one screen · the app already knows · checked as you type
In chat
Which orders?
Checkout, over $50
Which time range?
around Tuesday afternoon
✗ it guesses 12:00–18:00 · the wrong orders
2 questions · 2 extra round trips to the AI · still a guess
agentfootprint
1 · Two ways in
Backend functions give access to it.
User
All services ▾Last 7 days ▾Slowest ▾
By service
Checkout
Payments
Cart
Step 1/3
Step 2/3
Step 3/3
Chat
Which services got slower this week, and why?
Checkout and Payments slowed down after Tuesday’s deploy. Checkout’s slowest requests went from 220 to 610 ms. Cart and Login are flat.
which function the click calls
Agent
words → what you want
Functions
Data
MCP
1
2
3
1
2
3
✗
no screen built for this question
Which services got slower this week, and why?
Update, then Notify
Checkout ▾
On the screen
1 · fill in
Checkout · Tue 14:00 – 15:00
Over $50
✓ checked
2 · review: nothing changed
18 of 42 orders · over $50
3 · confirm
Refund 18
one screen · the app already knows · checked as you type
In chat
Which orders?
Checkout, over $50
Which time range?
around Tuesday afternoon
✗ it guesses 12:00–18:00 · the wrong orders
2 questions · 2 extra round trips to the AI · still a guess
agentfootprint
1 · Two ways in
A gateway is the front door to it.
User
All services ▾Last 7 days ▾Slowest ▾
By service
Checkout
Payments
Cart
Step 1/3
Step 2/3
Step 3/3
Chat
Which services got slower this week, and why?
Checkout and Payments slowed down after Tuesday’s deploy. Checkout’s slowest requests went from 220 to 610 ms. Cart and Login are flat.
which function the click calls
Agent
words → what you want
Functions
Data
MCP
1
2
3
1
2
3
✗
no screen built for this question
Which services got slower this week, and why?
Update, then Notify
Checkout ▾
On the screen
1 · fill in
Checkout · Tue 14:00 – 15:00
Over $50
✓ checked
2 · review: nothing changed
18 of 42 orders · over $50
3 · confirm
Refund 18
one screen · the app already knows · checked as you type
In chat
Which orders?
Checkout, over $50
Which time range?
around Tuesday afternoon
✗ it guesses 12:00–18:00 · the wrong orders
2 questions · 2 extra round trips to the AI · still a guess
agentfootprint
1 · Two ways in
Security at every layer.
User
All services ▾Last 7 days ▾Slowest ▾
By service
Checkout
Payments
Cart
Step 1/3
Step 2/3
Step 3/3
Chat
Which services got slower this week, and why?
Checkout and Payments slowed down after Tuesday’s deploy. Checkout’s slowest requests went from 220 to 610 ms. Cart and Login are flat.
which function the click calls
Agent
words → what you want
Functions
Data
MCP
1
2
3
1
2
3
✗
no screen built for this question
Which services got slower this week, and why?
Update, then Notify
Checkout ▾
On the screen
1 · fill in
Checkout · Tue 14:00 – 15:00
Over $50
✓ checked
2 · review: nothing changed
18 of 42 orders · over $50
3 · confirm
Refund 18
one screen · the app already knows · checked as you type
In chat
Which orders?
Checkout, over $50
Which time range?
around Tuesday afternoon
✗ it guesses 12:00–18:00 · the wrong orders
2 questions · 2 extra round trips to the AI · still a guess
every request is checked, at every layer
agentfootprint
1 · Two ways in
Then we design the UI: what people see and do.
User
All services ▾Last 7 days ▾Slowest ▾
By service
Checkout
Payments
Cart
Step 1/3
Step 2/3
Step 3/3
Chat
Which services got slower this week, and why?
Checkout and Payments slowed down after Tuesday’s deploy. Checkout’s slowest requests went from 220 to 610 ms. Cart and Login are flat.
which function the click calls
Agent
words → what you want
Functions
Data
MCP
1
2
3
1
2
3
✗
no screen built for this question
Which services got slower this week, and why?
Update, then Notify
Checkout ▾
On the screen
1 · fill in
Checkout · Tue 14:00 – 15:00
Over $50
✓ checked
2 · review: nothing changed
18 of 42 orders · over $50
3 · confirm
Refund 18
one screen · the app already knows · checked as you type
In chat
Which orders?
Checkout, over $50
Which time range?
around Tuesday afternoon
✗ it guesses 12:00–18:00 · the wrong orders
2 questions · 2 extra round trips to the AI · still a guess
screens · forms · tables · charts · workflows
agentfootprint
1 · Two ways in
People use it through the screens we built.
User
All services ▾Last 7 days ▾Slowest ▾
By service
Checkout
Payments
Cart
Step 1/3
Step 2/3
Step 3/3
Chat
Which services got slower this week, and why?
Checkout and Payments slowed down after Tuesday’s deploy. Checkout’s slowest requests went from 220 to 610 ms. Cart and Login are flat.
which function the click calls
Agent
words → what you want
Functions
Data
MCP
1
2
3
1
2
3
✗
no screen built for this question
Which services got slower this week, and why?
Update, then Notify
Checkout ▾
On the screen
1 · fill in
Checkout · Tue 14:00 – 15:00
Over $50
✓ checked
2 · review: nothing changed
18 of 42 orders · over $50
3 · confirm
Refund 18
one screen · the app already knows · checked as you type
In chat
Which orders?
Checkout, over $50
Which time range?
around Tuesday afternoon
✗ it guesses 12:00–18:00 · the wrong orders
2 questions · 2 extra round trips to the AI · still a guess
agentfootprint
1 · Two ways in
A person acts on a screen.
User
All services ▾Last 7 days ▾Slowest ▾
By service
Checkout
Payments
Cart
Step 1/3
Step 2/3
Step 3/3
Chat
Which services got slower this week, and why?
Checkout and Payments slowed down after Tuesday’s deploy. Checkout’s slowest requests went from 220 to 610 ms. Cart and Login are flat.
which function the click calls
Agent
words → what you want
Functions
Data
MCP
1
2
3
1
2
3
✗
no screen built for this question
Which services got slower this week, and why?
Update, then Notify
Checkout ▾
On the screen
1 · fill in
Checkout · Tue 14:00 – 15:00
Over $50
✓ checked
2 · review: nothing changed
18 of 42 orders · over $50
3 · confirm
Refund 18
one screen · the app already knows · checked as you type
In chat
Which orders?
Checkout, over $50
Which time range?
around Tuesday afternoon
✗ it guesses 12:00–18:00 · the wrong orders
2 questions · 2 extra round trips to the AI · still a guess
the user picks a filter
agentfootprint
1 · Two ways in
At run time, the action picks one route.
User
All services ▾Last 7 days ▾Slowest ▾
By service
Checkout
Payments
Cart
Step 1/3
Step 2/3
Step 3/3
Chat
Which services got slower this week, and why?
Checkout and Payments slowed down after Tuesday’s deploy. Checkout’s slowest requests went from 220 to 610 ms. Cart and Login are flat.
which function the click calls
Agent
words → what you want
Functions
Data
MCP
1
2
3
1
2
3
✗
no screen built for this question
Which services got slower this week, and why?
Update, then Notify
Checkout ▾
On the screen
1 · fill in
Checkout · Tue 14:00 – 15:00
Over $50
✓ checked
2 · review: nothing changed
18 of 42 orders · over $50
3 · confirm
Refund 18
one screen · the app already knows · checked as you type
In chat
Which orders?
Checkout, over $50
Which time range?
around Tuesday afternoon
✗ it guesses 12:00–18:00 · the wrong orders
2 questions · 2 extra round trips to the AI · still a guess
agentfootprint
1 · Two ways in
A function does the work, on its data.
User
All services ▾Last 7 days ▾Slowest ▾
By service
Checkout
Payments
Cart
Step 1/3
Step 2/3
Step 3/3
Chat
Which services got slower this week, and why?
Checkout and Payments slowed down after Tuesday’s deploy. Checkout’s slowest requests went from 220 to 610 ms. Cart and Login are flat.
which function the click calls
Agent
words → what you want
Functions
Data
MCP
1
2
3
1
2
3
✗
no screen built for this question
Which services got slower this week, and why?
Update, then Notify
Checkout ▾
On the screen
1 · fill in
Checkout · Tue 14:00 – 15:00
Over $50
✓ checked
2 · review: nothing changed
18 of 42 orders · over $50
3 · confirm
Refund 18
one screen · the app already knows · checked as you type
In chat
Which orders?
Checkout, over $50
Which time range?
around Tuesday afternoon
✗ it guesses 12:00–18:00 · the wrong orders
2 questions · 2 extra round trips to the AI · still a guess
agentfootprint
1 · Two ways in
The answer comes back as a chart.
User
All services ▾Last 7 days ▾Slowest ▾
By service
Checkout
Payments
Cart
Step 1/3
Step 2/3
Step 3/3
Chat
Which services got slower this week, and why?
Checkout and Payments slowed down after Tuesday’s deploy. Checkout’s slowest requests went from 220 to 610 ms. Cart and Login are flat.
which function the click calls
Agent
words → what you want
Functions
Data
MCP
1
2
3
1
2
3
✗
no screen built for this question
Which services got slower this week, and why?
Update, then Notify
Checkout ▾
On the screen
1 · fill in
Checkout · Tue 14:00 – 15:00
Over $50
✓ checked
2 · review: nothing changed
18 of 42 orders · over $50
3 · confirm
Refund 18
one screen · the app already knows · checked as you type
In chat
Which orders?
Checkout, over $50
Which time range?
around Tuesday afternoon
✗ it guesses 12:00–18:00 · the wrong orders
2 questions · 2 extra round trips to the AI · still a guess
agentfootprint
1 · Two ways in
Each screen only does what we built it to do.
User
All services ▾Last 7 days ▾Slowest ▾
By service
Checkout
Payments
Cart
Step 1/3
Step 2/3
Step 3/3
Chat
Which services got slower this week, and why?
Checkout and Payments slowed down after Tuesday’s deploy. Checkout’s slowest requests went from 220 to 610 ms. Cart and Login are flat.
which function the click calls
Agent
words → what you want
Functions
Data
MCP
1
2
3
1
2
3
✗
no screen built for this question
Which services got slower this week, and why?
Update, then Notify
Checkout ▾
On the screen
1 · fill in
Checkout · Tue 14:00 – 15:00
Over $50
✓ checked
2 · review: nothing changed
18 of 42 orders · over $50
3 · confirm
Refund 18
one screen · the app already knows · checked as you type
In chat
Which orders?
Checkout, over $50
Which time range?
around Tuesday afternoon
✗ it guesses 12:00–18:00 · the wrong orders
2 questions · 2 extra round trips to the AI · still a guess
one screen · one route · chosen in advance
agentfootprint
1 · Two ways in
① The chart: one route, to Search.
User
All services ▾Last 7 days ▾Slowest ▾
By service
Checkout
Payments
Cart
Step 1/3
Step 2/3
Step 3/3
Chat
Which services got slower this week, and why?
Checkout and Payments slowed down after Tuesday’s deploy. Checkout’s slowest requests went from 220 to 610 ms. Cart and Login are flat.
which function the click calls
Agent
words → what you want
Functions
Data
MCP
1
2
3
1
2
3
✗
no screen built for this question
Which services got slower this week, and why?
Update, then Notify
Checkout ▾
On the screen
1 · fill in
Checkout · Tue 14:00 – 15:00
Over $50
✓ checked
2 · review: nothing changed
18 of 42 orders · over $50
3 · confirm
Refund 18
one screen · the app already knows · checked as you type
In chat
Which orders?
Checkout, over $50
Which time range?
around Tuesday afternoon
✗ it guesses 12:00–18:00 · the wrong orders
2 questions · 2 extra round trips to the AI · still a guess
agentfootprint
1 · Two ways in
② The table: one route, to Compare.
User
All services ▾Last 7 days ▾Slowest ▾
By service
Checkout
Payments
Cart
Step 1/3
Step 2/3
Step 3/3
Chat
Which services got slower this week, and why?
Checkout and Payments slowed down after Tuesday’s deploy. Checkout’s slowest requests went from 220 to 610 ms. Cart and Login are flat.
which function the click calls
Agent
words → what you want
Functions
Data
MCP
1
2
3
1
2
3
✗
no screen built for this question
Which services got slower this week, and why?
Update, then Notify
Checkout ▾
On the screen
1 · fill in
Checkout · Tue 14:00 – 15:00
Over $50
✓ checked
2 · review: nothing changed
18 of 42 orders · over $50
3 · confirm
Refund 18
one screen · the app already knows · checked as you type
In chat
Which orders?
Checkout, over $50
Which time range?
around Tuesday afternoon
✗ it guesses 12:00–18:00 · the wrong orders
2 questions · 2 extra round trips to the AI · still a guess
agentfootprint
1 · Two ways in
③ The form is a workflow: Update, then Notify.
User
All services ▾Last 7 days ▾Slowest ▾
By service
Checkout
Payments
Cart
Step 1/3
Step 2/3
Step 3/3
Chat
Which services got slower this week, and why?
Checkout and Payments slowed down after Tuesday’s deploy. Checkout’s slowest requests went from 220 to 610 ms. Cart and Login are flat.
which function the click calls
Agent
words → what you want
Functions
Data
MCP
1
2
3
1
2
3
✗
no screen built for this question
Which services got slower this week, and why?
Update, then Notify
Checkout ▾
On the screen
1 · fill in
Checkout · Tue 14:00 – 15:00
Over $50
✓ checked
2 · review: nothing changed
18 of 42 orders · over $50
3 · confirm
Refund 18
one screen · the app already knows · checked as you type
In chat
Which orders?
Checkout, over $50
Which time range?
around Tuesday afternoon
✗ it guesses 12:00–18:00 · the wrong orders
2 questions · 2 extra round trips to the AI · still a guess
agentfootprint
1 · Two ways in
A user has a question.
User
All services ▾Last 7 days ▾Slowest ▾
By service
Checkout
Payments
Cart
Step 1/3
Step 2/3
Step 3/3
Chat
Which services got slower this week, and why?
Checkout and Payments slowed down after Tuesday’s deploy. Checkout’s slowest requests went from 220 to 610 ms. Cart and Login are flat.
which function the click calls
Agent
words → what you want
Functions
Data
MCP
1
2
3
1
2
3
✗
no screen built for this question
Which services got slower this week, and why?
Update, then Notify
Checkout ▾
On the screen
1 · fill in
Checkout · Tue 14:00 – 15:00
Over $50
✓ checked
2 · review: nothing changed
18 of 42 orders · over $50
3 · confirm
Refund 18
one screen · the app already knows · checked as you type
In chat
Which orders?
Checkout, over $50
Which time range?
around Tuesday afternoon
✗ it guesses 12:00–18:00 · the wrong orders
2 questions · 2 extra round trips to the AI · still a guess
agentfootprint
1 · Two ways in
Some questions have no screen.
User
All services ▾Last 7 days ▾Slowest ▾
By service
Checkout
Payments
Cart
Step 1/3
Step 2/3
Step 3/3
Chat
Which services got slower this week, and why?
Checkout and Payments slowed down after Tuesday’s deploy. Checkout’s slowest requests went from 220 to 610 ms. Cart and Login are flat.
which function the click calls
Agent
words → what you want
Functions
Data
MCP
1
2
3
1
2
3
✗
no screen built for this question
Which services got slower this week, and why?
Update, then Notify
Checkout ▾
On the screen
1 · fill in
Checkout · Tue 14:00 – 15:00
Over $50
✓ checked
2 · review: nothing changed
18 of 42 orders · over $50
3 · confirm
Refund 18
one screen · the app already knows · checked as you type
In chat
Which orders?
Checkout, over $50
Which time range?
around Tuesday afternoon
✗ it guesses 12:00–18:00 · the wrong orders
2 questions · 2 extra round trips to the AI · still a guess
no screen · no route for it
agentfootprint
1 · Two ways in
Chat: say it in your own words.
User
All services ▾Last 7 days ▾Slowest ▾
By service
Checkout
Payments
Cart
Step 1/3
Step 2/3
Step 3/3
Chat
Which services got slower this week, and why?
Checkout and Payments slowed down after Tuesday’s deploy. Checkout’s slowest requests went from 220 to 610 ms. Cart and Login are flat.
which function the click calls
Agent
words → what you want
Functions
Data
MCP
1
2
3
1
2
3
✗
no screen built for this question
Which services got slower this week, and why?
Update, then Notify
Checkout ▾
On the screen
1 · fill in
Checkout · Tue 14:00 – 15:00
Over $50
✓ checked
2 · review: nothing changed
18 of 42 orders · over $50
3 · confirm
Refund 18
one screen · the app already knows · checked as you type
In chat
Which orders?
Checkout, over $50
Which time range?
around Tuesday afternoon
✗ it guesses 12:00–18:00 · the wrong orders
2 questions · 2 extra round trips to the AI · still a guess
agentfootprint
1 · Two ways in
The AI works out what you actually want.
User
All services ▾Last 7 days ▾Slowest ▾
By service
Checkout
Payments
Cart
Step 1/3
Step 2/3
Step 3/3
Chat
Which services got slower this week, and why?
Checkout and Payments slowed down after Tuesday’s deploy. Checkout’s slowest requests went from 220 to 610 ms. Cart and Login are flat.
which function the click calls
Agent
words → what you want
Functions
Data
MCP
1
2
3
1
2
3
✗
no screen built for this question
Which services got slower this week, and why?
Update, then Notify
Checkout ▾
On the screen
1 · fill in
Checkout · Tue 14:00 – 15:00
Over $50
✓ checked
2 · review: nothing changed
18 of 42 orders · over $50
3 · confirm
Refund 18
one screen · the app already knows · checked as you type
In chat
Which orders?
Checkout, over $50
Which time range?
around Tuesday afternoon
✗ it guesses 12:00–18:00 · the wrong orders
2 questions · 2 extra round trips to the AI · still a guess
agentfootprint
1 · Two ways in
…and sends it through the same gateway.
User
All services ▾Last 7 days ▾Slowest ▾
By service
Checkout
Payments
Cart
Step 1/3
Step 2/3
Step 3/3
Chat
Which services got slower this week, and why?
Checkout and Payments slowed down after Tuesday’s deploy. Checkout’s slowest requests went from 220 to 610 ms. Cart and Login are flat.
which function the click calls
Agent
words → what you want
Functions
Data
MCP
1
2
3
1
2
3
✗
no screen built for this question
Which services got slower this week, and why?
Update, then Notify
Checkout ▾
On the screen
1 · fill in
Checkout · Tue 14:00 – 15:00
Over $50
✓ checked
2 · review: nothing changed
18 of 42 orders · over $50
3 · confirm
Refund 18
one screen · the app already knows · checked as you type
In chat
Which orders?
Checkout, over $50
Which time range?
around Tuesday afternoon
✗ it guesses 12:00–18:00 · the wrong orders
2 questions · 2 extra round trips to the AI · still a guess
agentfootprint
1 · Two ways in
What you want picks the functions.
User
All services ▾Last 7 days ▾Slowest ▾
By service
Checkout
Payments
Cart
Step 1/3
Step 2/3
Step 3/3
Chat
Which services got slower this week, and why?
Checkout and Payments slowed down after Tuesday’s deploy. Checkout’s slowest requests went from 220 to 610 ms. Cart and Login are flat.
which function the click calls
Agent
words → what you want
Functions
Data
MCP
1
2
3
1
2
3
✗
no screen built for this question
Which services got slower this week, and why?
Update, then Notify
Checkout ▾
On the screen
1 · fill in
Checkout · Tue 14:00 – 15:00
Over $50
✓ checked
2 · review: nothing changed
18 of 42 orders · over $50
3 · confirm
Refund 18
one screen · the app already knows · checked as you type
In chat
Which orders?
Checkout, over $50
Which time range?
around Tuesday afternoon
✗ it guesses 12:00–18:00 · the wrong orders
2 questions · 2 extra round trips to the AI · still a guess
agentfootprint
1 · Two ways in
The answer comes back as text.
User
All services ▾Last 7 days ▾Slowest ▾
By service
Checkout
Payments
Cart
Step 1/3
Step 2/3
Step 3/3
Chat
Which services got slower this week, and why?
Checkout and Payments slowed down after Tuesday’s deploy. Checkout’s slowest requests went from 220 to 610 ms. Cart and Login are flat.
which function the click calls
Agent
words → what you want
Functions
Data
MCP
1
2
3
1
2
3
✗
no screen built for this question
Which services got slower this week, and why?
Update, then Notify
Checkout ▾
On the screen
1 · fill in
Checkout · Tue 14:00 – 15:00
Over $50
✓ checked
2 · review: nothing changed
18 of 42 orders · over $50
3 · confirm
Refund 18
one screen · the app already knows · checked as you type
In chat
Which orders?
Checkout, over $50
Which time range?
around Tuesday afternoon
✗ it guesses 12:00–18:00 · the wrong orders
2 questions · 2 extra round trips to the AI · still a guess
agentfootprint
1 · Two ways in
Two ways in. One gateway.
User
All services ▾Last 7 days ▾Slowest ▾
By service
Checkout
Payments
Cart
Step 1/3
Step 2/3
Step 3/3
Chat
Which services got slower this week, and why?
Checkout and Payments slowed down after Tuesday’s deploy. Checkout’s slowest requests went from 220 to 610 ms. Cart and Login are flat.
which function the click calls
Agent
words → what you want
Functions
Data
MCP
1
2
3
1
2
3
✗
no screen built for this question
Which services got slower this week, and why?
Update, then Notify
Checkout ▾
On the screen
1 · fill in
Checkout · Tue 14:00 – 15:00
Over $50
✓ checked
2 · review: nothing changed
18 of 42 orders · over $50
3 · confirm
Refund 18
one screen · the app already knows · checked as you type
In chat
Which orders?
Checkout, over $50
Which time range?
around Tuesday afternoon
✗ it guesses 12:00–18:00 · the wrong orders
2 questions · 2 extra round trips to the AI · still a guess
a click: a direct action
words, then the AI acts
agentfootprint
1 · Two ways in
Now a task: refund the orders hit by Tuesday’s deploy.
User
All services ▾Last 7 days ▾Slowest ▾
By service
Checkout
Payments
Cart
Step 1/3
Step 2/3
Step 3/3
Chat
Which services got slower this week, and why?
Checkout and Payments slowed down after Tuesday’s deploy. Checkout’s slowest requests went from 220 to 610 ms. Cart and Login are flat.
which function the click calls
Agent
words → what you want
Functions
Data
MCP
1
2
3
1
2
3
✗
no screen built for this question
Which services got slower this week, and why?
Update, then Notify
Checkout ▾
On the screen
1 · fill in
Checkout · Tue 14:00 – 15:00
Over $50
✓ checked
2 · review: nothing changed
18 of 42 orders · over $50
3 · confirm
Refund 18
one screen · the app already knows · checked as you type
In chat
Which orders?
Checkout, over $50
Which time range?
around Tuesday afternoon
✗ it guesses 12:00–18:00 · the wrong orders
2 questions · 2 extra round trips to the AI · still a guess
Checkout · Tuesday 14:00–15:00 · only orders over $50
the refund form
agentfootprint
1 · Two ways in
Step 1 is gathering the input.
User
All services ▾Last 7 days ▾Slowest ▾
By service
Checkout
Payments
Cart
Step 1/3
Step 2/3
Step 3/3
Chat
Which services got slower this week, and why?
Checkout and Payments slowed down after Tuesday’s deploy. Checkout’s slowest requests went from 220 to 610 ms. Cart and Login are flat.
which function the click calls
Agent
words → what you want
Functions
Data
MCP
1
2
3
1
2
3
✗
no screen built for this question
Which services got slower this week, and why?
Update, then Notify
Checkout ▾
On the screen
1 · fill in
Checkout · Tue 14:00 – 15:00
Over $50
✓ checked
2 · review: nothing changed
18 of 42 orders · over $50
3 · confirm
Refund 18
one screen · the app already knows · checked as you type
In chat
Which orders?
Checkout, over $50
Which time range?
around Tuesday afternoon
✗ it guesses 12:00–18:00 · the wrong orders
2 questions · 2 extra round trips to the AI · still a guess
the screen’s three steps
agentfootprint
1 · Two ways in
Step 2 reviews it.
User
All services ▾Last 7 days ▾Slowest ▾
By service
Checkout
Payments
Cart
Step 1/3
Step 2/3
Step 3/3
Chat
Which services got slower this week, and why?
Checkout and Payments slowed down after Tuesday’s deploy. Checkout’s slowest requests went from 220 to 610 ms. Cart and Login are flat.
which function the click calls
Agent
words → what you want
Functions
Data
MCP
1
2
3
1
2
3
✗
no screen built for this question
Which services got slower this week, and why?
Update, then Notify
Checkout ▾
On the screen
1 · fill in
Checkout · Tue 14:00 – 15:00
Over $50
✓ checked
2 · review: nothing changed
18 of 42 orders · over $50
3 · confirm
Refund 18
one screen · the app already knows · checked as you type
In chat
Which orders?
Checkout, over $50
Which time range?
around Tuesday afternoon
✗ it guesses 12:00–18:00 · the wrong orders
2 questions · 2 extra round trips to the AI · still a guess
the screen’s three steps
agentfootprint
1 · Two ways in
Step 3 confirms.
User
All services ▾Last 7 days ▾Slowest ▾
By service
Checkout
Payments
Cart
Step 1/3
Step 2/3
Step 3/3
Chat
Which services got slower this week, and why?
Checkout and Payments slowed down after Tuesday’s deploy. Checkout’s slowest requests went from 220 to 610 ms. Cart and Login are flat.
which function the click calls
Agent
words → what you want
Functions
Data
MCP
1
2
3
1
2
3
✗
no screen built for this question
Which services got slower this week, and why?
Update, then Notify
Checkout ▾
On the screen
1 · fill in
Checkout · Tue 14:00 – 15:00
Over $50
✓ checked
2 · review: nothing changed
18 of 42 orders · over $50
3 · confirm
Refund 18
one screen · the app already knows · checked as you type
In chat
Which orders?
Checkout, over $50
Which time range?
around Tuesday afternoon
✗ it guesses 12:00–18:00 · the wrong orders
2 questions · 2 extra round trips to the AI · still a guess
the screen’s three steps
agentfootprint
1 · Two ways in
In chat, the agent has to ask.
User
All services ▾Last 7 days ▾Slowest ▾
By service
Checkout
Payments
Cart
Step 1/3
Step 2/3
Step 3/3
Chat
Which services got slower this week, and why?
Checkout and Payments slowed down after Tuesday’s deploy. Checkout’s slowest requests went from 220 to 610 ms. Cart and Login are flat.
which function the click calls
Agent
words → what you want
Functions
Data
MCP
1
2
3
1
2
3
✗
no screen built for this question
Which services got slower this week, and why?
Update, then Notify
Checkout ▾
On the screen
1 · fill in
Checkout · Tue 14:00 – 15:00
Over $50
✓ checked
2 · review: nothing changed
18 of 42 orders · over $50
3 · confirm
Refund 18
one screen · the app already knows · checked as you type
In chat
Which orders?
Checkout, over $50
Which time range?
around Tuesday afternoon
✗ it guesses 12:00–18:00 · the wrong orders
2 questions · 2 extra round trips to the AI · still a guess
the screen’s three steps
agentfootprint
1 · Two ways in
Every answer is another round trip to the AI.
User
All services ▾Last 7 days ▾Slowest ▾
By service
Checkout
Payments
Cart
Step 1/3
Step 2/3
Step 3/3
Chat
Which services got slower this week, and why?
Checkout and Payments slowed down after Tuesday’s deploy. Checkout’s slowest requests went from 220 to 610 ms. Cart and Login are flat.
which function the click calls
Agent
words → what you want
Functions
Data
MCP
1
2
3
1
2
3
✗
no screen built for this question
Which services got slower this week, and why?
Update, then Notify
Checkout ▾
On the screen
1 · fill in
Checkout · Tue 14:00 – 15:00
Over $50
✓ checked
2 · review: nothing changed
18 of 42 orders · over $50
3 · confirm
Refund 18
one screen · the app already knows · checked as you type
In chat
Which orders?
Checkout, over $50
Which time range?
around Tuesday afternoon
✗ it guesses 12:00–18:00 · the wrong orders
2 questions · 2 extra round trips to the AI · still a guess
the screen’s three steps
agentfootprint
1 · Two ways in
And the answer can come back wrong.
User
All services ▾Last 7 days ▾Slowest ▾
By service
Checkout
Payments
Cart
Step 1/3
Step 2/3
Step 3/3
Chat
Which services got slower this week, and why?
Checkout and Payments slowed down after Tuesday’s deploy. Checkout’s slowest requests went from 220 to 610 ms. Cart and Login are flat.
which function the click calls
Agent
words → what you want
Functions
Data
MCP
1
2
3
1
2
3
✗
no screen built for this question
Which services got slower this week, and why?
Update, then Notify
Checkout ▾
On the screen
1 · fill in
Checkout · Tue 14:00 – 15:00
Over $50
✓ checked
2 · review: nothing changed
18 of 42 orders · over $50
3 · confirm
Refund 18
one screen · the app already knows · checked as you type
In chat
Which orders?
Checkout, over $50
Which time range?
around Tuesday afternoon
✗ it guesses 12:00–18:00 · the wrong orders
2 questions · 2 extra round trips to the AI · still a guess
the screen’s three steps
agentfootprint
1 · Two ways in
Slower, costlier, and less certain.
User
All services ▾Last 7 days ▾Slowest ▾
By service
Checkout
Payments
Cart
Step 1/3
Step 2/3
Step 3/3
Chat
Which services got slower this week, and why?
Checkout and Payments slowed down after Tuesday’s deploy. Checkout’s slowest requests went from 220 to 610 ms. Cart and Login are flat.
which function the click calls
Agent
words → what you want
Functions
Data
MCP
1
2
3
1
2
3
✗
no screen built for this question
Which services got slower this week, and why?
Update, then Notify
Checkout ▾
On the screen
1 · fill in
Checkout · Tue 14:00 – 15:00
Over $50
✓ checked
2 · review: nothing changed
18 of 42 orders · over $50
3 · confirm
Refund 18
one screen · the app already knows · checked as you type
In chat
Which orders?
Checkout, over $50
Which time range?
around Tuesday afternoon
✗ it guesses 12:00–18:00 · the wrong orders
2 questions · 2 extra round trips to the AI · still a guess
the screen’s three steps
agentfootprint
1 · Two ways in
Each one is missing something.
User
All services ▾Last 7 days ▾Slowest ▾
By service
Checkout
Payments
Cart
Step 1/3
Step 2/3
Step 3/3
Chat
Which services got slower this week, and why?
Checkout and Payments slowed down after Tuesday’s deploy. Checkout’s slowest requests went from 220 to 610 ms. Cart and Login are flat.
which function the click calls
Agent
words → what you want
Functions
Data
MCP
1
2
3
1
2
3
✗
no screen built for this question
Which services got slower this week, and why?
Update, then Notify
Checkout ▾
On the screen
1 · fill in
Checkout · Tue 14:00 – 15:00
Over $50
✓ checked
2 · review: nothing changed
18 of 42 orders · over $50
3 · confirm
Refund 18
one screen · the app already knows · checked as you type
In chat
Which orders?
Checkout, over $50
Which time range?
around Tuesday afternoon
✗ it guesses 12:00–18:00 · the wrong orders
2 questions · 2 extra round trips to the AI · still a guess
only the paths we designed
a paragraph to read
agentfootprint
1 · Two ways in
How do we get the best of both?
User
All services ▾Last 7 days ▾Slowest ▾
By service
Checkout
Payments
Cart
Step 1/3
Step 2/3
Step 3/3
Chat
Which services got slower this week, and why?
Checkout and Payments slowed down after Tuesday’s deploy. Checkout’s slowest requests went from 220 to 610 ms. Cart and Login are flat.
which function the click calls
Agent
words → what you want
Functions
Data
MCP
1
2
3
1
2
3
✗
no screen built for this question
Which services got slower this week, and why?
Update, then Notify
Checkout ▾
On the screen
1 · fill in
Checkout · Tue 14:00 – 15:00
Over $50
✓ checked
2 · review: nothing changed
18 of 42 orders · over $50
3 · confirm
Refund 18
one screen · the app already knows · checked as you type
In chat
Which orders?
Checkout, over $50
Which time range?
around Tuesday afternoon
✗ it guesses 12:00–18:00 · the wrong orders
2 questions · 2 extra round trips to the AI · still a guess
precise · structured · visual · on the path
in your own words
agentfootprint
1 · Two ways in
One answer · do it all in chat
All-in-chat puts
the work on the user.
The chat · the AI asks, you type, it writes back
The user’s job · everything the app used to do
agentfootprint
1 · Two ways in
One answer · do it all in chat
All-in-chat puts
the work on the user.
The chat · the AI asks, you type, it writes back
The user’s job · everything the app used to do
agentfootprint
1 · Two ways in
Back to basics
That's why we built
visual interfaces.
Form
gives structured input
Selection
says exactly which item
The screen carries it, not the user.
The stack · what the user carries in chat
Chart · you see the pattern, nothing to read
Table · side by side, nothing to remember
Form · its fields ask, and only for what’s allowed
Selection · says which one, the context stays on screen
All of it · off the user, onto the screen
agentfootprint
1 · Two ways in
Back to basics
That's why we built
visual interfaces.
Form
gives structured input
Selection
says exactly which item
The screen carries it, not the user.
The stack · what the user carries in chat
Chart · you see the pattern, nothing to read
Table · side by side, nothing to remember
Form · its fields ask, and only for what’s allowed
Selection · says which one, the context stays on screen
All of it · off the user, onto the screen
agentfootprint
1 · Two ways in
Back to basics
That's why we built
visual interfaces.
Form
gives structured input
Selection
says exactly which item
The screen carries it, not the user.
The stack · what the user carries in chat
Chart · you see the pattern, nothing to read
Table · side by side, nothing to remember
Form · its fields ask, and only for what’s allowed
Selection · says which one, the context stays on screen
All of it · off the user, onto the screen
agentfootprint
1 · Two ways in
Back to basics
That's why we built
visual interfaces.
Form
gives structured input
Selection
says exactly which item
The screen carries it, not the user.
The stack · what the user carries in chat
Chart · you see the pattern, nothing to read
Table · side by side, nothing to remember
Form · its fields ask, and only for what’s allowed
Selection · says which one, the context stays on screen
All of it · off the user, onto the screen
agentfootprint
1 · Two ways in
Back to basics
That's why we built
visual interfaces.
Form
gives structured input
Selection
says exactly which item
The screen carries it, not the user.
The stack · what the user carries in chat
Chart · you see the pattern, nothing to read
Table · side by side, nothing to remember
Form · its fields ask, and only for what’s allowed
Selection · says which one, the context stays on screen
All of it · off the user, onto the screen
agentfootprint
1 · Two ways in
Back to basics
That's why we built
visual interfaces.
Form
gives structured input
Selection
says exactly which item
The screen carries it, not the user.
The stack · what the user carries in chat
Chart · you see the pattern, nothing to read
Table · side by side, nothing to remember
Form · its fields ask, and only for what’s allowed
Selection · says which one, the context stays on screen
All of it · off the user, onto the screen
agentfootprint
02
Part 2 of 7
Try 1: draw the UI
in the chat
What we do today, and what it leaves out.
1Two ways in2Try 1: UI in chat3Try 2: AI clicks4When is it done?5Action Binding6In practice7Two of us
agentfootprint
2 · Try 1: UI in chat
What we do today
So we bring the screen
into the chat.
Form
gives structured input
Selection
says exactly which item
The AI draws it,
right in its reply.
agentfootprint
2 · Try 1: UI in chat
What we do today
So we bring the screen
into the chat.
Form
gives structured input
Selection
says exactly which item
The AI draws it,
right in its reply.
agentfootprint
2 · Try 1: UI in chat
What it looks like today
All of it lands
in the chat.
One chat window · the same task as before
Refund the Checkout orders hit by Tuesday’s deploy.
You ask · a real task, in your own words
thinking…
The AI thinks · you wait
It asks you · a form, rebuilt in the chat
It works · you watch the steps go by
The answer · a chart, drawn in the chat
Refund the Checkout orders hit by Tuesday’s deploy.
thinking…
And Payments?
Only the ones over $50.
thinking…
the chart scrolled away
Follow-ups · and you say “over $50” again
Refund the Checkout orders hit by Tuesday’s deploy.
thinking…
And Payments?
Only the ones over $50.
thinking…
where was that chart?
Scroll back · to find the answer
One stream. The user waits, watches, and scrolls.
What it took · from the user
agentfootprint
2 · Try 1: UI in chat
What it looks like today
All of it lands
in the chat.
One chat window · the same task as before
Refund the Checkout orders hit by Tuesday’s deploy.
You ask · a real task, in your own words
thinking…
The AI thinks · you wait
It asks you · a form, rebuilt in the chat
It works · you watch the steps go by
The answer · a chart, drawn in the chat
Refund the Checkout orders hit by Tuesday’s deploy.
thinking…
And Payments?
Only the ones over $50.
thinking…
the chart scrolled away
Follow-ups · and you say “over $50” again
Refund the Checkout orders hit by Tuesday’s deploy.
thinking…
And Payments?
Only the ones over $50.
thinking…
where was that chart?
Scroll back · to find the answer
One stream. The user waits, watches, and scrolls.
What it took · from the user
agentfootprint
2 · Try 1: UI in chat
What it looks like today
All of it lands
in the chat.
One chat window · the same task as before
Refund the Checkout orders hit by Tuesday’s deploy.
You ask · a real task, in your own words
thinking…
The AI thinks · you wait
It asks you · a form, rebuilt in the chat
It works · you watch the steps go by
The answer · a chart, drawn in the chat
Refund the Checkout orders hit by Tuesday’s deploy.
thinking…
And Payments?
Only the ones over $50.
thinking…
the chart scrolled away
Follow-ups · and you say “over $50” again
Refund the Checkout orders hit by Tuesday’s deploy.
thinking…
And Payments?
Only the ones over $50.
thinking…
where was that chart?
Scroll back · to find the answer
One stream. The user waits, watches, and scrolls.
What it took · from the user
agentfootprint
2 · Try 1: UI in chat
What it looks like today
All of it lands
in the chat.
One chat window · the same task as before
Refund the Checkout orders hit by Tuesday’s deploy.
You ask · a real task, in your own words
thinking…
The AI thinks · you wait
It asks you · a form, rebuilt in the chat
It works · you watch the steps go by
The answer · a chart, drawn in the chat
Refund the Checkout orders hit by Tuesday’s deploy.
thinking…
And Payments?
Only the ones over $50.
thinking…
the chart scrolled away
Follow-ups · and you say “over $50” again
Refund the Checkout orders hit by Tuesday’s deploy.
thinking…
And Payments?
Only the ones over $50.
thinking…
where was that chart?
Scroll back · to find the answer
One stream. The user waits, watches, and scrolls.
What it took · from the user
agentfootprint
2 · Try 1: UI in chat
What it looks like today
All of it lands
in the chat.
One chat window · the same task as before
Refund the Checkout orders hit by Tuesday’s deploy.
You ask · a real task, in your own words
thinking…
The AI thinks · you wait
It asks you · a form, rebuilt in the chat
It works · you watch the steps go by
The answer · a chart, drawn in the chat
Refund the Checkout orders hit by Tuesday’s deploy.
thinking…
And Payments?
Only the ones over $50.
thinking…
the chart scrolled away
Follow-ups · and you say “over $50” again
Refund the Checkout orders hit by Tuesday’s deploy.
thinking…
And Payments?
Only the ones over $50.
thinking…
where was that chart?
Scroll back · to find the answer
One stream. The user waits, watches, and scrolls.
What it took · from the user
agentfootprint
2 · Try 1: UI in chat
What it looks like today
All of it lands
in the chat.
One chat window · the same task as before
Refund the Checkout orders hit by Tuesday’s deploy.
You ask · a real task, in your own words
thinking…
The AI thinks · you wait
It asks you · a form, rebuilt in the chat
It works · you watch the steps go by
The answer · a chart, drawn in the chat
Refund the Checkout orders hit by Tuesday’s deploy.
thinking…
And Payments?
Only the ones over $50.
thinking…
the chart scrolled away
Follow-ups · and you say “over $50” again
Refund the Checkout orders hit by Tuesday’s deploy.
thinking…
And Payments?
Only the ones over $50.
thinking…
where was that chart?
Scroll back · to find the answer
One stream. The user waits, watches, and scrolls.
What it took · from the user
agentfootprint
2 · Try 1: UI in chat
What it looks like today
All of it lands
in the chat.
One chat window · the same task as before
Refund the Checkout orders hit by Tuesday’s deploy.
You ask · a real task, in your own words
thinking…
The AI thinks · you wait
It asks you · a form, rebuilt in the chat
It works · you watch the steps go by
The answer · a chart, drawn in the chat
Refund the Checkout orders hit by Tuesday’s deploy.
thinking…
And Payments?
Only the ones over $50.
thinking…
the chart scrolled away
Follow-ups · and you say “over $50” again
Refund the Checkout orders hit by Tuesday’s deploy.
thinking…
And Payments?
Only the ones over $50.
thinking…
where was that chart?
Scroll back · to find the answer
One stream. The user waits, watches, and scrolls.
What it took · from the user
agentfootprint
2 · Try 1: UI in chat
What it looks like today
All of it lands
in the chat.
One chat window · the same task as before
Refund the Checkout orders hit by Tuesday’s deploy.
You ask · a real task, in your own words
thinking…
The AI thinks · you wait
It asks you · a form, rebuilt in the chat
It works · you watch the steps go by
The answer · a chart, drawn in the chat
Refund the Checkout orders hit by Tuesday’s deploy.
thinking…
And Payments?
Only the ones over $50.
thinking…
the chart scrolled away
Follow-ups · and you say “over $50” again
Refund the Checkout orders hit by Tuesday’s deploy.
thinking…
And Payments?
Only the ones over $50.
thinking…
where was that chart?
Scroll back · to find the answer
One stream. The user waits, watches, and scrolls.
What it took · from the user
agentfootprint
2 · Try 1: UI in chat
What it looks like today
All of it lands
in the chat.
One chat window · the same task as before
Refund the Checkout orders hit by Tuesday’s deploy.
You ask · a real task, in your own words
thinking…
The AI thinks · you wait
It asks you · a form, rebuilt in the chat
It works · you watch the steps go by
The answer · a chart, drawn in the chat
Refund the Checkout orders hit by Tuesday’s deploy.
thinking…
And Payments?
Only the ones over $50.
thinking…
the chart scrolled away
Follow-ups · and you say “over $50” again
Refund the Checkout orders hit by Tuesday’s deploy.
thinking…
And Payments?
Only the ones over $50.
thinking…
where was that chart?
Scroll back · to find the answer
One stream. The user waits, watches, and scrolls.
What it took · from the user
agentfootprint
2 · Try 1: UI in chat
What we do today
UI in the chat:
input, process, output.
① Input
a tool’s form asks for it
② Process
the AI’s steps, as they run
③ Output
the AI draws it on the fly
But it all stays inside the chat. The work belongs in the app.
Three places UI shows up in the chat today
① Input · a form asks for what a tool needs
② Process · which step is running, what came back
③ Output · generative UI: charts and forms the AI draws on the fly
But · all three are rebuilt inside the chat
agentfootprint
2 · Try 1: UI in chat
What we do today
UI in the chat:
input, process, output.
① Input
a tool’s form asks for it
② Process
the AI’s steps, as they run
③ Output
the AI draws it on the fly
But it all stays inside the chat. The work belongs in the app.
Three places UI shows up in the chat today
① Input · a form asks for what a tool needs
② Process · which step is running, what came back
③ Output · generative UI: charts and forms the AI draws on the fly
But · all three are rebuilt inside the chat
agentfootprint
2 · Try 1: UI in chat
What we do today
UI in the chat:
input, process, output.
① Input
a tool’s form asks for it
② Process
the AI’s steps, as they run
③ Output
the AI draws it on the fly
But it all stays inside the chat. The work belongs in the app.
Three places UI shows up in the chat today
① Input · a form asks for what a tool needs
② Process · which step is running, what came back
③ Output · generative UI: charts and forms the AI draws on the fly
But · all three are rebuilt inside the chat
agentfootprint
2 · Try 1: UI in chat
What we do today
UI in the chat:
input, process, output.
① Input
a tool’s form asks for it
② Process
the AI’s steps, as they run
③ Output
the AI draws it on the fly
But it all stays inside the chat. The work belongs in the app.
Three places UI shows up in the chat today
① Input · a form asks for what a tool needs
② Process · which step is running, what came back
③ Output · generative UI: charts and forms the AI draws on the fly
But · all three are rebuilt inside the chat
agentfootprint
2 · Try 1: UI in chat
What we do today
UI in the chat:
input, process, output.
① Input
a tool’s form asks for it
② Process
the AI’s steps, as they run
③ Output
the AI draws it on the fly
But it all stays inside the chat. The work belongs in the app.
Three places UI shows up in the chat today
① Input · a form asks for what a tool needs
② Process · which step is running, what came back
③ Output · generative UI: charts and forms the AI draws on the fly
But · all three are rebuilt inside the chat
agentfootprint
03
Part 3 of 7
Try 2: let the AI
click the app
Like a person would, and where it breaks.
1Two ways in2Try 1: UI in chat3Try 2: AI clicks4When is it done?5Action Binding6In practice7Two of us
agentfootprint
3 · Try 2: AI clicks
Acting in the app
Every agent has to answer
three questions.
your-app.com/orders
Service ▾Last 7 days ▾Apply
①Where do I go?
navigation · finding the right page
②What can I do here?
actions · what this page offers
③Did it work?
completion · can I go on
Two designs: from the screen, or from the page’s own tools.
agentfootprint
3 · Try 2: AI clicks
Acting in the app
Every agent has to answer
three questions.
your-app.com/orders
Service ▾Last 7 days ▾Apply
①Where do I go?
navigation · finding the right page
②What can I do here?
actions · what this page offers
③Did it work?
completion · can I go on
Two designs: from the screen, or from the page’s own tools.
agentfootprint
3 · Try 2: AI clicks
Acting in the app
Every agent has to answer
three questions.
your-app.com/orders
Service ▾Last 7 days ▾Apply
①Where do I go?
navigation · finding the right page
②What can I do here?
actions · what this page offers
③Did it work?
completion · can I go on
Two designs: from the screen, or from the page’s own tools.
agentfootprint
3 · Try 2: AI clicks
Acting in the app
Every agent has to answer
three questions.
your-app.com/orders
Service ▾Last 7 days ▾Apply
①Where do I go?
navigation · finding the right page
②What can I do here?
actions · what this page offers
③Did it work?
completion · can I go on
Two designs: from the screen, or from the page’s own tools.
agentfootprint
3 · Try 2: AI clicks
Acting in the app
Every agent has to answer
three questions.
your-app.com/orders
Service ▾Last 7 days ▾Apply
①Where do I go?
navigation · finding the right page
②What can I do here?
actions · what this page offers
③Did it work?
completion · can I go on
Two designs: from the screen, or from the page’s own tools.
agentfootprint
3 · Try 2: AI clicks
From the outside
Browser-use and computer-use
work from the screen.
agent
screenshot
<div class="toolbar">
<select>…</select>
<button>Apply</button>
</div>
<svg class="chart">
<path d="…"/>
</svg>
the page’s code
main
toolbar
combobox "Service"
button "Apply"
img "chart"
screen-reader view
ITS TOOLSscreenshot · click
type · scroll
① Where do I go? No map:
it explores, like a new user
② What can I do here?
Guess from pixels and labels.
③ Did it work? No signal:
wait, look again, guess.
agentfootprint
3 · Try 2: AI clicks
From the outside
Browser-use and computer-use
work from the screen.
agent
screenshot
<div class="toolbar">
<select>…</select>
<button>Apply</button>
</div>
<svg class="chart">
<path d="…"/>
</svg>
the page’s code
main
toolbar
combobox "Service"
button "Apply"
img "chart"
screen-reader view
ITS TOOLSscreenshot · click
type · scroll
① Where do I go? No map:
it explores, like a new user
② What can I do here?
Guess from pixels and labels.
③ Did it work? No signal:
wait, look again, guess.
agentfootprint
3 · Try 2: AI clicks
From the outside
Browser-use and computer-use
work from the screen.
agent
screenshot
<div class="toolbar">
<select>…</select>
<button>Apply</button>
</div>
<svg class="chart">
<path d="…"/>
</svg>
the page’s code
main
toolbar
combobox "Service"
button "Apply"
img "chart"
screen-reader view
ITS TOOLSscreenshot · click
type · scroll
① Where do I go? No map:
it explores, like a new user
② What can I do here?
Guess from pixels and labels.
③ Did it work? No signal:
wait, look again, guess.
agentfootprint
3 · Try 2: AI clicks
From the outside
Browser-use and computer-use
work from the screen.
agent
screenshot
<div class="toolbar">
<select>…</select>
<button>Apply</button>
</div>
<svg class="chart">
<path d="…"/>
</svg>
the page’s code
main
toolbar
combobox "Service"
button "Apply"
img "chart"
screen-reader view
ITS TOOLSscreenshot · click
type · scroll
① Where do I go? No map:
it explores, like a new user
② What can I do here?
Guess from pixels and labels.
③ Did it work? No signal:
wait, look again, guess.
agentfootprint
3 · Try 2: AI clicks
From the outside
Browser-use and computer-use
work from the screen.
agent
screenshot
<div class="toolbar">
<select>…</select>
<button>Apply</button>
</div>
<svg class="chart">
<path d="…"/>
</svg>
the page’s code
main
toolbar
combobox "Service"
button "Apply"
img "chart"
screen-reader view
ITS TOOLSscreenshot · click
type · scroll
① Where do I go? No map:
it explores, like a new user
② What can I do here?
Guess from pixels and labels.
③ Did it work? No signal:
wait, look again, guess.
agentfootprint
3 · Try 2: AI clicks
From the outside
Browser-use and computer-use
work from the screen.
agent
screenshot
<div class="toolbar">
<select>…</select>
<button>Apply</button>
</div>
<svg class="chart">
<path d="…"/>
</svg>
the page’s code
main
toolbar
combobox "Service"
button "Apply"
img "chart"
screen-reader view
ITS TOOLSscreenshot · click
type · scroll
① Where do I go? No map:
it explores, like a new user
② What can I do here?
Guess from pixels and labels.
③ Did it work? No signal:
wait, look again, guess.
agentfootprint
3 · Try 2: AI clicks
From the outside
Browser-use and computer-use
work from the screen.
agent
screenshot
<div class="toolbar">
<select>…</select>
<button>Apply</button>
</div>
<svg class="chart">
<path d="…"/>
</svg>
the page’s code
main
toolbar
combobox "Service"
button "Apply"
img "chart"
screen-reader view
ITS TOOLSscreenshot · click
type · scroll
① Where do I go? No map:
it explores, like a new user
② What can I do here?
Guess from pixels and labels.
③ Did it work? No signal:
wait, look again, guess.
agentfootprint
3 · Try 2: AI clicks
From the outside
Browser-use and computer-use
work from the screen.
agent
screenshot
<div class="toolbar">
<select>…</select>
<button>Apply</button>
</div>
<svg class="chart">
<path d="…"/>
</svg>
the page’s code
main
toolbar
combobox "Service"
button "Apply"
img "chart"
screen-reader view
ITS TOOLSscreenshot · click
type · scroll
① Where do I go? No map:
it explores, like a new user
② What can I do here?
Guess from pixels and labels.
③ Did it work? No signal:
wait, look again, guess.
agentfootprint
3 · Try 2: AI clicks
From the outside
Browser-use and computer-use
work from the screen.
agent
screenshot
<div class="toolbar">
<select>…</select>
<button>Apply</button>
</div>
<svg class="chart">
<path d="…"/>
</svg>
the page’s code
main
toolbar
combobox "Service"
button "Apply"
img "chart"
screen-reader view
ITS TOOLSscreenshot · click
type · scroll
① Where do I go? No map:
it explores, like a new user
② What can I do here?
Guess from pixels and labels.
③ Did it work? No signal:
wait, look again, guess.
agentfootprint
3 · Try 2: AI clicks
From the page itself
WebMCP: the page offers
its own tools.
your-app.com/orders
Service ▾Last 7 days ▾Apply
① tools exist per page · navigate, the list changes
→ filter_orders({ service: "Checkout", range: "Tue 14–15" })
③ it returned · but is the screen ready?
← { ok: true }
agentfootprint
3 · Try 2: AI clicks
From the page itself
WebMCP: the page offers
its own tools.
your-app.com/orders
Service ▾Last 7 days ▾Apply
① tools exist per page · navigate, the list changes
→ filter_orders({ service: "Checkout", range: "Tue 14–15" })
③ it returned · but is the screen ready?
← { ok: true }
agentfootprint
3 · Try 2: AI clicks
From the page itself
WebMCP: the page offers
its own tools.
your-app.com/orders
Service ▾Last 7 days ▾Apply
① tools exist per page · navigate, the list changes
→ filter_orders({ service: "Checkout", range: "Tue 14–15" })
③ it returned · but is the screen ready?
← { ok: true }
agentfootprint
3 · Try 2: AI clicks
From the page itself
WebMCP: the page offers
its own tools.
your-app.com/orders
Service ▾Last 7 days ▾Apply
① tools exist per page · navigate, the list changes
→ filter_orders({ service: "Checkout", range: "Tue 14–15" })
③ it returned · but is the screen ready?
← { ok: true }
agentfootprint
3 · Try 2: AI clicks
From the page itself
WebMCP: the page offers
its own tools.
your-app.com/orders
Service ▾Last 7 days ▾Apply
① tools exist per page · navigate, the list changes
→ filter_orders({ service: "Checkout", range: "Tue 14–15" })
③ it returned · but is the screen ready?
← { ok: true }
agentfootprint
3 · Try 2: AI clicks
Side by side
Neither one knows
when it’s done.
Computer use · browser useWebMCP
① Where do I go?explores, with no mapstill navigates · tools change per page
② What can I do here?reads the screen · clickscalls the page’s tools ✓
③ Did it work?waits, looks again, guesses ✗it returned · the screen? ✗
The hard question is the third: when is an action done?
agentfootprint
3 · Try 2: AI clicks
Side by side
Neither one knows
when it’s done.
Computer use · browser useWebMCP
① Where do I go?explores, with no mapstill navigates · tools change per page
② What can I do here?reads the screen · clickscalls the page’s tools ✓
③ Did it work?waits, looks again, guesses ✗it returned · the screen? ✗
The hard question is the third: when is an action done?
agentfootprint
3 · Try 2: AI clicks
Side by side
Neither one knows
when it’s done.
Computer use · browser useWebMCP
① Where do I go?explores, with no mapstill navigates · tools change per page
② What can I do here?reads the screen · clickscalls the page’s tools ✓
③ Did it work?waits, looks again, guesses ✗it returned · the screen? ✗
The hard question is the third: when is an action done?
agentfootprint
3 · Try 2: AI clicks
Side by side
Neither one knows
when it’s done.
Computer use · browser useWebMCP
① Where do I go?explores, with no mapstill navigates · tools change per page
② What can I do here?reads the screen · clickscalls the page’s tools ✓
③ Did it work?waits, looks again, guesses ✗it returned · the screen? ✗
The hard question is the third: when is an action done?
agentfootprint
3 · Try 2: AI clicks
Side by side
Neither one knows
when it’s done.
Computer use · browser useWebMCP
① Where do I go?explores, with no mapstill navigates · tools change per page
② What can I do here?reads the screen · clickscalls the page’s tools ✓
③ Did it work?waits, looks again, guesses ✗it returned · the screen? ✗
The hard question is the third: when is an action done?
agentfootprint
3 · Try 2: AI clicks
Why it matters
If it can’t tell it’s done…
Wrong targetacts on what the screen showed a moment ago
Retry loopsnot sure it worked, so it does it again
Slowwaits, then looks again, on every step
Costlyevery check is another model call
And when it failsno record of what happened, or why
Slower, costlier, and less certain.
agentfootprint
3 · Try 2: AI clicks
Why it matters
If it can’t tell it’s done…
Wrong targetacts on what the screen showed a moment ago
Retry loopsnot sure it worked, so it does it again
Slowwaits, then looks again, on every step
Costlyevery check is another model call
And when it failsno record of what happened, or why
Slower, costlier, and less certain.
agentfootprint
3 · Try 2: AI clicks
Why it matters
If it can’t tell it’s done…
Wrong targetacts on what the screen showed a moment ago
Retry loopsnot sure it worked, so it does it again
Slowwaits, then looks again, on every step
Costlyevery check is another model call
And when it failsno record of what happened, or why
Slower, costlier, and less certain.
agentfootprint
3 · Try 2: AI clicks
Why it matters
If it can’t tell it’s done…
Wrong targetacts on what the screen showed a moment ago
Retry loopsnot sure it worked, so it does it again
Slowwaits, then looks again, on every step
Costlyevery check is another model call
And when it failsno record of what happened, or why
Slower, costlier, and less certain.
agentfootprint
3 · Try 2: AI clicks
Why it matters
If it can’t tell it’s done…
Wrong targetacts on what the screen showed a moment ago
Retry loopsnot sure it worked, so it does it again
Slowwaits, then looks again, on every step
Costlyevery check is another model call
And when it failsno record of what happened, or why
Slower, costlier, and less certain.
agentfootprint
3 · Try 2: AI clicks
Why it matters
If it can’t tell it’s done…
Wrong targetacts on what the screen showed a moment ago
Retry loopsnot sure it worked, so it does it again
Slowwaits, then looks again, on every step
Costlyevery check is another model call
And when it failsno record of what happened, or why
Slower, costlier, and less certain.
agentfootprint
3 · Try 2: AI clicks
Why it matters
If it can’t tell it’s done…
Wrong targetacts on what the screen showed a moment ago
Retry loopsnot sure it worked, so it does it again
Slowwaits, then looks again, on every step
Costlyevery check is another model call
And when it failsno record of what happened, or why
Slower, costlier, and less certain.
agentfootprint
3 · Try 2: AI clicks
Where we are
Three ways to act.
Three problems.
① Backend tools
The LLM calls your backend directly, outside the app’s workflow. Every call needs its own permission.
secure? supervised?
② Generative UI
It rebuilds UI the app already has, inside the chat. Each redraw loses what you typed, and your scroll.
costly · a second UI
③ Acting in the app
Computer use and WebMCP act in the app. But neither knows when an action is done, or why it failed.
reliable enough for work?
How do we address all three?
agentfootprint
3 · Try 2: AI clicks
Where we are
Three ways to act.
Three problems.
① Backend tools
The LLM calls your backend directly, outside the app’s workflow. Every call needs its own permission.
secure? supervised?
② Generative UI
It rebuilds UI the app already has, inside the chat. Each redraw loses what you typed, and your scroll.
costly · a second UI
③ Acting in the app
Computer use and WebMCP act in the app. But neither knows when an action is done, or why it failed.
reliable enough for work?
How do we address all three?
agentfootprint
3 · Try 2: AI clicks
Where we are
Three ways to act.
Three problems.
① Backend tools
The LLM calls your backend directly, outside the app’s workflow. Every call needs its own permission.
secure? supervised?
② Generative UI
It rebuilds UI the app already has, inside the chat. Each redraw loses what you typed, and your scroll.
costly · a second UI
③ Acting in the app
Computer use and WebMCP act in the app. But neither knows when an action is done, or why it failed.
reliable enough for work?
How do we address all three?
agentfootprint
3 · Try 2: AI clicks
Where we are
Three ways to act.
Three problems.
① Backend tools
The LLM calls your backend directly, outside the app’s workflow. Every call needs its own permission.
secure? supervised?
② Generative UI
It rebuilds UI the app already has, inside the chat. Each redraw loses what you typed, and your scroll.
costly · a second UI
③ Acting in the app
Computer use and WebMCP act in the app. But neither knows when an action is done, or why it failed.
reliable enough for work?
How do we address all three?
agentfootprint
3 · Try 2: AI clicks
Where we are
Three ways to act.
Three problems.
① Backend tools
The LLM calls your backend directly, outside the app’s workflow. Every call needs its own permission.
secure? supervised?
② Generative UI
It rebuilds UI the app already has, inside the chat. Each redraw loses what you typed, and your scroll.
costly · a second UI
③ Acting in the app
Computer use and WebMCP act in the app. But neither knows when an action is done, or why it failed.
reliable enough for work?
How do we address all three?
agentfootprint
3 · Try 2: AI clicks
From the inside
No need to guess.
We built the app.
your-app.com/orders
Service ▾Last 7 days ▾Apply
The agent is insidein the app, logged in as the user · no outside agent
We know the actionApply → filter_orders(service, range)
But when is it done?that happens in the code that runs when you click
Most of us never look there. Let’s.
agentfootprint
3 · Try 2: AI clicks
From the inside
No need to guess.
We built the app.
your-app.com/orders
Service ▾Last 7 days ▾Apply
The agent is insidein the app, logged in as the user · no outside agent
We know the actionApply → filter_orders(service, range)
But when is it done?that happens in the code that runs when you click
Most of us never look there. Let’s.
agentfootprint
3 · Try 2: AI clicks
From the inside
No need to guess.
We built the app.
your-app.com/orders
Service ▾Last 7 days ▾Apply
The agent is insidein the app, logged in as the user · no outside agent
We know the actionApply → filter_orders(service, range)
But when is it done?that happens in the code that runs when you click
Most of us never look there. Let’s.
agentfootprint
3 · Try 2: AI clicks
From the inside
No need to guess.
We built the app.
your-app.com/orders
Service ▾Last 7 days ▾Apply
The agent is insidein the app, logged in as the user · no outside agent
We know the actionApply → filter_orders(service, range)
But when is it done?that happens in the code that runs when you click
Most of us never look there. Let’s.
agentfootprint
3 · Try 2: AI clicks
From the inside
No need to guess.
We built the app.
your-app.com/orders
Service ▾Last 7 days ▾Apply
The agent is insidein the app, logged in as the user · no outside agent
We know the actionApply → filter_orders(service, range)
But when is it done?that happens in the code that runs when you click
Most of us never look there. Let’s.
agentfootprint
04
Part 4 of 7
When is an
action complete?
To fix it, the AI must know when the app is done.
To give the agent a “done” signal, we first need to see why apps lost it.
1Two ways in2Try 1: UI in chat3Try 2: AI clicks4When is it done?5Action Binding6In practice7Two of us
agentfootprint
4 · When is it done?
In the early web, the logic lived on the server.
orders.example.com
Apply
24 rows
Server
validate
filter
totals
format
login
save
just the view
Apply
the whole form
a whole new page
loading…
6 rows ✓
around 2000 · every click loads a new page
agentfootprint
4 · When is it done?
The old page is thrown away. A new one loads.
orders.example.com
Apply
24 rows
Server
validate
filter
totals
format
login
save
just the view
Apply
the whole form
a whole new page
loading…
6 rows ✓
done = the page finished loading · the browser tells you
agentfootprint
4 · When is it done?
Then came AJAX, and jQuery.
orders.example.com
Apply
24 rows
Apply
Apply
6 rows ✓
the code that runs on click
$('#apply').click(function () {
check(form);
rows = $.ajax(url); // waits
$('#table').html(rows);
}); // returns
The browser’s loop
① click
② run it
changes the page
③ draw
around 2008 · update part of the page, without reloading
agentfootprint
4 · When is it done?
All the logic sits in one function.
orders.example.com
Apply
24 rows
Apply
Apply
6 rows ✓
the code that runs on click
$('#apply').click(function () {
check(form);
rows = $.ajax(url); // waits
$('#table').html(rows);
}); // returns
The browser’s loop
① click
② run it
changes the page
③ draw
the page update too · when it returns, it’s done
agentfootprint
4 · When is it done?
Then the logic moved into the page.
orders.example.com
Apply
24 rows
6 rows ✓
Server
validate
filter
totals
format
login
save
the view + the business logic
mostly data, now
an app that never reloads the page
Apply
only data, please
just the data · 6 rows
around 2015 · the logic moves into the browser
agentfootprint
4 · When is it done?
Only the data comes back. So when is it done?
orders.example.com
Apply
24 rows
6 rows ✓
Server
validate
filter
totals
format
login
save
the view + the business logic
mostly data, now
an app that never reloads the page
Apply
only data, please
just the data · 6 rows
no new page · no page load
agentfootprint
4 · When is it done?
Returned is not done.
orders.example.com
OrdersReports
Apply
p95 · 610 ms
24 rows
Apply
Apply
6 rows ✓
our code: runs on click
function onApply() {
update('Checkout');
} // returns · framework takes over
The app’s data
service: All
service: Checkout
rows: loading…
rows: 6
The framework
redraws the screen
→ later
The browser’s loop
① take a task
a click, or data
② run it
updates the data
③ draw
loop 1
loop 2
loop 3
loop 4
loop 5
other tasks run
⇢ request sent · waiting
⇠ the response · 6 rows
all changes at once
loading…
loop 2loop 3
③ draw: a spinner
③ draw: a badge · not ours
state: our 6 rows · not shown
③ draw: our 6 rows ✓
③ draw: a live chart updates · not ours
Reports 3
p95 · 598 ms ↻
loop 1a spinnerours
loop 2a badgenot ours
loop 3chart updatesnot ours
loop 4data arrivesnot shown yet
loop 5our 6 rowsours ✓
Outside agent
click(Apply) · loop 1
⏳ checks until it’s sure · right, but slow
📷 +1 model call
📷 +1 model call
📷 +1 model call
📷 +1 model call
📷 +1 model call
1 · Filter
2 · Select
3 · Approve
starts in loop 2 · on the old rows · wrong row ✗
loop 1select
📷 +1 call
loop 2not ours
📷 +1 call
loop 3not ours
📷 +1 call
loop 4data
📷 +1 call
loop 5our row ✓
📷 +1 call
loop 1approve
📷 +1 call
loop 2not ours
📷 +1 call
loop 3not ours
📷 +1 call
loop 4data
📷 +1 call
loop 5approved ✓
📷 +1 call
model calls 5
model calls 10
model calls 15
done ✓ 1 call
done ✓ 1 call
done ✓ 1 call
model calls 153
our code: loop 1 · our rows: loop 5 · other changes in between
agentfootprint
4 · When is it done?
Three changes on screen. Which one came from the click?
orders.example.com
OrdersReports
Apply
p95 · 610 ms
24 rows
Apply
Apply
6 rows ✓
our code: runs on click
function onApply() {
update('Checkout');
} // returns · framework takes over
The app’s data
service: All
service: Checkout
rows: loading…
rows: 6
The framework
redraws the screen
→ later
The browser’s loop
① take a task
a click, or data
② run it
updates the data
③ draw
loop 1
loop 2
loop 3
loop 4
loop 5
other tasks run
⇢ request sent · waiting
⇠ the response · 6 rows
all changes at once
loading…
loop 2loop 3
③ draw: a spinner
③ draw: a badge · not ours
state: our 6 rows · not shown
③ draw: our 6 rows ✓
③ draw: a live chart updates · not ours
Reports 3
p95 · 598 ms ↻
loop 1a spinnerours
loop 2a badgenot ours
loop 3chart updatesnot ours
loop 4data arrivesnot shown yet
loop 5our 6 rowsours ✓
Outside agent
click(Apply) · loop 1
⏳ checks until it’s sure · right, but slow
📷 +1 model call
📷 +1 model call
📷 +1 model call
📷 +1 model call
📷 +1 model call
1 · Filter
2 · Select
3 · Approve
starts in loop 2 · on the old rows · wrong row ✗
loop 1select
📷 +1 call
loop 2not ours
📷 +1 call
loop 3not ours
📷 +1 call
loop 4data
📷 +1 call
loop 5our row ✓
📷 +1 call
loop 1approve
📷 +1 call
loop 2not ours
📷 +1 call
loop 3not ours
📷 +1 call
loop 4data
📷 +1 call
loop 5approved ✓
📷 +1 call
model calls 5
model calls 10
model calls 15
done ✓ 1 call
done ✓ 1 call
done ✓ 1 call
model calls 153
agentfootprint
4 · When is it done?
It looks too early.
orders.example.com
OrdersReports
Apply
p95 · 610 ms
24 rows
Apply
Apply
6 rows ✓
our code: runs on click
function onApply() {
update('Checkout');
} // returns · framework takes over
The app’s data
service: All
service: Checkout
rows: loading…
rows: 6
The framework
redraws the screen
→ later
The browser’s loop
① take a task
a click, or data
② run it
updates the data
③ draw
loop 1
loop 2
loop 3
loop 4
loop 5
other tasks run
⇢ request sent · waiting
⇠ the response · 6 rows
all changes at once
loading…
loop 2loop 3
③ draw: a spinner
③ draw: a badge · not ours
state: our 6 rows · not shown
③ draw: our 6 rows ✓
③ draw: a live chart updates · not ours
Reports 3
p95 · 598 ms ↻
loop 1a spinnerours
loop 2a badgenot ours
loop 3chart updatesnot ours
loop 4data arrivesnot shown yet
loop 5our 6 rowsours ✓
Outside agent
click(Apply) · loop 1
⏳ checks until it’s sure · right, but slow
📷 +1 model call
📷 +1 model call
📷 +1 model call
📷 +1 model call
📷 +1 model call
1 · Filter
2 · Select
3 · Approve
starts in loop 2 · on the old rows · wrong row ✗
loop 1select
📷 +1 call
loop 2not ours
📷 +1 call
loop 3not ours
📷 +1 call
loop 4data
📷 +1 call
loop 5our row ✓
📷 +1 call
loop 1approve
📷 +1 call
loop 2not ours
📷 +1 call
loop 3not ours
📷 +1 call
loop 4data
📷 +1 call
loop 5approved ✓
📷 +1 call
model calls 5
model calls 10
model calls 15
done ✓ 1 call
done ✓ 1 call
done ✓ 1 call
model calls 153
it reads a change that wasn’t its click
agentfootprint
4 · When is it done?
Or it waits, to be safe.
orders.example.com
OrdersReports
Apply
p95 · 610 ms
24 rows
Apply
Apply
6 rows ✓
our code: runs on click
function onApply() {
update('Checkout');
} // returns · framework takes over
The app’s data
service: All
service: Checkout
rows: loading…
rows: 6
The framework
redraws the screen
→ later
The browser’s loop
① take a task
a click, or data
② run it
updates the data
③ draw
loop 1
loop 2
loop 3
loop 4
loop 5
other tasks run
⇢ request sent · waiting
⇠ the response · 6 rows
all changes at once
loading…
loop 2loop 3
③ draw: a spinner
③ draw: a badge · not ours
state: our 6 rows · not shown
③ draw: our 6 rows ✓
③ draw: a live chart updates · not ours
Reports 3
p95 · 598 ms ↻
loop 1a spinnerours
loop 2a badgenot ours
loop 3chart updatesnot ours
loop 4data arrivesnot shown yet
loop 5our 6 rowsours ✓
Outside agent
click(Apply) · loop 1
⏳ checks until it’s sure · right, but slow
📷 +1 model call
📷 +1 model call
📷 +1 model call
📷 +1 model call
📷 +1 model call
1 · Filter
2 · Select
3 · Approve
starts in loop 2 · on the old rows · wrong row ✗
loop 1select
📷 +1 call
loop 2not ours
📷 +1 call
loop 3not ours
📷 +1 call
loop 4data
📷 +1 call
loop 5our row ✓
📷 +1 call
loop 1approve
📷 +1 call
loop 2not ours
📷 +1 call
loop 3not ours
📷 +1 call
loop 4data
📷 +1 call
loop 5approved ✓
📷 +1 call
model calls 5
model calls 10
model calls 15
done ✓ 1 call
done ✓ 1 call
done ✓ 1 call
model calls 153
every check: a screenshot + a model call
agentfootprint
4 · When is it done?
So it waits at every step. Many model calls.
orders.example.com
OrdersReports
Apply
p95 · 610 ms
24 rows
Apply
Apply
6 rows ✓
our code: runs on click
function onApply() {
update('Checkout');
} // returns · framework takes over
The app’s data
service: All
service: Checkout
rows: loading…
rows: 6
The framework
redraws the screen
→ later
The browser’s loop
① take a task
a click, or data
② run it
updates the data
③ draw
loop 1
loop 2
loop 3
loop 4
loop 5
other tasks run
⇢ request sent · waiting
⇠ the response · 6 rows
all changes at once
loading…
loop 2loop 3
③ draw: a spinner
③ draw: a badge · not ours
state: our 6 rows · not shown
③ draw: our 6 rows ✓
③ draw: a live chart updates · not ours
Reports 3
p95 · 598 ms ↻
loop 1a spinnerours
loop 2a badgenot ours
loop 3chart updatesnot ours
loop 4data arrivesnot shown yet
loop 5our 6 rowsours ✓
Outside agent
click(Apply) · loop 1
⏳ checks until it’s sure · right, but slow
📷 +1 model call
📷 +1 model call
📷 +1 model call
📷 +1 model call
📷 +1 model call
1 · Filter
2 · Select
3 · Approve
starts in loop 2 · on the old rows · wrong row ✗
loop 1select
📷 +1 call
loop 2not ours
📷 +1 call
loop 3not ours
📷 +1 call
loop 4data
📷 +1 call
loop 5our row ✓
📷 +1 call
loop 1approve
📷 +1 call
loop 2not ours
📷 +1 call
loop 3not ours
📷 +1 call
loop 4data
📷 +1 call
loop 5approved ✓
📷 +1 call
model calls 5
model calls 10
model calls 15
done ✓ 1 call
done ✓ 1 call
done ✓ 1 call
model calls 153
one early look breaks the chain
agentfootprint
4 · When is it done?
It’s the best they can do from outside.
orders.example.com
OrdersReports
Apply
p95 · 610 ms
24 rows
Apply
Apply
6 rows ✓
our code: runs on click
function onApply() {
update('Checkout');
} // returns · framework takes over
The app’s data
service: All
service: Checkout
rows: loading…
rows: 6
The framework
redraws the screen
→ later
The browser’s loop
① take a task
a click, or data
② run it
updates the data
③ draw
loop 1
loop 2
loop 3
loop 4
loop 5
other tasks run
⇢ request sent · waiting
⇠ the response · 6 rows
all changes at once
loading…
loop 2loop 3
③ draw: a spinner
③ draw: a badge · not ours
state: our 6 rows · not shown
③ draw: our 6 rows ✓
③ draw: a live chart updates · not ours
Reports 3
p95 · 598 ms ↻
loop 1a spinnerours
loop 2a badgenot ours
loop 3chart updatesnot ours
loop 4data arrivesnot shown yet
loop 5our 6 rowsours ✓
Outside agent
click(Apply) · loop 1
⏳ checks until it’s sure · right, but slow
📷 +1 model call
📷 +1 model call
📷 +1 model call
📷 +1 model call
📷 +1 model call
1 · Filter
2 · Select
3 · Approve
starts in loop 2 · on the old rows · wrong row ✗
loop 1select
📷 +1 call
loop 2not ours
📷 +1 call
loop 3not ours
📷 +1 call
loop 4data
📷 +1 call
loop 5our row ✓
📷 +1 call
loop 1approve
📷 +1 call
loop 2not ours
📷 +1 call
loop 3not ours
📷 +1 call
loop 4data
📷 +1 call
loop 5approved ✓
📷 +1 call
model calls 5
model calls 10
model calls 15
done ✓ 1 call
done ✓ 1 call
done ✓ 1 call
model calls 153
inside your app, the app knows: started · data changed · done
agentfootprint
4 · When is it done?
What if the app told us exactly when it’s done?
orders.example.com
OrdersReports
Apply
p95 · 610 ms
24 rows
Apply
Apply
6 rows ✓
our code: runs on click
function onApply() {
update('Checkout');
} // returns · framework takes over
The app’s data
service: All
service: Checkout
rows: loading…
rows: 6
The framework
redraws the screen
→ later
The browser’s loop
① take a task
a click, or data
② run it
updates the data
③ draw
loop 1
loop 2
loop 3
loop 4
loop 5
other tasks run
⇢ request sent · waiting
⇠ the response · 6 rows
all changes at once
loading…
loop 2loop 3
③ draw: a spinner
③ draw: a badge · not ours
state: our 6 rows · not shown
③ draw: our 6 rows ✓
③ draw: a live chart updates · not ours
Reports 3
p95 · 598 ms ↻
loop 1a spinnerours
loop 2a badgenot ours
loop 3chart updatesnot ours
loop 4data arrivesnot shown yet
loop 5our 6 rowsours ✓
Outside agent
click(Apply) · loop 1
⏳ checks until it’s sure · right, but slow
📷 +1 model call
📷 +1 model call
📷 +1 model call
📷 +1 model call
📷 +1 model call
1 · Filter
2 · Select
3 · Approve
starts in loop 2 · on the old rows · wrong row ✗
loop 1select
📷 +1 call
loop 2not ours
📷 +1 call
loop 3not ours
📷 +1 call
loop 4data
📷 +1 call
loop 5our row ✓
📷 +1 call
loop 1approve
📷 +1 call
loop 2not ours
📷 +1 call
loop 3not ours
📷 +1 call
loop 4data
📷 +1 call
loop 5approved ✓
📷 +1 call
model calls 5
model calls 10
model calls 15
done ✓ 1 call
done ✓ 1 call
done ✓ 1 call
model calls 153
agentfootprint
4 · When is it done?
No waits. No guesses. 15 → 3.
orders.example.com
OrdersReports
Apply
p95 · 610 ms
24 rows
Apply
Apply
6 rows ✓
our code: runs on click
function onApply() {
update('Checkout');
} // returns · framework takes over
The app’s data
service: All
service: Checkout
rows: loading…
rows: 6
The framework
redraws the screen
→ later
The browser’s loop
① take a task
a click, or data
② run it
updates the data
③ draw
loop 1
loop 2
loop 3
loop 4
loop 5
other tasks run
⇢ request sent · waiting
⇠ the response · 6 rows
all changes at once
loading…
loop 2loop 3
③ draw: a spinner
③ draw: a badge · not ours
state: our 6 rows · not shown
③ draw: our 6 rows ✓
③ draw: a live chart updates · not ours
Reports 3
p95 · 598 ms ↻
loop 1a spinnerours
loop 2a badgenot ours
loop 3chart updatesnot ours
loop 4data arrivesnot shown yet
loop 5our 6 rowsours ✓
Outside agent
click(Apply) · loop 1
⏳ checks until it’s sure · right, but slow
📷 +1 model call
📷 +1 model call
📷 +1 model call
📷 +1 model call
📷 +1 model call
1 · Filter
2 · Select
3 · Approve
starts in loop 2 · on the old rows · wrong row ✗
loop 1select
📷 +1 call
loop 2not ours
📷 +1 call
loop 3not ours
📷 +1 call
loop 4data
📷 +1 call
loop 5our row ✓
📷 +1 call
loop 1approve
📷 +1 call
loop 2not ours
📷 +1 call
loop 3not ours
📷 +1 call
loop 4data
📷 +1 call
loop 5approved ✓
📷 +1 call
model calls 5
model calls 10
model calls 15
done ✓ 1 call
done ✓ 1 call
done ✓ 1 call
model calls 153
one call per step · each step starts the moment the last one is done
agentfootprint
05
Part 5 of 7
Action Binding
The app tells the AI what it can do, and when it’s done.
1Two ways in2Try 1: UI in chat3Try 2: AI clicks4When is it done?5Action Binding6In practice7Two of us
agentfootprint
5 · Action Binding
An agent inside the app.
orders.example.com
OrdersReports
Apply
p95 · 610 ms
24 rows
6 rows ✓
loading…
Inside agent
orders.filter('Checkout')
a bound action
The app reports
●startedloop 1
●changed · the data updatedloop 4
●done · the screen is readyloop 5
reports
✓ result: done · 6 rows on screen
continue
inspect
ask
report
1 · Filter
on done →
2 · Select
on done →
3 · Approve
model calls: 3
Activity log · you and the agent
you opened Orders 14:02:00
agent orders.filter · started 14:02:01
changed · done ✓ 420 ms
you selected 14:00–15:00 14:02:04
agent orders.approve · done ✓ 14:02:06
// the LLM reads this, not only the chat
▸Reliability
·Audit
·Security
·Consent
·Control
·How you use it
same login · same screen · same rules as you
it runs in your app, not in a separate browser
agentfootprint
5 · Action Binding
It doesn’t click Apply. It calls the action.
orders.example.com
OrdersReports
Apply
p95 · 610 ms
24 rows
6 rows ✓
loading…
Inside agent
orders.filter('Checkout')
a bound action
The app reports
●startedloop 1
●changed · the data updatedloop 4
●done · the screen is readyloop 5
reports
✓ result: done · 6 rows on screen
continue
inspect
ask
report
1 · Filter
on done →
2 · Select
on done →
3 · Approve
model calls: 3
Activity log · you and the agent
you opened Orders 14:02:00
agent orders.filter · started 14:02:01
changed · done ✓ 420 ms
you selected 14:00–15:00 14:02:04
agent orders.approve · done ✓ 14:02:06
// the LLM reads this, not only the chat
▸Reliability
·Audit
·Security
·Consent
·Control
·How you use it
a bound action = a named action the app offers the agent
agentfootprint
5 · Action Binding
The app reports each stage.
orders.example.com
OrdersReports
Apply
p95 · 610 ms
24 rows
6 rows ✓
loading…
Inside agent
orders.filter('Checkout')
a bound action
The app reports
●startedloop 1
●changed · the data updatedloop 4
●done · the screen is readyloop 5
reports
✓ result: done · 6 rows on screen
continue
inspect
ask
report
1 · Filter
on done →
2 · Select
on done →
3 · Approve
model calls: 3
Activity log · you and the agent
you opened Orders 14:02:00
agent orders.filter · started 14:02:01
changed · done ✓ 420 ms
you selected 14:00–15:00 14:02:04
agent orders.approve · done ✓ 14:02:06
// the LLM reads this, not only the chat
▸Reliability
·Audit
·Security
·Consent
·Control
·How you use it
started · changed · done · no timers, no screenshots
agentfootprint
5 · Action Binding
The next action fires the moment it’s done.
orders.example.com
OrdersReports
Apply
p95 · 610 ms
24 rows
6 rows ✓
loading…
Inside agent
orders.filter('Checkout')
a bound action
The app reports
●startedloop 1
●changed · the data updatedloop 4
●done · the screen is readyloop 5
reports
✓ result: done · 6 rows on screen
continue
inspect
ask
report
1 · Filter
on done →
2 · Select
on done →
3 · Approve
model calls: 3
Activity log · you and the agent
you opened Orders 14:02:00
agent orders.filter · started 14:02:01
changed · done ✓ 420 ms
you selected 14:00–15:00 14:02:04
agent orders.approve · done ✓ 14:02:06
// the LLM reads this, not only the chat
▸Reliability
·Audit
·Security
·Consent
·Control
·How you use it
agentfootprint
5 · Action Binding
It gets the result, not a screenshot.
orders.example.com
OrdersReports
Apply
p95 · 610 ms
24 rows
6 rows ✓
loading…
Inside agent
orders.filter('Checkout')
a bound action
The app reports
●startedloop 1
●changed · the data updatedloop 4
●done · the screen is readyloop 5
reports
✓ result: done · 6 rows on screen
continue
inspect
ask
report
1 · Filter
on done →
2 · Select
on done →
3 · Approve
model calls: 3
Activity log · you and the agent
you opened Orders 14:02:00
agent orders.filter · started 14:02:01
changed · done ✓ 420 ms
you selected 14:00–15:00 14:02:04
agent orders.approve · done ✓ 14:02:06
// the LLM reads this, not only the chat
▸Reliability
·Audit
·Security
·Consent
·Control
·How you use it
agentfootprint
5 · Action Binding
Each report is written down.
orders.example.com
OrdersReports
Apply
p95 · 610 ms
24 rows
6 rows ✓
loading…
Inside agent
orders.filter('Checkout')
a bound action
The app reports
●startedloop 1
●changed · the data updatedloop 4
●done · the screen is readyloop 5
reports
✓ result: done · 6 rows on screen
continue
inspect
ask
report
1 · Filter
on done →
2 · Select
on done →
3 · Approve
model calls: 3
Activity log · you and the agent
you opened Orders 14:02:00
agent orders.filter · started 14:02:01
changed · done ✓ 420 ms
you selected 14:00–15:00 14:02:04
agent orders.approve · done ✓ 14:02:06
// the LLM reads this, not only the chat
✓Reliability
▸Audit
·Security
·Consent
·Control
·How you use it
the app logs every report
agentfootprint
5 · Action Binding
Into one ledger: the activity log.
orders.example.com
OrdersReports
Apply
p95 · 610 ms
24 rows
6 rows ✓
loading…
Inside agent
orders.filter('Checkout')
a bound action
The app reports
●startedloop 1
●changed · the data updatedloop 4
●done · the screen is readyloop 5
reports
✓ result: done · 6 rows on screen
continue
inspect
ask
report
1 · Filter
on done →
2 · Select
on done →
3 · Approve
model calls: 3
Activity log · you and the agent
you opened Orders 14:02:00
agent orders.filter · started 14:02:01
changed · done ✓ 420 ms
you selected 14:00–15:00 14:02:04
agent orders.approve · done ✓ 14:02:06
// the LLM reads this, not only the chat
✓Reliability
▸Audit
·Security
·Consent
·Control
·How you use it
agentfootprint
5 · Action Binding
Every action is logged, from both users.
orders.example.com
OrdersReports
Apply
p95 · 610 ms
24 rows
6 rows ✓
loading…
Inside agent
orders.filter('Checkout')
a bound action
The app reports
●startedloop 1
●changed · the data updatedloop 4
●done · the screen is readyloop 5
reports
✓ result: done · 6 rows on screen
continue
inspect
ask
report
1 · Filter
on done →
2 · Select
on done →
3 · Approve
model calls: 3
Activity log · you and the agent
you opened Orders 14:02:00
agent orders.filter · started 14:02:01
changed · done ✓ 420 ms
you selected 14:00–15:00 14:02:04
agent orders.approve · done ✓ 14:02:06
// the LLM reads this, not only the chat
✓Reliability
▸Audit
·Security
·Consent
·Control
·How you use it
agentfootprint
5 · Action Binding
The LLM reads the log, not only the chat.
orders.example.com
OrdersReports
Apply
p95 · 610 ms
24 rows
6 rows ✓
loading…
Inside agent
orders.filter('Checkout')
a bound action
The app reports
●startedloop 1
●changed · the data updatedloop 4
●done · the screen is readyloop 5
reports
✓ result: done · 6 rows on screen
continue
inspect
ask
report
1 · Filter
on done →
2 · Select
on done →
3 · Approve
model calls: 3
Activity log · you and the agent
you opened Orders 14:02:00
agent orders.filter · started 14:02:01
changed · done ✓ 420 ms
you selected 14:00–15:00 14:02:04
agent orders.approve · done ✓ 14:02:06
// the LLM reads this, not only the chat
✓Reliability
▸Audit
·Security
·Consent
·Control
·How you use it
what really happened, not only what was said
agentfootprint
5 · Action Binding
Why not go straight to the backend?
Straight to the backend
✓ allowed?
✗ offered right now?
✗ in order?
✗ confirmed by the user?
✗ + extra rules just for the agent = a second set of rules
Through the app
✓ allowed?
✓ offered right now?
✓ in order?
✓ confirmed by the user?
✓ one set of rules · you see it happen, and can step in
✓Reliability
✓Audit
▸Security
·Consent
·Control
·How you use it
agentfootprint
5 · Action Binding
The backend still checks what’s allowed.
Straight to the backend
✓ allowed?
✗ offered right now?
✗ in order?
✗ confirmed by the user?
✗ + extra rules just for the agent = a second set of rules
Through the app
✓ allowed?
✓ offered right now?
✓ in order?
✓ confirmed by the user?
✓ one set of rules · you see it happen, and can step in
✓Reliability
✓Audit
▸Security
·Consent
·Control
·How you use it
the screen’s rules are gone
agentfootprint
5 · Action Binding
Getting them back means a second set of rules.
Straight to the backend
✓ allowed?
✗ offered right now?
✗ in order?
✗ confirmed by the user?
✗ + extra rules just for the agent = a second set of rules
Through the app
✓ allowed?
✓ offered right now?
✓ in order?
✓ confirmed by the user?
✓ one set of rules · you see it happen, and can step in
✓Reliability
✓Audit
▸Security
·Consent
·Control
·How you use it
agentfootprint
5 · Action Binding
Through the app: the same bound actions.
Straight to the backend
✓ allowed?
✗ offered right now?
✗ in order?
✗ confirmed by the user?
✗ + extra rules just for the agent = a second set of rules
Through the app
✓ allowed?
✓ offered right now?
✓ in order?
✓ confirmed by the user?
✓ one set of rules · you see it happen, and can step in
✓Reliability
✓Audit
▸Security
·Consent
·Control
·How you use it
agentfootprint
5 · Action Binding
One set of rules.
Straight to the backend
✓ allowed?
✗ offered right now?
✗ in order?
✗ confirmed by the user?
✗ + extra rules just for the agent = a second set of rules
Through the app
✓ allowed?
✓ offered right now?
✓ in order?
✓ confirmed by the user?
✓ one set of rules · you see it happen, and can step in
✓Reliability
✓Audit
▸Security
·Consent
·Control
·How you use it
agentfootprint
5 · Action Binding
The backend decides what’s allowed. The app, what’s offered.
Straight to the backend
✓ allowed?
✗ offered right now?
✗ in order?
✗ confirmed by the user?
✗ + extra rules just for the agent = a second set of rules
Through the app
✓ allowed?
✓ offered right now?
✓ in order?
✓ confirmed by the user?
✓ one set of rules · you see it happen, and can step in
✓Reliability
✓Audit
▸Security
·Consent
·Control
·How you use it
agentfootprint
5 · Action Binding
The happy path: a workflow you already built.
User
Agent
just another user
Your app
a workflow you built
✓ same checks as you
“Refund every Checkout order hit by Tuesday’s deploy.”
✗ no screen · no bound action
The agent asks first
Refund 42 Checkout orders
why: slowed by Tuesday’s deploy
Add context
Approve
Cancel
only orders over $50 → 18
Activity log
you approved · 18 orders
agent orders.refund · done ✓
✓Reliability
✓Audit
✓Security
▸Consent
·Control
·How you use it
already built · already secured
agentfootprint
5 · Action Binding
But what about a request no workflow covers?
User
Agent
just another user
Your app
a workflow you built
✓ same checks as you
“Refund every Checkout order hit by Tuesday’s deploy.”
✗ no screen · no bound action
The agent asks first
Refund 42 Checkout orders
why: slowed by Tuesday’s deploy
Add context
Approve
Cancel
only orders over $50 → 18
Activity log
you approved · 18 orders
agent orders.refund · done ✓
✓Reliability
✓Audit
✓Security
▸Consent
·Control
·How you use it
agentfootprint
5 · Action Binding
The agent asks the human first.
User
Agent
just another user
Your app
a workflow you built
✓ same checks as you
“Refund every Checkout order hit by Tuesday’s deploy.”
✗ no screen · no bound action
The agent asks first
Refund 42 Checkout orders
why: slowed by Tuesday’s deploy
Add context
Approve
Cancel
only orders over $50 → 18
Activity log
you approved · 18 orders
agent orders.refund · done ✓
✓Reliability
✓Audit
✓Security
▸Consent
·Control
·How you use it
human in the loop
agentfootprint
5 · Action Binding
The user adds context, and approves.
User
Agent
just another user
Your app
a workflow you built
✓ same checks as you
“Refund every Checkout order hit by Tuesday’s deploy.”
✗ no screen · no bound action
The agent asks first
Refund 42 Checkout orders
why: slowed by Tuesday’s deploy
Add context
Approve
Cancel
only orders over $50 → 18
Activity log
you approved · 18 orders
agent orders.refund · done ✓
✓Reliability
✓Audit
✓Security
▸Consent
·Control
·How you use it
agentfootprint
5 · Action Binding
Consent, then the same checks.
User
Agent
just another user
Your app
a workflow you built
✓ same checks as you
“Refund every Checkout order hit by Tuesday’s deploy.”
✗ no screen · no bound action
The agent asks first
Refund 42 Checkout orders
why: slowed by Tuesday’s deploy
Add context
Approve
Cancel
only orders over $50 → 18
Activity log
you approved · 18 orders
agent orders.refund · done ✓
✓Reliability
✓Audit
✓Security
▸Consent
·Control
·How you use it
your approval can’t give the agent more than you’re allowed
agentfootprint
5 · Action Binding
Not everything gets bound.
Your app’s capabilities
Filter orders
✓
Select a row
✓
Approve a refund
✓
Export CSV
✓
Delete account
Change billing
Invite an admin
Edit pricing rules
Bulk re-index
Reset passwords
View audit log
Search
Each bound action keeps
Straight to the backend
bulk and background work · no simulated clicks
✓Reliability
✓Audit
✓Security
✓Consent
▸Control
·How you use it
agentfootprint
5 · Action Binding
Developers bind what the agent should use.
Your app’s capabilities
Filter orders
✓
Select a row
✓
Approve a refund
✓
Export CSV
✓
Delete account
Change billing
Invite an admin
Edit pricing rules
Bulk re-index
Reset passwords
View audit log
Search
Each bound action keeps
Straight to the backend
bulk and background work · no simulated clicks
✓Reliability
✓Audit
✓Security
✓Consent
▸Control
·How you use it
agentfootprint
5 · Action Binding
Bound actions keep their rules.
Your app’s capabilities
Filter orders
✓
Select a row
✓
Approve a refund
✓
Export CSV
✓
Delete account
Change billing
Invite an admin
Edit pricing rules
Bulk re-index
Reset passwords
View audit log
Search
Each bound action keeps
Straight to the backend
bulk and background work · no simulated clicks
✓Reliability
✓Audit
✓Security
✓Consent
▸Control
·How you use it
agentfootprint
5 · Action Binding
Not everything goes through the screen.
Your app’s capabilities
Filter orders
✓
Select a row
✓
Approve a refund
✓
Export CSV
✓
Delete account
Change billing
Invite an admin
Edit pricing rules
Bulk re-index
Reset passwords
View audit log
Search
Each bound action keeps
Straight to the backend
bulk and background work · no simulated clicks
✓Reliability
✓Audit
✓Security
✓Consent
▸Control
·How you use it
agentfootprint
5 · Action Binding
Using it: four steps.
1DefineGive the action a name and say what it needs: “filter orders, by service”.defineAction('orders.filter')
2ConnectHook it to the screen that already does it. No new UI.connectAction(app, filter)
3OfferThe app offers it to the agent, under the same rules as you.agent.offers(filter)
4Run & confirmThe agent runs it. The app reports started · changed · done.agent.invoke(offer)
In HCI Footprint
const filter = defineAction('orders.filter', {
does, invocation, mutate, …
});
const connection = connectAction(runtime, filter, {
node, instance, input, enabled
});
const agent = runtime.forPrincipal('agent');
const [offer] = agent.offers(filter);
const invocation = agent.invoke(offer);
await invocation.whenInvoked;
connection.settle(invocation.transition, { … });
await invocation.whenEffectSettled;
Definename the action, not a click
Connecthook it to the screen that does it
Offerthe app offers it to the agent
Run · started?did the app take it? (“invoked”)performedrefusedfailed
Confirm · done?did it really work? (“settled”)verifiedrefusedabandoned
✓Reliability
✓Audit
✓Security
✓Consent
✓Control
▸How you use it
agentfootprint
5 · Action Binding
Action Binding
What it gives you.
✓Reliability
✓Audit
✓Security
✓Consent
✓Control
✓How you use it
One app. Two users. One set of rules.reliability · audit · security · consent · control
agentfootprint
06
Part 6 of 7
Chat starts it,
your screens finish it
What it looks like in practice.
1Two ways in2Try 1: UI in chat3Try 2: AI clicks4When is it done?5Action Binding6In practice7Two of us
agentfootprint
6 · In practice
Does a request that starts in chat have to stay in chat?
The answer
A request can begin in chat
without every step
happening in chat.
The answer
A request can begin in chat
without every step
happening in chat.
A conversation doesn't have to stay text.
agentfootprint
6 · In practice
Does a request that starts in chat have to stay in chat?
The answer
A request can begin in chat
without every step
happening in chat.
The answer
A request can begin in chat
without every step
happening in chat.
A conversation doesn't have to stay text.
agentfootprint
6 · In practice
One sentence, several steps.
Chat
Help me investigate this spike.
thinking…
working… step 2 of 4
✗ which step ran? what came back?
what is it about to do?
Your app · progress
step · input · status · output
Get the time range
you select it on the chart
✓ done
14:02–14:09
Get the service
you pick it in the selector
✓ done
Service B
Run the analysis
14:02–14:09 · Service B
Approve
Cancel
Show the results
your chart and table
queued
—
every step visible, in the app you already have
each step → one row in the app
four steps, one request
agentfootprint
6 · In practice
In chat, you get a spinner.
Chat
Help me investigate this spike.
thinking…
working… step 2 of 4
✗ which step ran? what came back?
what is it about to do?
Your app · progress
step · input · status · output
Get the time range
you select it on the chart
✓ done
14:02–14:09
Get the service
you pick it in the selector
✓ done
Service B
Run the analysis
14:02–14:09 · Service B
Approve
Cancel
Show the results
your chart and table
queued
—
every step visible, in the app you already have
each step → one row in the app
agentfootprint
6 · In practice
You can’t see what it did.
Chat
Help me investigate this spike.
thinking…
working… step 2 of 4
✗ which step ran? what came back?
what is it about to do?
Your app · progress
step · input · status · output
Get the time range
you select it on the chart
✓ done
14:02–14:09
Get the service
you pick it in the selector
✓ done
Service B
Run the analysis
14:02–14:09 · Service B
Approve
Cancel
Show the results
your chart and table
queued
—
every step visible, in the app you already have
each step → one row in the app
agentfootprint
6 · In practice
The app can show the first step.
Chat
Help me investigate this spike.
thinking…
working… step 2 of 4
✗ which step ran? what came back?
what is it about to do?
Your app · progress
step · input · status · output
Get the time range
you select it on the chart
✓ done
14:02–14:09
Get the service
you pick it in the selector
✓ done
Service B
Run the analysis
14:02–14:09 · Service B
Approve
Cancel
Show the results
your chart and table
queued
—
every step visible, in the app you already have
each step → one row in the app
input · status · output
agentfootprint
6 · In practice
Every step: input, status, output.
Chat
Help me investigate this spike.
thinking…
working… step 2 of 4
✗ which step ran? what came back?
what is it about to do?
Your app · progress
step · input · status · output
Get the time range
you select it on the chart
✓ done
14:02–14:09
Get the service
you pick it in the selector
✓ done
Service B
Run the analysis
14:02–14:09 · Service B
Approve
Cancel
Show the results
your chart and table
queued
—
every step visible, in the app you already have
each step → one row in the app
agentfootprint
6 · In practice
And when a step needs you, it asks right there.
Chat
Help me investigate this spike.
thinking…
working… step 2 of 4
✗ which step ran? what came back?
what is it about to do?
Your app · progress
step · input · status · output
Get the time range
you select it on the chart
✓ done
14:02–14:09
Get the service
you pick it in the selector
✓ done
Service B
Run the analysis
14:02–14:09 · Service B
Approve
Cancel
Show the results
your chart and table
queued
—
every step visible, in the app you already have
each step → one row in the app
approval, in place
agentfootprint
6 · In practice
One row for each step the assistant takes.
Chat
Help me investigate this spike.
thinking…
working… step 2 of 4
✗ which step ran? what came back?
what is it about to do?
Your app · progress
step · input · status · output
Get the time range
you select it on the chart
✓ done
14:02–14:09
Get the service
you pick it in the selector
✓ done
Service B
Run the analysis
14:02–14:09 · Service B
Approve
Cancel
Show the results
your chart and table
queued
—
every step visible, in the app you already have
each step → one row in the app
agentfootprint
6 · In practice
The same request, in the dashboard
“Help me investigate this spike.”
Results
no results yet
Help me investigate
this spike.
1
Select it on the chart, and pick a service.
Chat · the request starts in words
2
You give the input · select the time range, right on the chart
3
You give the input · pick the service, in the app’s own selector
Running the analysis with
selected range
Service B
4
The app does the work · its own analysis, the same as when you click
you see what it uses, before it runs
Deploy v2.14 rolled outService Bslowest requests +340 ms
Cache miss rate ×3Service Bfrom 14:03
DB connections maxedorders-db14:04–14:09
5
You see the results · in your current chart and table
selected resultWhat about this one?
6
Back to chat · from what you selected
agentfootprint
6 · In practice
The same request, in the dashboard
“Help me investigate this spike.”
Results
no results yet
Help me investigate
this spike.
1
Select it on the chart, and pick a service.
Chat · the request starts in words
2
You give the input · select the time range, right on the chart
3
You give the input · pick the service, in the app’s own selector
Running the analysis with
selected range
Service B
4
The app does the work · its own analysis, the same as when you click
you see what it uses, before it runs
Deploy v2.14 rolled outService Bslowest requests +340 ms
Cache miss rate ×3Service Bfrom 14:03
DB connections maxedorders-db14:04–14:09
5
You see the results · in your current chart and table
selected resultWhat about this one?
6
Back to chat · from what you selected
agentfootprint
6 · In practice
The same request, in the dashboard
“Help me investigate this spike.”
Results
no results yet
Help me investigate
this spike.
1
Select it on the chart, and pick a service.
Chat · the request starts in words
2
You give the input · select the time range, right on the chart
3
You give the input · pick the service, in the app’s own selector
Running the analysis with
selected range
Service B
4
The app does the work · its own analysis, the same as when you click
you see what it uses, before it runs
Deploy v2.14 rolled outService Bslowest requests +340 ms
Cache miss rate ×3Service Bfrom 14:03
DB connections maxedorders-db14:04–14:09
5
You see the results · in your current chart and table
selected resultWhat about this one?
6
Back to chat · from what you selected
agentfootprint
6 · In practice
The same request, in the dashboard
“Help me investigate this spike.”
Results
no results yet
Help me investigate
this spike.
1
Select it on the chart, and pick a service.
Chat · the request starts in words
2
You give the input · select the time range, right on the chart
3
You give the input · pick the service, in the app’s own selector
Running the analysis with
selected range
Service B
4
The app does the work · its own analysis, the same as when you click
you see what it uses, before it runs
Deploy v2.14 rolled outService Bslowest requests +340 ms
Cache miss rate ×3Service Bfrom 14:03
DB connections maxedorders-db14:04–14:09
5
You see the results · in your current chart and table
selected resultWhat about this one?
6
Back to chat · from what you selected
agentfootprint
6 · In practice
The same request, in the dashboard
“Help me investigate this spike.”
Results
no results yet
Help me investigate
this spike.
1
Select it on the chart, and pick a service.
Chat · the request starts in words
2
You give the input · select the time range, right on the chart
3
You give the input · pick the service, in the app’s own selector
Running the analysis with
selected range
Service B
4
The app does the work · its own analysis, the same as when you click
you see what it uses, before it runs
Deploy v2.14 rolled outService Bslowest requests +340 ms
Cache miss rate ×3Service Bfrom 14:03
DB connections maxedorders-db14:04–14:09
5
You see the results · in your current chart and table
selected resultWhat about this one?
6
Back to chat · from what you selected
agentfootprint
6 · In practice
The same request, in the dashboard
“Help me investigate this spike.”
Results
no results yet
Help me investigate
this spike.
1
Select it on the chart, and pick a service.
Chat · the request starts in words
2
You give the input · select the time range, right on the chart
3
You give the input · pick the service, in the app’s own selector
Running the analysis with
selected range
Service B
4
The app does the work · its own analysis, the same as when you click
you see what it uses, before it runs
Deploy v2.14 rolled outService Bslowest requests +340 ms
Cache miss rate ×3Service Bfrom 14:03
DB connections maxedorders-db14:04–14:09
5
You see the results · in your current chart and table
selected resultWhat about this one?
6
Back to chat · from what you selected
agentfootprint
6 · In practice
The same request, in the dashboard
“Help me investigate this spike.”
Results
no results yet
Help me investigate
this spike.
1
Select it on the chart, and pick a service.
Chat · the request starts in words
2
You give the input · select the time range, right on the chart
3
You give the input · pick the service, in the app’s own selector
Running the analysis with
selected range
Service B
4
The app does the work · its own analysis, the same as when you click
you see what it uses, before it runs
Deploy v2.14 rolled outService Bslowest requests +340 ms
Cache miss rate ×3Service Bfrom 14:03
DB connections maxedorders-db14:04–14:09
5
You see the results · in your current chart and table
selected resultWhat about this one?
6
Back to chat · from what you selected
agentfootprint
6 · In practice
Chat and UI
Chat for flexibility.
UI for precision.
Chat
For what no screen was built for.
Existing UI
Exact picks, clear forms, charts you can read.
Complementary parts of the same application.
agentfootprint
6 · In practice
Chat and UI
Chat for flexibility.
UI for precision.
Chat
For what no screen was built for.
Existing UI
Exact picks, clear forms, charts you can read.
Complementary parts of the same application.
agentfootprint
6 · In practice
Chat and UI
Chat for flexibility.
UI for precision.
Chat
For what no screen was built for.
Existing UI
Exact picks, clear forms, charts you can read.
Complementary parts of the same application.
agentfootprint
6 · In practice
Chat and UI
Chat for flexibility.
UI for precision.
Chat
For what no screen was built for.
Existing UI
Exact picks, clear forms, charts you can read.
Complementary parts of the same application.
agentfootprint
07
Part 7 of 7
One application,
two of us
What does each side get?
1Two ways in2Try 1: UI in chat3Try 2: AI clicks4When is it done?5Action Binding6In practice7Two of us
agentfootprint
7 · Two of us
For developers
No rebuild around chat.
Help me investigate this spike.
It handles the requests no screen was built for.
one permission model
every action logged
knows when it’s done · fewer model calls
agentfootprint
7 · Two of us
For developers
No rebuild around chat.
Help me investigate this spike.
It handles the requests no screen was built for.
one permission model
every action logged
knows when it’s done · fewer model calls
agentfootprint
7 · Two of us
For developers
No rebuild around chat.
Help me investigate this spike.
It handles the requests no screen was built for.
one permission model
every action logged
knows when it’s done · fewer model calls
agentfootprint
7 · Two of us
For developers
No rebuild around chat.
Help me investigate this spike.
It handles the requests no screen was built for.
one permission model
every action logged
knows when it’s done · fewer model calls
agentfootprint
7 · Two of us
For developers
No rebuild around chat.
Help me investigate this spike.
It handles the requests no screen was built for.
one permission model
every action logged
knows when it’s done · fewer model calls
agentfootprint
7 · Two of us
For users
One shared workspace.
Express an intention
in chat
Give precise input
in the UI
Watch the app respond
in its own views
Inspect the outcome
in context
Continue
whichever way fits next
And you stay in charge: you see every step, and approve what matters.
agentfootprint
7 · Two of us
For users
One shared workspace.
Express an intention
in chat
Give precise input
in the UI
Watch the app respond
in its own views
Inspect the outcome
in context
Continue
whichever way fits next
And you stay in charge: you see every step, and approve what matters.
agentfootprint
7 · Two of us
For users
One shared workspace.
Express an intention
in chat
Give precise input
in the UI
Watch the app respond
in its own views
Inspect the outcome
in context
Continue
whichever way fits next
And you stay in charge: you see every step, and approve what matters.
agentfootprint
The central idea
Chat expands what users can ask.
The existing UI gives structure to input, actions and results.
HCI Footprint, our library, connects them.
The agent never calls your backend directly:
you see it, approve it, and it’s logged.
Libraries · LinkedIn · writing
and how this talk was made
Thank you.
Questions, and pushback, welcome.
agentfootprint
The central idea
Chat expands what users can ask.
The existing UI gives structure to input, actions and results.
HCI Footprint, our library, connects them.
The agent never calls your backend directly:
you see it, approve it, and it’s logged.
Libraries · LinkedIn · writing
and how this talk was made
Thank you.
Questions, and pushback, welcome.
agentfootprint
The central idea
Chat expands what users can ask.
The existing UI gives structure to input, actions and results.
HCI Footprint, our library, connects them.
The agent never calls your backend directly:
you see it, approve it, and it’s logged.
Libraries · LinkedIn · writing
and how this talk was made
Thank you.
Questions, and pushback, welcome.
agentfootprint
The central idea
Chat expands what users can ask.
The existing UI gives structure to input, actions and results.
HCI Footprint, our library, connects them.
The agent never calls your backend directly:
you see it, approve it, and it’s logged.
Libraries · LinkedIn · writing
and how this talk was made
Thank you.
Questions, and pushback, welcome.
agentfootprint
The central idea
Chat expands what users can ask.
The existing UI gives structure to input, actions and results.
HCI Footprint, our library, connects them.
The agent never calls your backend directly:
you see it, approve it, and it’s logged.
Libraries · LinkedIn · writing
and how this talk was made
Thank you.
Questions, and pushback, welcome.
agentfootprint
Define the action, not a click.
1DefineGive the action a name and say what it needs: “filter orders, by service”.defineAction('orders.filter')
2ConnectHook it to the screen that already does it. No new UI.connectAction(app, filter)
3OfferThe app offers it to the agent, under the same rules as you.agent.offers(filter)
4Run & confirmThe agent runs it. The app reports started · changed · done.agent.invoke(offer)
In HCI Footprint
const filter = defineAction('orders.filter', {
does, invocation, mutate, …
});
const connection = connectAction(runtime, filter, {
node, instance, input, enabled
});
const agent = runtime.forPrincipal('agent');
const [offer] = agent.offers(filter);
const invocation = agent.invoke(offer);
await invocation.whenInvoked;
connection.settle(invocation.transition, { … });
await invocation.whenEffectSettled;
Definename the action, not a click
Connecthook it to the screen that does it
Offerthe app offers it to the agent
Run · started?did the app take it? (“invoked”)performedrefusedfailed
Confirm · done?did it really work? (“settled”)verifiedrefusedabandoned
Backup · the code
agentfootprint
Connect it to the screen that does it.
1DefineGive the action a name and say what it needs: “filter orders, by service”.defineAction('orders.filter')
2ConnectHook it to the screen that already does it. No new UI.connectAction(app, filter)
3OfferThe app offers it to the agent, under the same rules as you.agent.offers(filter)
4Run & confirmThe agent runs it. The app reports started · changed · done.agent.invoke(offer)
In HCI Footprint
const filter = defineAction('orders.filter', {
does, invocation, mutate, …
});
const connection = connectAction(runtime, filter, {
node, instance, input, enabled
});
const agent = runtime.forPrincipal('agent');
const [offer] = agent.offers(filter);
const invocation = agent.invoke(offer);
await invocation.whenInvoked;
connection.settle(invocation.transition, { … });
await invocation.whenEffectSettled;
Definename the action, not a click
Connecthook it to the screen that does it
Offerthe app offers it to the agent
Run · started?did the app take it? (“invoked”)performedrefusedfailed
Confirm · done?did it really work? (“settled”)verifiedrefusedabandoned
Backup · the code
agentfootprint
Offer it to the agent.
1DefineGive the action a name and say what it needs: “filter orders, by service”.defineAction('orders.filter')
2ConnectHook it to the screen that already does it. No new UI.connectAction(app, filter)
3OfferThe app offers it to the agent, under the same rules as you.agent.offers(filter)
4Run & confirmThe agent runs it. The app reports started · changed · done.agent.invoke(offer)
In HCI Footprint
const filter = defineAction('orders.filter', {
does, invocation, mutate, …
});
const connection = connectAction(runtime, filter, {
node, instance, input, enabled
});
const agent = runtime.forPrincipal('agent');
const [offer] = agent.offers(filter);
const invocation = agent.invoke(offer);
await invocation.whenInvoked;
connection.settle(invocation.transition, { … });
await invocation.whenEffectSettled;
Definename the action, not a click
Connecthook it to the screen that does it
Offerthe app offers it to the agent
Run · started?did the app take it? (“invoked”)performedrefusedfailed
Confirm · done?did it really work? (“settled”)verifiedrefusedabandoned
Backup · the code
agentfootprint
Run it: started?
1DefineGive the action a name and say what it needs: “filter orders, by service”.defineAction('orders.filter')
2ConnectHook it to the screen that already does it. No new UI.connectAction(app, filter)
3OfferThe app offers it to the agent, under the same rules as you.agent.offers(filter)
4Run & confirmThe agent runs it. The app reports started · changed · done.agent.invoke(offer)
In HCI Footprint
const filter = defineAction('orders.filter', {
does, invocation, mutate, …
});
const connection = connectAction(runtime, filter, {
node, instance, input, enabled
});
const agent = runtime.forPrincipal('agent');
const [offer] = agent.offers(filter);
const invocation = agent.invoke(offer);
await invocation.whenInvoked;
connection.settle(invocation.transition, { … });
await invocation.whenEffectSettled;
Definename the action, not a click
Connecthook it to the screen that does it
Offerthe app offers it to the agent
Run · started?did the app take it? (“invoked”)performedrefusedfailed
Confirm · done?did it really work? (“settled”)verifiedrefusedabandoned
Backup · the code
agentfootprint
Confirm it worked: done?
1DefineGive the action a name and say what it needs: “filter orders, by service”.defineAction('orders.filter')
2ConnectHook it to the screen that already does it. No new UI.connectAction(app, filter)
3OfferThe app offers it to the agent, under the same rules as you.agent.offers(filter)
4Run & confirmThe agent runs it. The app reports started · changed · done.agent.invoke(offer)
In HCI Footprint
const filter = defineAction('orders.filter', {
does, invocation, mutate, …
});
const connection = connectAction(runtime, filter, {
node, instance, input, enabled
});
const agent = runtime.forPrincipal('agent');
const [offer] = agent.offers(filter);
const invocation = agent.invoke(offer);
await invocation.whenInvoked;
connection.settle(invocation.transition, { … });
await invocation.whenEffectSettled;
Definename the action, not a click
Connecthook it to the screen that does it
Offerthe app offers it to the agent
Run · started?did the app take it? (“invoked”)performedrefusedfailed
Confirm · done?did it really work? (“settled”)verifiedrefusedabandoned
Backup · the code
done comes from the app, not from the screen