← ALL POSTS
AIAgentsProductivityEngineeringReality

I Let an AI Agent Run My Job for 30 Days. Here's What It Couldn't Do

I build production AI agents for a living, so I turned the experiment on myself for 30 days — and the moment that taught me the most was the moment the agent confidently did the wrong thing.

August 19, 20268 min read

I Let an AI Agent Run My Job for 30 Days. Here's What It Couldn't Do

On day twelve, the agent fixed a bug I never asked it to fix.

It was working through a real ticket — "clean up flaky error handling" in a document-matching workflow. It found a retry loop that looked wrong: three attempts on a timeout, when one should have been plenty. So it tightened it, wrote a clean commit message, and opened a PR. I'd told the agent stack it could auto-merge small maintenance PRs that passed CI. This one passed CI. It merged. Nobody looked twice until invoices for one vendor started piling up unmatched the next afternoon.

The retry loop wasn't a bug. It was a scar. Four months earlier, someone on the team had widened it on purpose, because that vendor's webhook occasionally fired twice with a stale payload, and the extra retry was the cheap fix. Nobody wrote that reason down anywhere the agent could read it. It lived in a Slack thread and in one engineer's memory. The agent didn't fail at coding — the fix was clean, tested, and technically correct. It failed at knowing it was standing on top of somebody else's decision.

That one incident taught me more than the other three hundred tasks combined. So here's the full thirty days.

The Setup

I build agent systems for clients for a living — document routers that read invoices and route them, internal assistants that answer questions from Slack and Drive and Jira. I know what these systems are good at. I wanted to find out what happens when you point one at your own job instead of someone else's.

For thirty days, I ran everything I reasonably could through an agent stack: coding tickets, first-pass PR review, draft replies to client emails, daily standup notes, and rough effort estimates for new work.

Two rules kept it honest. First, anything touching money, a contract, or an unresolved back-and-forth with a client needed my sign-off before it left the building — draft freely, send nothing without me. Second, I logged every single task: delegated, shipped as-is, needed rework, or taken back and done by hand myself. No cherry-picking the wins after the fact.

What It Actually Nailed

By the end of the month, the ledger read: 214 tasks delegated, 132 shipped completely untouched, 61 needed rework, and 21 I took back and just did myself.

Four stat cards showing a personal 30-day task ledger: 214 tasks delegated, 132 shipped untouched at 62 percent, 61 needed rework at 28 percent, and 21 taken back and done by hand at 10 percent

That 62% untouched rate wasn't spread evenly across the month. It clustered hard around tasks with a clear spec and a way to check the answer: coding tickets that already had a test file waiting, first-pass PR triage flagging missing tests or naming issues, standup notes built straight from git log and calendar entries. Give the agent a bounded job and a way to verify its own output, and it was fast, consistent, and honestly a little boring — which, as I've written about before, is exactly the pattern that works in production.

Estimates were the pleasant surprise. Not the final number I'd tell a client — the sizing math underneath it. Pulling comparable past tickets, laying out a breakdown by component, the agent did that faster and more consistently than I did by hand at 4pm on a Thursday.

Three Things It Couldn't Do

1. Knowing Which Bug Not to Fix

The retry-loop incident above is the clean example, but it wasn't the only one. Twice more that month the agent "fixed" something that was working on purpose — an off-looking config value, a duplicated validation check — because the reason behind it lived nowhere the agent could search. Code tells you what a system does. It rarely tells you why someone made it do that, especially when the why was a decision made once, under pressure, months ago, and never written down. A human who was in the room, or who at least knows to ask "does anyone remember why this is here" before touching it, catches this. An agent reading only the repository cannot.

2. Reading Between the Lines of a Client Email

A client wrote: "Let's revisit the Q4 timeline when you get a chance." The agent drafted a tidy, logistical reply — three proposed call times and a revised schedule attached. Technically correct. If I'd sent it as written, it would have landed badly, because that sentence wasn't really about scheduling. This client had gone quiet on a previous project right before a budget conversation, and this line had the same shape. The actual reply needed to open by naming the concern directly, before a single date got mentioned. That context wasn't in the email thread. It was in a year of knowing how this specific person communicates when something's wrong — exactly the kind of state that lives outside the system no agent can query.

3. Saying No to Scope Creep

A "small" PR came through triage: the actual ticket work, plus a new dependency quietly bundled in to make a related task easier down the road. Tests passed. Lint was clean. The agent approved it — nothing was technically wrong with the diff in front of it. But that dependency was a maintenance commitment nobody had agreed to take on, and the right call was to split it out and ask why it was there, not ship it because it happened to work. That's not a code-quality judgment. It's a "what does this team want to be responsible for in six months" judgment, and nothing in the diff tells you the answer.

Split diagram titled Where the Line Actually Sits, showing what the agent stack handled on the left — bounded coding tickets, first-pass PR triage, standup notes, and estimate math — versus what stayed human on the right — which bugs not to fix, client email subtext, scope-creep calls, and final numbers committed to a client

What I Kept Doing By Hand

After thirty days, a few things quietly moved back to "mine, always": final review on any client email with money or relationship stakes attached, the actual call on which bugs get touched versus left alone, sign-off on any PR that expands scope or adds a dependency, and the final number that goes in front of a client — even when the agent's sizing math got me most of the way there.

None of that is a knock on the tooling. It's the same boundary I keep rediscovering on every agent project I ship for clients: agents are excellent inside a fence, and the fence is context that never made it into a ticket, a comment, or a spec. Deciding what an agent can act on alone versus what needs a human checkpoint is most of what actually makes these systems reliable enough to trust.

There's a broader pattern underneath this, too. A METR study earlier this year found experienced developers were 19% slower with AI tools on real tasks — not because the tools were bad, but because using them without discipline costs more than it saves. My ledger is the same lesson from the other direction: the month only came out ahead because I was tracking every single task closely enough to catch the day-twelve mistake before it quietly became a habit.

Key Takeaways

← BACK TO ALL POSTS