Agentic-Engineering on dwmkerr.com

Building Evidence with Agents

Fri, 17 Jul 2026 00:00:00 +0000

One of the most useful things I have found for agents when developing software is gathering evidence. Specifically, when changes are proposed to some kind of system (lets say for example via a pull request), that as much evidence of correctness is gathered as possible.

As an example, this is a pull request for a spike that shows how to integrate OKF exports into Ark. The specific details of this change are not important, what is important is that it will change the CLI in a fairly specific way. Previously, I would normally have asked whoever worked on this to record some screenshots - it at least lets me eyeball how it looks. With agentic, we can add a tonne of this with little effort:

This pull request has the changes, but also a lot of evidence attached - not just unit test results or coverage (which is simply hygiene), but also screenshots of what the new CLI looks like, and even recordings.

The whole pull request - and all of this evidence - was created async with a single run of Claude Code with Fable.

Why Evidence

Essentially - skip the code or spec review until you can see what the output is like. This is the smell-test, if it looks directionally correct, then you continue with the more formal review process. This is increasingly important as features end-to-end are built without supervision.

The most valuable forms of evidence include:

Links to ephemerally provisioned environments where you can actually see and interact with what has been built
Screenshots of user interfaces showing before / after
Video recordings of user interfaces (even better than the above)
Screenshots / recordings of terminal interfaces
Recordings of other types of channel interface - for example, mobile emulator recordings (for checking what things’ll look like on iOS or Android)

What does this mean in practice and what should your team do?

What you should do differently

Pretty simple - invest the time in building evidence gathering machinery. It pays compound interest, and allows you to avoid reviewing something that doesn’t look right. Spend more time on the machinery around the pipeline rather than the specific features themselves (or at least balance it).

A ready-to-go tool for your tooling - terminal recordings

Shellwright is Playwright for the shell: an MCP server giving agents a real PTY to type into, screenshot, and record as GIFs. I use this all the time for projects with CLIs, and specifically built it so that agents can provide evidence of changes such as recordings before and after.

Recap of the example in the opening: an agent built ark okf export overnight - a command that exports a Kubernetes cluster’s agent configuration as markdown files (Google’s Open Knowledge Format). Tests passed. But the real claim was “any agent can now read your cluster from plain files, no cluster access” - so the agent recorded a live Claude Code session in the exported directory, driven through Shellwright:

The key screenshot - the answer cites the files and line numbers it read:

That one image did more for my review than the test suite: the files are readable, the links work, the premise holds.

Improving the machinery

The first recording had a flaw - the answer appeared only in the GIF’s final frame, a fraction of a second. I said “I don’t see Claude reading though?” and the agent re-recorded. A real review finding, from a recording, in seconds.

The fixes fed back into the tooling: a hold_last_ms option so GIF endings don’t flash past, and a shell_wait_for tool for the “is the TUI still busy?” problem (PR here). The machinery improves.

That’s it

This doesn’t replace tests or reading the code - it sequences the review. See it work, check it’s the right thing, then read the diff.

The spike PR is here, the Shellwright improvements here.

Fable vs My Chemical Romance

Fri, 10 Jul 2026 00:00:00 +0000

I’ve been spending 15 minutes each morning with Fable - planning a days worth of development, discussing features and architecture, scheduling the work and kicking it off. End of day I spend 15 minutes reviewing and rinse and repeat. This is half a ’testing Fable’ exercise, and half a ‘get better at scheduling a days worth of work’ exercise. However, on a drive back from a physiotherapy appointment, I was thinking why not kick off a monster job? That would just be fun.

Update: this started as a bit of fun, but it turned out to be a hard enough long-horizon task that I have codified it into a Statement of Work I now use to test model and agent capability. The SOW and demo clips are at the end.

At the time the music on Spotify was My Chemical Romance, who I’d just seen at Anfield. The concert was great so I decided the goal would be “make me a home version. singalong karaoke style lyrics on the screen. official videos where they exist, live footage otherwise”. I’d also prompt a few tips around process but let it run.

This is not a good test of coding practices (with my teams that is a process of evolution as we work out how best to move towards agentic engineering), just a fun open ended exercise.

I created a README.md with the following mission:

Create a full concert-length video I can project on my TV: a home version of the
My Chemical Romance gig I went to. Check YouTube and my Spotify playlist to find
the setlist. Make 10 mini videos with some styles, karaoke sing-along lyrics
running on them, cool visuals behind. Official music videos for songs like The
Black Parade where they exist, live concert recordings otherwise. Build the
machine to do this: use subagents, skills, create tools, create CLIs, run them,
build all the machinery you need to solve, execute and verify this. You're in a
git repo: make a scratch folder and commit as you go. Keep a task list and a
journal of key events. Build and run the machine until it works.

And the prompt:

Tuning process

A lot of guidance suggests ‘define the goal, not the process’ (see Think with Google and this piece by Tyler Folkman). Anthropic have also built modes like ultracode and teams where Claude will set up the appropriate teams for the work (or try to). This is generally where I differ: for open-ended work I’m happy to define the goal and let the agent find the path, but for more complex, audited or sensitive processes, where predictability and explainability matter, building a framework for the operations is important. This was obviously not one of those cases, just a quick experiment. So I did suggest a few things:

Playwright for browser automation, to perform more complex searches and analysis
Sub-agents, to preserve context
Building small, simple tools that can be composed together into larger systems
Building machines that use those tools
Building verifications and checkpoints
Creating a human-in-the-loop point where I can review samples and the execution plan

Essentially; build yourself tools, build a machine to solve my problem, then run the machine and we’ll tune the machine afterwards.

The result

About $500 worth of tokens and many hours later (this used less than I expected, I thought I’d see more cost from image processing). To be clear on the money: this ran on my personal Max subscription, not anything work-related, and that $500 is the API-equivalent value of the tokens burned, not a bill. On a Max plan you pay the flat monthly fee, and Fable draws it down faster than Opus because it is priced at twice the rate.

How it ran

Under the covers, after reviewing the execution plan and the session logs, this is roughly what happened.

A 30-agent research sweep to work out the real setlist and find a good source for each of the programme items. Playwright let it compare sources and do real research on the open web, which is expensive in tokens but vastly increases the options it has.
A pipeline to gather the source video and audio for each song.
A lyrics-to-karaoke compiler that turns synced LRC files into ASS subtitle karaoke, word-level where it could get the timing, line-level where it could not.
A renderer with three modes: official music video, live footage over studio audio, and a generated visualizer backdrop for the audio-only tracks.
An audio-video sync tool that aligns live footage to the studio track using chroma and onset features, and an assembler that stitches everything into ten act-titled mini-films.
A verification pass (39 agents) checking every segment for resolution, duration, and lyric sync. I rate this step highly; it mirrors the process I follow on many professional jobs.

The output was 105 minutes of concert across ten files, plus a little web app to swap backgrounds and lyric styles per song.

Is it impressive? I don’t know, I don’t do video editing / researching tasks. My 15 mins per day with Fable write-up coming soon showed me results I can comment on more realistically (a more complex software engineering project).

The hardest part: syncing footage to studio audio

The bit that took the most work was using live concert footage as the visuals while playing the clean studio recording as the audio. A live band never plays at exactly the album tempo, so if you just mute the footage and drop the studio track underneath, the two drift apart. Within a minute the drummer is visibly hitting things you can’t hear.

Here is how the machine solved it for one song, House of Wolves, with pro-shot footage:

It takes a mel-spectrogram (a picture of the sound) of both the live audio and the studio master, then computes a mapping between them with chroma-CQT dynamic time warping, which matches on harmonic content rather than the raw waveform. The footage is chopped at every camera cut and each shot is placed on the studio timeline at its matched position, sped up or slowed down by an amount small enough not to notice (clamped to 12%, red for faster, blue for slower). The clever bit is re-anchoring at every camera cut: the eye already accepts a jump at a cut, so drift can only build up within a single shot, never across the whole song. Here 129 cuts became 35 anchored chunks, and it reports the honest numbers for the result (93ms alignment error, 100% coverage, verdict GOOD).

None of this was a technique I gave it. It researched the approach and built the tool itself, including working out the re-anchor-at-cuts trick on its own. The diagram here just visualises what it did, with the real numbers from the run.

Interesting observations

For a job like this I only stepped in a handful of times. Probably the most meaningful suggestion I made was an idea: could it read the beat from the drummer’s arms, the crowd head-banging and the stage lights flashing, and work out the rhythm that way? It went and tried.

Left to itself, it never reached for another model. All of the video and audio sync work was done with tooling it wrote itself, and it stayed entirely inside its own capabilities. It only started looking at other models like Gemini for the video processing after I suggested that a different model might be better suited to that part of the job. On its own it did not seem to consider that reaching outside itself might be the better move, which felt like a notable blind spot for an open-ended task.

The other thing that made me laugh: at one point Fable’s own safeguards flagged one of its own messages and it fell back to Opus 4.8 mid-task, then carried on. Almost certainly the lyrical content again - My Chemical Romance lyrics are dark, and earlier one of its sub-agents had already tripped a content filter for echoing them back.

I don’t think I’ll run this exercise again - it was a bit of fun on the side that I thought I’d share before I get into my wider Fable experiments. I’ve got a couple of other gigs coming up anyway: Misery Index, Chelsea Wolfe and Igorrr.

If you want the real thing: The Black Parade on Spotify and My Chemical Romance. 🖤

The SOW

The hard part, syncing live footage to a studio track, is now a standalone brief I can hand to any model or agent. It runs at two levels: silent footage first, where it has to identify the song and align it from the picture alone, then the same footage with audio. Effort (tokens, tool calls, time) is measured but not scored. What is judged is whether the result actually plays and stays in sync.


The SOWs	github.com/dwmkerr/agentic-engineering-sows
Level 1 input, silent	watch
Level 2 input, with audio	watch
Fable output	watch

For the record, the whole session, the original build plus days of tinkering, editing and this write-up, cost $485.43 across three models: Fable 5 did the build ($322.65), Opus 4.8 covered the safeguard fall-backs and most of the later editing ($162.07), and a little Haiku for search ($0.70). Roughly 2.3m input and 1.7m output tokens, around 990 tool calls and 24 web searches, 80+ sub-agents spun up across ultracode workflows, 9h 25m of API time spread across about five days, and 7,246 lines of code added. Measured, not scored.

Bipolar Disorder, Dysregulation & AI

Tue, 09 Jun 2026 00:00:00 +0000

At AI Native DevCon London 2026 I gave a talk about bipolar disorder, what living with it has forced me to learn, and how the same patterns keep showing up in the way we work with AI.

What the talk is about

Managing bipolar disorder is largely about regulation. There are things that push your state around - sleep, rhythm, stress, people, substances - and the work is to spot the ones that destabilise you (dysregulators) and counterbalance them with the ones that steady you (regulators).

This is hard partly because the condition can come with compromised interoception - your sense of your own internal state, the thing you’d normally rely on to tell where you are. When that’s unreliable, you can’t fully trust your own read on yourself, so you lean on what’s around you instead.

Once you’ve spent years pathologising your own behaviour, looking for what’s adaptive and what isn’t, you start to recognise the same shapes elsewhere. Over the last few years I’ve seen a lot of them in AI-native engineering:

Transformative shifts in core beliefs
Addiction
Grandiosity
Attentional fragmentation
Pressured speaking
Maladaptive creativity

The counterweights - the regulators for each of these - are described in the talk.

Agentic Engineering Protocols: The Ralph Wiggum Loop

Wed, 04 Mar 2026 00:00:00 +0000

The Ralph Wiggum Loop

The Ralph Wiggum Loop is one of the earliest and most simple agentic development patterns, a somewhat whimsical and silly approach to getting a coding agent to run continuously to solve a problem, with an extremely simple orchestration pattern. Ralph is not a pattern I’d really recommend for real-world problems but you’ll see it around and hear about it and it’s fun to explore and play with.

The pattern is simply “run an agent in a loop, with a single prompt, until it thinks it’s done”. Very few instructions, almost no orchestration at all, often with no intermediate steps tracked, it can be surprisingly effective for certain types of task. Ralph Wiggum is a character from The Simpsons who is not particularly smart but kind of good natured and persistent.

This is part of the Agentic Engineering Protocols series:

Agentic Engineering Protocols: Intro and Superpowers

What It Is

At its core, the Ralph protocol is an endless loop:

# Loop until claude thinks it's done...
while true; do
 # Run an agent such as claude with instructions".
 claude -p "Your desired end state description"
done

# Grab a cup of tea.

The prompt describes what done should be. Each iteration, the agent runs a single session, does some work, and exits. The loop feeds it the same prompt again.

No instructions on how to to the end state are included. No orchestration patterns are used. You’re basically letting the agent figure it out. Typically it will change files, build and test, and repeat. The state of the code itself implies where the agent is in the process.

There’s an official Anthropic plugin that adds stop hooks and completion detection, but the original Ralph that went viral is just a dumb loop.

Why

Recent LLMs have pretty big context windows and agents are increasingly (somewhat disturbingly) smart. But at the time the loop was shared, context windows were smaller and failures were more common. In those cases the agent process (such as claude) would terminate early (for example due to running out of context). Typically some work would have been done, the code would be in an intermediate state, and then the next iteration picks up from there.

Increasingly complex tasks can be one-shot nowadays, context windows are huge, agents are far more smart, skills are more effective, subagents are more efficient, and this pattern is less common to see, but its fun to try it out.

Ralph Wiggum vs a Tedious Task

My team had been wanting to move a large feature from our open source project Ark (a Kubernetes based agentic toolkit) to our marketplace of smaller modules. This means deleting a lot of code, eliminating tests, removing docs and cross references, screenshots, and so on. It’s kind of tedious - remove stuff, test, rinse and repeat. Claude would probably one-shot it well given how many integration tests and verifications we have as specs. But I decided this was a good way to show off Ralph’s style.

When we’re adding code, we try to be rigorous. Specs, red/green tests, docs, skills to follow architectural patterns and so on. Reasonably careful orchestration, and a good amount of discussion if needed. Eliminating code is more straightforward.

Running the Loop

We need to eliminate code from low-level operators, a CLI, two SDKs, docs, tests, deployments, pipelines, specs. We don’t need to one shot it, we could quite easily ask the agent “remove a chunk, then stop and test” and then have it repeat this process.

Here’s the loop (slightly simplified, ralph-loop.sh is the original):

#!/bin/bash

# Seed a progress file for Claude to maintain between iterations.
touch progress.md

PROMPT='Evaluations are no longer core in Ark, we are moving them to a marketplace.
Remove the CRDs, docs, demos, integration tests, references, SDKs, UI, the lot.
Read progress.md for what has been done so far. Update it as you work.
Do one task, update progress.md, commit, then stop.'

while [ $ITERATIONS -lt 100 ]; do
 ITERATIONS=$((ITERATIONS + 1))

 echo "$PROMPT" | claude -p --dangerously-skip-permissions

 git add -A && git commit -m "ralph: iteration $ITERATIONS" --allow-empty

 sleep 5
done

Quite open ended, few notes on ‘how’, just what we want. I added instructions to track process in a file to make it easier to show what happened at the end, and we terminate the loop if the agent thinks it is fully complete (this is janky, don’t do it, but it’s in-line with common samples and shows the pattern, you’re more likely to see claude loop saying it’s got nothing to do and have to terminate it).

What Happened

This took 8 iterations. Here’s the git log:

The progress.md is quite interesting (this is just a snippet):

### CI/CD and Infrastructure Removal Details
Edited 7 files:
.github/workflows/cicd.yaml:
- Removed ark-evaluator from xray-container-scan matrix
- Removed all install-evaluator references from setup-e2e calls
- Removed !evaluated from standard test chainsaw selector
- Removed e2e-tests-evaluated from report-coverage and check-release needs lists
.github/workflows/deploy.yml — removed ark-evaluator from deploy container build matrix
### Remaining References Found (iteration 6)
Searched entire codebase for evaluat references. Found and fixed:
- RBAC resource lists in broker tests and tenant charts
- Argo workflow evaluation templates
- Chainsaw summary script evaluation functions
- MCP tools evaluation status references
- README project description
- Architecture diagrams
- Claude skills referencing evaluator paths
Preserved (generic English, not Ark CRDs):
- "evaluation-driven methodology" in walkthrough docs
- "ease of evaluation" in disclaimer
- Langfuse/Phoenix observability descriptions
- Historical CHANGELOG entries

We eliminated tests, parts of the build pipeline, docs references, generated artifacts, skills, samples, demo workflows and so on.

The final PR is at refactor: remove evaluations from core.

451 files changed, 58,763 lines removed — which as a tech lead is a wonderful feeling. This will now be a pluggable module in our marketplace, easier to test in isolation and evolve at a higher pace. Our codebase is smaller and easier to work with.

The loop won’t work if the agent has to ask for permissions, so I YOLO-d it in a container, using a less-privileged identity (I have an over-engineered pattern for running work with different identities which I can share if anyone is interested).

What the loop missed

There were a few things that didn’t work.

The more complex integration tests failed. The repo has skills that explain how to run these tests locally, but the skill never fired and the integration tests weren’t run. This we could potentially deal with using a pre-push hook or similar, although the suite is time-consuming.

A process that generates types in an SDK based on the resources in the cluster was not run, leaving us with some stale code. This has been a persistent problem we haven’t got around to dealing with and an example of “if the process is so complex the LLM gets it wrong more than twice, it’s too complex”. It’s on the backlog but run only occasionally.

One type that lived in the evaluations code was used elsewhere and got deleted. This would’ve been caught if the integration tests ran.

Asides from these issues, Ralph did a solid job at the mechanical work. The gaps are on our side - our agents should be configured to not believe that a piece of work is complete until all evidence says so. This could be as simple as improving the CLAUDE.md to say that when a shippable unit of work is done, integration tests must pass locally, a hook, or whatever. Everything like this is an opportunity to improve the overall software-engineering engine - if the agent doesn’t one-shot it we improve the system and do better next time.

Thoughts on Ralph

It’s brilliant. A dumb loop with a great name. The work of AI is stressful and disregulating. Ralph brightened up my day. I can’t see myself using the pattern, building the script, testing it and so on in a real-world task, but I really wanted to show it off.

For somewhat mindless “just go and make it happen” work it does an OK job. Nowadays I’d typically go for teams (writeup coming soon) for tasks where I am less specific about structure and orchestration. But if you have a chance give Ralph a go it’s quite satisfying.

To try the official plugin in Claude, install it:

claude plugin marketplace add anthropics/claude-code
claude plugin install ralph-wiggum@claude-plugins-official

Then run it:

/ralph-loop "Your prompt here" --max-iterations 10 --completion-promise "DONE"

Agentic Engineering Protocols series:

Agentic Engineering Protocols: Intro and Superpowers

Agentic Engineering Protocols: Intro and Superpowers

Wed, 11 Feb 2026 00:00:00 +0000

In this series I’m going to demonstrate and discuss methodologies and tools for “Agentic Engineering”. Agentic engineering loosely refers to modern techniques for software engineering that are heavily focused around using agents as much as possible to accelerate work, improve resiliency, automate testing and so on. “Protocols” are what I’ve called some of the specific patterns and tools that are used, such as the Ralph Method, Superpowers, BMAD and Openspec.

Almost all of these patterns anchor around clarifying the user’s “intent”, building a specification, breaking down into a task list, executing, testing (typically red/green) and performing additional evaluations such as agent assisted exploratory testing. Artifacts built along the way such as goals, specs and task lists are usually tracked in a repo as markdown files for later analysis. Some of these protocols are quick and easy to try out, some are more complex and opinionated.

I’ll update this series periodically with a real-world demonstration of some of the popular protocols that are out there. But to get started, we’ll take a look at Superpowers.

Next in this series: Agentic Engineering Protocols: The Ralph Wiggum Loop

Obra Superpowers

Superpowers is a very popular (+50K stars at time of writing) framework for agentic software development.

Superpowers takes you through a common sense flow for software development, essentially:

Brainstorm, clarify intent, challenge assumptions
Document the goals, build a plan, create a design
Create an isolated environment / worktree to work in
Execute the plan, following TDD methodologies
Review the results and integrate

This is handled with a combination of slash-commands, sub-agents and skills. Superpowers is a good place to start because it’s quick and easy to set up and play with. It’s less sophisticated than other protocols, but almost all protocols follow a similar pattern (build a spec, then a plan, implement it, and attack and challenge along the way).

As a demonstration, I’ll use my Shellwright project. Shellwright lets an LLM run a terminal and record what’s going on (a bit like Playwright browser automation but for the shell). I can ask my agent to record an action in the shell and get a video of the result (this recording shows the agent opening and closing Vim):

My new requirement is simple - I need to be able to optionally include the terminal cursor in the recording and screenshots.

This is a small simple change that with good instructions and a sensible implementation protocol we should be able to knock out fast. We could just prompt for it, but a protocol like Superpowers will help make the process more structured.

Brainstorming

You start using Superpowers by brainstorming. Use the /superpowers:brainstorm slash command and describe what you want:

/superpowers:brainstorm I want to be able to optionally show the cursor in Shellwright recordings or screenshots

Permissions are requested, the codebase is explored, clarification is requested:

In this example, the brainstorm process actually changed the design considerably. Rather than my initial idea of a parameter to show or hide the cursor, the recommendation was to just respect the state of the in-memory terminal.

This means in most cases there’ll be a cursor, it’s only certain programs such as Vim that hide it at certain times. We don’t need to let the caller choose to turn it on or off, the terminal does that itself.

This was a good suggestion, it follows the principle of least astonishment. It keeps my API clean, and I can always make things more configurable in the future if I need to. Being challenged is essential, it’s one of the best things about working with other experienced engineers, and LLMs don’t tend to do it unless really instructed to do so.

Approaches are now suggested, with trade-offs:

I chose the approach that keeps the main flow as-is and adds a separate pass for the cursor rendering, which feels like a good separation of concerns as well as a little less risky.

There was a bit more discussion around design, all common sense stuff. Finally a plan is written. You can see it in the pull request in docs/plans/2026-02-13-cursor-visibility-design.md.

Superpowers does a decent amount of exploration using subagents and uses specific planning skills to build the plan. The structure of the plan is less rigorous or prescriptive than some other protocols but it’s also pretty easy to follow.

This structured planning approach is essential, almost every protocol I use spends a lot of time building plans and specs (except for protocols that are more around exploration and experimentation, which is really just to build data for a plan later on).

An early mistake

Interestingly here’s where superpowers (or to be fair, the LLM) struggled. An assumption had been made that the underlying terminal emulator (xterm.js) actually allowed you to check cursor visibility. When moving into the design the agent quickly decided that there is no visibility field exposed in the in the xterm API, and then suggested just always showing the cursor.

LLMs can be weirdly defeatist sometimes. I kicked off my own researcher agent and asked it to investigate (this agent clones repos, looks into code, uses a headless browser if needed to read docs, asks for content it can’t find and so on, and tries to find at least 2-3 data points to corroborate findings).

It quickly found that the API is exposed, backed up by three data points (docs on the API, a unit test for it, and a GitHub issue that discusses it). The research output is at ./docs/plans/research

Anyway, Superpowers had asked me to review the plan, so I saw the issue and made the correction (tracking the research results as well for future reference).

Implementation

Once the plan is approved, we move to execution. Two modes are offered - subagent driven, which iterates on each task via subagents in the current session with review at each step, or a parallel session (which is where you kick off a completely new process and execute from there).

In this case I chose to run a new session as I didn’t expect to have to do any more orchestration from my main session:

claude '/superpowers:executing-plans docs/plans/2026-02-13-cursor-visibility.md'

Additional skills are run to make sure we have a working location - Superpowers asks where we want to keep our Git worktree¹. This is a really nice feature, I had something similar in one of my own protocols (creating a .sandboxes folder) but Superpowers is explicit with what is going on:

This skill also checks to see whether .worktrees/ is git-ignored and suggests to ignore it if it isn’t.

The half-time mistake

Halfway through implementation, the agent got confused and couldn’t find the visibility field mentioned in the plan. It then decided the best approach was to just always show the cursor. I was watching so that I could grab screenshots and take notes so quickly saw the mistake.

At this point I was reminded of something from my favourite CLAUDE.md file:

Notice Confusion

Your strength as a reasoning system is being more confused by fiction than by reality. When something surprises you, that’s not noise—the universe is telling you your model is wrong in a specific way.

Stop. Don’t push past it.

Identify: What did you believe that turned out false?

Log it: “I assumed X, but actually Y. My model of Z was wrong.” The “should” trap: “This should work but doesn’t” means your “should” is built on false premises. The map doesn’t match territory. Don’t debug reality—debug your map.

The author is clearly very experienced with the implicit biases in these tools.

I challenged the LLM to discover why the research had three data points saying the visibility field existed, but the local library seemed to not. After a few nudges it worked out that the field was added in a more recent version of the library and we needed an upgrade.

Review

The next stage is to review the implementation and test.

Late game wobble

Superpowers’ test plan in this case was ‘manual verification’ - not even a single unit test. I was very surprised, it’ll normally create failing tests first and then show them passing, following TDD patterns. The red/green approach is great for challenging assumptions early.

Why testing was not part of the plan I don’t know. Shellwright has few unit tests, and a fairly complex evaluation pattern. Perhaps this caused confusion.

To be fair, testing Shellwright properly is quite hard. I use unit tests, but an end to end test means giving an LLM access to the MCP server, asking it to issue commands and record videos. Deterministically verifying these is hard (for various reasons that are described in the project).

To handle more complex testing scenarios, I have prompts stored in markdown files. These are instructions that are passed during CI/CD to an agent that has the Shellwright MCP server installed. Each prompt produces a video, and I show a table in the pull request with the “before” and “after” for each scenario. This means I can eyeball how the videos look and get a sanity check that output is looking sensible. A table like this is added to pull requests:

I’ve opened a PR here that I’ll keep open so you can see it - check the “View Recordings Evalution” section:

https://github.com/dwmkerr/shellwright/pull/60

Superpowers didn’t notice any of my (fairly well documented) evaluation approaches. However, when I instructed it to RTFM it went ahead and created a new reference video. Here it is - a shell recording showing the Ctrl+A shortcut that moves the cursor to the beginning of the line:

Instructing the LLM to follow my evaluation patterns was the final intervention I needed to make.

Finishing Development

Superpowers moves into the ‘finishing’ stage at this point:

This is when tests are re-run, the build is checked again, there’s a final verification of the work and so on. The final PR is below:

https://github.com/dwmkerr/shellwright/pull/57

It’s adequate. I was surprised to not get challenged on unit tests on this one, normally superpowers does a better job. A quick look at the evaluation records that are produced shows what I’d expect - old recordings don’t have a cursor, new ones do:

The code changes are small. No documentation changes have been made (and that’s probably OK).

Thoughts on Superpowers

I feel I might have given Superpowers a slightly rough ride, for whatever reason, maybe because I decided to use this particular task for the write up, it wasn’t as smooth as other times.

But in reality, Superpowers is great. It takes almost no setup, is not particularly opinionated and follows common sense patterns. If you want a gentle introduction to orchestration protocols its a great place to start. It’s easy to change later and the foundational concepts you’ll see in many other protocols.

Some of my projects have more opinionated processes, many are more spec-driven, some I’m still experimenting with, some are more team orientated. But if I’m just jumping into something new, or looking at an issue in another project, I’ll often use Superpowers at first and then pivot if it struggles. Used well it can handle far far more complex tasks than I demoed here.

This brief whirlwind tour got a little convoluted. Things worth noting:

Major version changes

This ‘small’ change required a bump from v5 to v6 of xterm-js. The model / agent / protocol didn’t warn at any stage that this is risky or should at least be considered with caution. This is the sort of thing I call a ‘signal’ - something in a change that might indicate more attention is needed.

Protocols are not rules

Superpowers normally red/greens tests (i.e. creates failing tests to show that the desired behaviour is not present, then makes changes, then shows that the tests now pass), which is excellent - TDD is a great pattern. But even well encoded protocols are not rules, we still rely on the LLM to follow these instructions.

It is still surprising how quickly models can stray away from instructions.

You can get some enforcement by using hooks (and some of the other protocols I’ll describe do this), but it is safest to assume the model will stray from time to time.

Evaluations and test harnesses are important

My intent was a small, common sense change to the output of my project. But it required a major version upgrade of an essential library, as well as change to the main rendering functionality.

This project is hard to test in a completely deterministic fashion. I feel that the effort put into being able to see videos side-by-side of before and after changes is hugely beneficial. I probably spend more time nowadays building ’the machine’ (such as the evaluation system, the automated testing) than the code, and that’s a good way to go.

Anything that provides evidence of correctness is essential if we want to be able to make changes quickly.

LLMs hallucinate, make assumptions, and mistakes compound

There were a few pretty glaring mistakes:

During planning, the name of a parameter was hallucinated
During implementation, when the parameter couldn’t be found, an assumption that no such parameter existed was made
During implementation, confusion (research saying a parameter existed, code saying it didn’t) was not escalated - instead, assumptions were made

Each mistake required intervention. Superpowers (and similar protocols) are filled with instructions to avoid bias, instruct agents to stop when confused, don’t make assumptions, and so on, but it still happens. It’s in the nature of the technology - completions models are designed to helpfully complete your request.

Practice and experience helps here. As soon as something seems weird, I’ll often kick off research agents. They run independently and are instructed to gather evidence - find code, find issues, find websites, cite sources. Identifying signals (things that seem intuitively wrong, or that might be consequential, risky, and so on) is important.

Try it out

Give Superpowers a whirl. Its quick to install and it interactively guides you through the process and protocol.

claude plugin marketplace add obra/superpowers-marketplace
claude plugin install superpowers@superpowers-marketplace
claude "/superpowers:brainstorm I want to build an agentic SLDC protocol"

Agentic Engineering Protocols series:

Agentic Engineering Protocols: The Ralph Wiggum Loop

I find worktrees really janky, mostly because you can’t have the same branch open in multiple worktrees. In my flow I have my repos structured in folders like: ~/repos/github/org/project/[branchname] and just do a full-fat checkout (and a tmux tab per branch). Its heavier on the filesystem but you have freedom to switch branches at will (and can open a tmux session with a tab for each branch you are working on with a single command). ↩︎