Slowing Down My Complete Integrated, Automated, and Intelligent Writing System

Last week, I started developing a writing system that (for now) focuses on writing corporate blog articles for my new RPO business line. My goal is to rank on Google and gain visibility in AI chat by leveraging a full-stack of AI agents.

Like most entrepreneurs, discipline is my biggest challenge. I know I need to develop a prototype, but the deeper I get into the project, the more opportunities present themselves. Therefore, this post is an effort to build accountability for a minimum viable scope.

The project started with two agents – a writer and a reviewer. As I started, I discovered that both were producing low and inconsistent quality. In short, I expected each to do too much work, so I started splitting them up.

Writing Tasks

A little over a decade ago, I met a ghost writer at a networking event. He was on a writing team with one of the big names in mystery novels. This chance meeting completely disrupted my mental model of how authorship and publishing work. Until that time, I genuinely believed (like most people, I think) that all the mass market trade novels were written by a lonely writer toiling away in their study.

The reality of mass market media is that it’s built by teams.

A writer like James Patterson employs a team of writers. While he contributes the concept, outline, and various prose sections, his role is primarily that of an editor.

The greatest comedians operate in a similar way, collectively developing and workshopping jokes. Their perspective is what’s important, but they huddle with other comedians to gather input and refine the content and delivery.

I like to think of LLMs as the best-educated analysts you’ll ever meet.

The LLM has all the knowledge in the world, but without context or direction, it can’t do anything with it. That’s just like a top-tier analyst coming out of an Ivy League school. They are exceptional performers, but they lack the experience and relationships to develop something of tremendous value on their own.

With these two concepts in mind, I revisited the writer agent. I split it into four components and wrote a distinct system prompt for each.

1. The Outliner defines the structure and key points based on various SEO and user concept inputs, like keywords, intent, and site structure
2. The Researcher takes the outline and collects facts, stories, and other data (with citations) to develop a trove of content for the writer to incorporate
3. The Writer develops a first draft of the piece using the structure from the Outliner and content from the Researcher
4. The Re-Writer comes in after the review loop to implement only the recommended changes without compromising the spirit of the piece

These four agents operate under different models and temperatures, depending on their requirements. They exchange information as JSON objects, allowing a programmatic control structure to intervene and govern their interaction.

Reviewing Tasks

The editorial role is filled by what I call the Polishing Loop. This consists of the following six components:

1. The Tone & Voice Reviewer examines the draft to call out inconsistencies with how we want to deliver the message
2. The SEO Reviewer approaches the structure and content to ensure keywords are well represented, and the call to action matches intent, among other factors
3. The Humanizer looks for entropy and other elements to ensure the article doesn’t come off as a machine-generated piece
4. The Compliance Reviewer checks to be sure the article isn’t making claims that would expose us to unnecessary risk, e.g., financial or HR advice
5. The Fact Checker reviews the research provided against what was included in the article to ensure the Writer didn’t hallucinate any facts
6. The Orchestrator pulls together all the reviewer content, prioritizes it, and delivers instructions to the Re-Writer on required changes

These reviewers operate within a programmatic loop that prioritizes critical and major modifications in the early phases and conceptual polishing in the final phase.

The Polish Loop is where I found most of my challenges. This can quickly become expensive if not well controlled, as each reviewer may push an agenda that conflicts with others. That is why the Orchestrator and clear programmatic rules are so critical.

Next Steps

All system prompts are drafted with instructions and v1.0 JSON schemas for information exchange. Now, I need to wire it up, which is where all the risk lies.

When starting a project, unknown unknowns are what end up delaying it.

My objective was to put together a simple writer bot, but that quickly blew up. I ended up including components to format the output, connect it to the hosting platform, and other items. Now, I’m primarily focused on just getting a great output in Markdown format. We can worry about everything else thereafter.