For 2 months I published every single week.
No junk posts, no ai slop, none of that. Posts on topics I actually wanted to write about, that I’m competent enough to talk about. Same day every week, like clockwork. By every rule you get handed, I was doing it right.
And my subscriber count just sat there.
You know that feeling? You’re not lazy, you’re showing up every single week, and the graph stays flat while you quietly wonder if you’re just not good enough.
I was being consistent. And here’s the part nobody tells you when they hand you the “just be consistent” line: consistency only moves the needle if you’re also iterating on the process behind it. Showing up isn’t the same as improving.
So I did what everyone does. I blamed my writing. I tightened hooks. I studied structure. I rewrote openings until they were sharp. The line stayed flat.
It took me longer than I’d like to admit to see the real problem. It wasn’t my writing.
Good content works on 3 pillars
Here’s the simplest way to think about it.
Anything worth publishing sits on three big pillars:
What you like
What you actually know
What is trendy
Look at your last 5 posts. You’ll almost certainly find pillars one and two all over them. The stuff you enjoy, filtered through what you know, through the lens of your expertise. That’s most creators. That’s exactly where I was a couple of weeks ago too.
The third pillar is the one everybody avoids.
Think about it. The first two are free. They live in your head. Instant, always there, no extra work, no research. The third one is trickier. It lives outside you, in your readers, their comments, THEIR heads, the posts they’re actually engaging with across your niche. The only way to get it is to go out and look.
And going out to look is the part everyone skips. I know, because for 2 months I did it by hand.
Every Sunday, the same five competitor newsletters. Open the page. Scroll the recent posts. Read. Check what fit my framework. Paste the good bits into Claude for inspiration. A whole ritual, start to finish, by hand. Then one Sunday I asked myself the question I’d been avoiding. Would a real automation expert keep doing this by hand? I knew the answer. I was just tired. And tired is a terrible place to start writing from.
Because by hand, that’s all research ever becomes. You skim subject lines. You glance at what got engagement. You paste something into a doc and tell yourself that’s research. It’s pattern-matching that resets to zero every week and never gives you the same answer twice.
So here’s what quietly happens. When research is painful, you skip it. Not on purpose. You just keep writing pillars one and two, straight from your head, into a feed that moved on without telling you.
It feels like authenticity. It just doesn’t grow your list.
There was an article this year where the author looked at 3,230 notes from nine creators in the AI and tech space. Three posted almost identical volumes, 475 to 499 notes each. Their per-note engagement averaged 8.7, 33.5, and 37.7 likes. Same volume. Same niche. A four-to-one gap.
The difference wasn’t how often they posted. It was whether their content kept tracking what the audience wanted, or quietly repeated what the writer already liked.
The creator at the bottom? Her best posts had nothing to do with the niche she thought she was in. Her audience had moved and her feed hadn’t noticed. She was standing on pillars one and two, writing from her head, while the third pillar sat there ignored.
That’s the real reason most Substacks grow slow. Not the writing. The missing third pillar. And the only way to put weight on it is to research your niche properly: what your competitors’ readers are actually responding to, which topics keep climbing, and which ones you keep walking past. Done right, that stops being a Sunday headache and starts telling you what to write before you write it.
That’s what the rest of this is about. Most people’s version of research is broken at the root, so let me walk you through the three stages of it, and the one that actually compounds.
Why most “automation” is still Stage 1
From what I’ve seen, there are 3 stages of niche research. Most people are stuck at Stage 1 and don’t know there’s anything past it.
Stage 1 is basic scraping. You, or a tool, pull recent posts from a few competitors. You read them. You hunt for patterns. Better than nothing. But it resets every single time. Each session starts cold. No memory of last week, no comparison against what you’ve already covered, no accumulation. You gather more and more, it turns into noise, and you walk away with less clarity than you started with.
Stage 2 is context-aware research. An agent that knows who you are before it starts looking. Your content pillars. Your audience’s actual pain points. The angles you’ve already written. The topics you’d never touch. With that sorted, the output stops being “here’s what’s trendy” and becomes “here’s what’s worth writing, given who you are.” Same raw input. Completely different answer.
Stage 3 is research that compounds. The agent doesn’t just scrape now. It remembers what it found last week. It compares this week against the last four. It notices you ignored the same gap twice. It tells you what changed since the last run. The fourth scan is sharper than the first because every run adds to the picture.
Almost everyone I know is at Stage 1, or they don’t do research at all. A few stumbled into Stage 2 without naming it. Stage 3 is where the real edge is, and it’s where I ended up.
I built a Stage 3 version using Claude Cowork and one MCP connector. Yeah I know, sounds complicated, but it really isn’t. Bear with me and I’ll fully walk you through. Here’s what Stage 3 gives you concretely.
What Stage 3 actually feels like
The week after I set it up, I sat down on a Tuesday morning, typed one sentence into Claude, and went to make coffee. 10 minutes later there was a research brief sitting in my project folder telling me:
which topics were heating up across the newsletters I track
which posts actually overlapped with my audience
which titles and hooks pulled the most in the last 30 days
And at the bottom, one quiet line: “You’ve skipped gap X in your last 3 articles.”
That last line is the whole thing. That’s the agent learning from its own history instead of starting cold.
Not “here are ten content ideas.” Ten ideas is just more noise.
It had enough memory to notice I kept walking past the same opening, week after week, and it put that in front of me. That’s closer to having an editor than a research tool.
The Sunday before, I’d burned two hours and still felt unsure what to write. The week after, I had more useful signal in 10 minutes than a month of manual sessions had given me.
The agent didn’t improve my writing, but I just stopped guessing about that third pillar.
How it actually finds the gaps
When I first describe this, people assume the “find me an opportunity” part is magic, or worse, fluff. It isn’t. It’s boring set math, which is exactly why it works.
Stay with me here, this part matters.
Every run, the agent has four things loaded about you: your content pillars, the topics you’ve already covered, the topics you refuse to write about, and what you’ve defined as a gap worth your time. Then it pulls every competitor post from the last 30 days with the title, full text, and engagement numbers attached.
A real opportunity has to clear four filters at once:
It sits next to one of your pillars, not off in some random niche
It’s pulling actual engagement right now
You haven’t already written it
It’s not on your “never touch” list
The topic that survives all four is the gap. That intersection is the entire trick. No vibes, no “the AI just knows.” It’s the third pillar, made visible.
Then the research log adds the one thing a single scan can’t see: time. Because each run writes down what it found, the next run can say “engagement on this angle has climbed three weeks straight” or “this is the third scan in a row you’ve ignored this.” A one-off tool can’t do that. A system with memory can.
That’s the difference between a scraper and an editor.
What this is actually made of
Here’s the whole architecture. Four pieces. Knowing them upfront is enough to start building your own version, but it’s not required.
1. The scraper (Apify). Pulls the recent posts from each newsletter on your watchlist. Titles, full text, engagement metrics, publish dates. You set the list once and it runs on demand. The actor we’ll be using is called fatihtahta/substack-scraper, and it pulls straight from Substack’s public data. For five newsletters it usually costs around $0.60 a run and finishes in under 90 seconds. The exact cost moves with how many posts you pull and how much full text each one carries, since you pay per result. A heavy week might be a dollar, a light one thirty cents. Apify’s free plan gives you $5 of usage every month, which covers this comfortably.
2. The context layer (four plain-text files). This is the piece most people skip, and it’s exactly why their AI research comes out generic. Four highly specific markdown files that tell the agent who your audience is, what you’ve already covered, what you avoid, and what counts as a real gap for you. Skip this layer and every brief comes back as the same trending topics your competitors already wrote.
3. The orchestration layer (Claude + a skill file). One instruction file that tells Claude what to do, in what order, and what the output should look like. Claude reads your context, calls the scraper, and runs everything through your editorial lens.
4. The output and memory layer (two files in your project folder). Each scan drops a dated brief file into your folder, and appends one line to a running research log that lives right next to it. No second app, no database to name correctly. That log is what turns this from a one-time tool into a system that compounds.
Those four pieces are the whole thing. The section below shows exactly how to wire them together. No code. About 30 to 45 minutes of your time to set it up once. After that it runs from a single sentence, or on a schedule.
What you get below this line
The first half gave you the diagnosis: slow growth is almost never a writing problem, it’s the missing third pillar.
Below, I walk you through building the system that fixes it for your own niche. No code. No engineering background. Every file, every prompt, every step, including the messy parts. Like what to do when the first scan comes back looking generic, which it will, and how to fix it in 5 minutes.
This is the exact system I run every 2 weeks. It costs about $0.60 a scan. I’m not going back to doing it by hand.
Let’s build it.
I know it looks technical at first glance. It isn’t. I’ll walk you through each step like you’ve never done this before, because the first time someone told me to “just configure your MCP connector,” I had no clue what that meant either.
So bear with me, ok? Here’s what we’re building: a research agent that lives inside one Claude Cowork project, knows your newsletter cold, scrapes your competitors on command, remembers what it finds over time, and drops a clean brief straight into your project folder whenever you ask.
You need three things:
A Claude account (the $20/month Pro plan is fine for this)
An Apify account (free to start, Apify gives you $5 of usage a month, costs cents per run)
About 30 to 45 minutes, most of it spent on your own context
That’s it. No database setup, no second app to keep in sync. Now let’s dive in.
Step 0: Set up your project (do everything here)
This is the step that makes or breaks the build, and the original version of this guide skipped it. Do not run these steps across a bunch of separate chats. The whole thing lives in one place.
In Claude Cowork, create a new project called something like Substack Research. Connect it to a real folder on your machine.
