AI CEO LabInstagram Competitor Analysis: Scrape Them, Build A Remix Dashboard
Pull the last sixty posts from every creator you want to learn from, transcribe the reels, score each post against that account's own median, tag the hook mechanic, and put it all on one page you can filter. Apify does the scraping. The page does the thinking.
The first time I did this properly I pulled four hundred and twenty two posts from nine creators in my space and had a machine transcribe every reel. Then I sorted the whole thing not by views, but by how far each post was above its own account's normal. The page that came out of that changed what I made for the next month.
The finding was not subtle. Seventy seven percent of the top posts ended with a comment keyword. The winners were forty five to seventy five seconds of screen recording, or a green screen list. Nobody was winning with a polished talking head and a stat.
I could not have seen that by scrolling. Scrolling shows me what the algorithm wants me to see, in the order it wants, with no baseline. The dashboard shows me every post at once, scored honestly, with the transcript one click away.
This guide is how I build it. Apify does the pulling. A transcription API does the listening. A short script does the maths and writes one HTML page. No app, no subscription, no login. The same pipeline later pulled eight hundred and twenty eight reels across forty accounts for about two dollars.
The three levels
You follow your competitors, scroll their grids when you remember, and screenshot the ones that look big.
You have a list of accounts. A scraper pulls their last sixty posts with play counts, captions and video URLs. Reels get transcribed. Each post is scored against its account's median and tagged by hook mechanic. One page, filters by creator and tag, sorted by outlier ratio.
The scrape reruns monthly, new posts land in a table, the page rebuilds itself, and an agent hands you the five posts above five times their account's normal with the mechanic named and a remix angle in your voice.
The mental model
Scroll their grids and screenshot what looks big.
Pull everything, score every post against its own account's median, tag the mechanic, put it on one page, remix the mechanics in your voice.
| Role | Talks to you | Job |
|---|---|---|
| The scrape | First | Apify Instagram scraper, last 60 posts per account, plays, likes, comments, caption, video URL |
| The transcript | Per reel | Transcription API on the video URL, stored next to the post |
| The ratio | Per post | Plays divided by the account's median plays. Badges at 5x and 1.5x |
| The tag | Per post | One hook mechanic from caption plus transcript |
| The page | Always | One HTML file, sort by ratio, filter by creator and tag, transcript in a modal |
Scrape, transcribe, score, tag, page
1. Pick the accounts
Nine to fifteen accounts that sell to the same person you do or make the content you wish you made. Not the biggest names. The ones whose normal is close to yours, because their outliers are the ones you can actually copy. Write the handles in a list file.
2. Run the scraper
Apify's Instagram scraper takes profile URLs and a results limit. I ask for the last sixty posts per account. It returns the play count, likes, comments, caption, timestamp, product type and the video URL for reels. Each result costs a fraction of a cent. The token lives in an environment variable and is passed in the request, never typed into a chat.
3. Transcribe the reels
For every result with a video URL, send it to a transcription API and store the text with the post. In my nine creator pull, three hundred and forty eight of four hundred and twenty two posts came back with a transcript. The transcript is what lets you tag the mechanic and read the first spoken line without watching four hundred videos.
4. Score against the account's own median
Per account, take the median play count of its posts. Per post, ratio equals plays divided by that median. Badges: five times and above is an outlier, one and a half to five is above baseline, everything else is baseline. Median, not mean, because one viral post would hide every other winner on the account.
5. Tag the hook mechanic
A short rule list reads the caption and transcript and assigns one tag: job killer, free alternative, dashboard reveal, hidden feature, curated number, contrarian, absurd analogy, leaked secret, urgency, platform tactics, story, or other. Crude keyword rules are fine for a first pass. The point is to be able to filter the page by mechanic and see which ones your niche rewards.
6. Write the page
One static HTML file. Data baked in at build time, no client side API calls, no login. A spotlight row with each creator's single best post. Cards with thumbnail, handle, tag, plays, comments and ratio. Filters for creator and tag, sort by ratio, comments or plays. Click a card and the transcript opens in a modal with the permalink. Mark it noindex. It is a private swipe file.
7. Read it once, properly
Sort by ratio. Read the top twenty transcripts. Write down: the share that end with a comment keyword, the dominant format, the dominant length, and the mechanic that repeats. In mine it was the keyword gate at seventy seven percent, screen recording reveals and green screen lists, forty five to seventy five seconds. That paragraph is worth more than the page.
8. Remix, never copy
Take the mechanic, not the words. A 'free alternative' post about their tool becomes a 'free alternative' post about the thing your audience pays for. A dashboard reveal becomes your dashboard. Write it in your voice, plain, to one person. The transcript is there so you can see the structure, not so you can read it out.
Starter prompts
Paste these as written. They are short on purpose, because the long ones drift.
Build my competitor remix dashboard. Accounts: [handle1, handle2, ...]. Using the Apify Instagram scraper (actor apify/instagram-scraper) with the token from the APIFY_TOKEN environment variable, pull the last 60 posts per account with plays, likes, comments, caption, timestamp, product type and video URL. For every reel with a video URL, transcribe it with the transcription API and store the text with the post. Per account compute the MEDIAN plays; per post compute ratio = plays / account median; badge 5x and above as outlier, 1.5 to 5x as above baseline. Tag each post with one hook mechanic from caption plus transcript: job killer, free alternative, dashboard reveal, hidden feature, curated number, contrarian, absurd analogy, leaked secret, urgency, platform tactics, story, other. Write one static HTML page, noindex, data baked in: a spotlight row with each creator's top post, cards with thumbnail, handle, tag, plays, comments, ratio; filters by creator and tag; sort by ratio, comments, plays; a modal with the full caption, transcript and permalink. Then give me the report: per creator post count, median plays, median comments, top posts by ratio; the share of top 20 posts that end with a comment keyword; dominant format and length; the three mechanics that repeat; and five remix angles in my voice, plain words, one person. Never invent a number; if a field is missing say so.
Here is a post from the dashboard: caption [caption], transcript [transcript], ratio [Nx]. Name the mechanic in one line. Then write my version of the same mechanic about [my topic] for [my one person], in short lines, plain words, no stats, no borrowed sentence. Give me three first lines and pick one.
Rescrape the same accounts for posts newer than [date], append only new rows, recompute each account's median with the new posts included, rebuild the page, and tell me anything new above 5x with its mechanic.
From a one off pull to a pipeline
The first pull can be done by hand through the Apify console and a chat window. The version that runs monthly needs three things to change.
The scrape becomes an API call
Instead of clicking Start in the console, the script posts the input to the actor's run endpoint with the token from the environment and reads the dataset items back. Same actor, same input, no clicking.
The rows get a home
A table keyed by post id (I use a Supabase table with the account, shortcode, plays, caption, transcript, ratio and scraped date) so a rescrape appends and never duplicates. A JSON file works at small scale.
The page is generated, not edited
A build script reads the rows, tags, scores and writes the HTML. You never open the HTML in an editor. Rerun the script and the page is current.
Give it to your agent, three ways
Same skill, three worlds. Pick the one you actually use. The skill file at the bottom of this page is the instructions in every case.
An agent with connectors (Claude Desktop, Claude Code, Grok Bot)
Apify publishes an MCP server, so Claude can run the scraper directly and read the dataset back. Large datasets should be saved to a file and processed with a script rather than read into the chat.
- Claude Code: add the Apify MCP with the command below, authenticate with your Apify token when prompted. Save the skill file from the bottom of this page.
- Say "build my remix dashboard" with the handles. The agent calls the actor, saves the dataset to a file, runs the transcription and scoring in Python, and writes the page.
- Claude Desktop: add the same MCP as a connector in Settings. For the transcription and page build it will still need a shell, so this world is best in Claude Code.
- Grok Bot and Hermes: same skill, same MCP or the REST calls in the API block. They need the two tokens in their environment.
claude mcp add --transport http apify https://mcp.apify.com # then /mcp to authenticate export DEEPGRAM_API_KEY=... # transcription, env only
ChatGPT (a project or a custom GPT)
ChatGPT has no Apify connector, so run the scrape in the Apify console or with the curl below and bring the file to it. ChatGPT's data analysis can do the medians, ratios, tags and even write the HTML.
- Layer 1: in the Apify console run the Instagram scraper with your profile URLs, results type posts, results limit 60. Download the dataset as JSON. Create a ChatGPT Project with the skill below as instructions, upload the JSON, say "build my remix dashboard". It computes medians and ratios, tags mechanics from captions, and writes the page as a downloadable HTML file. Transcripts will be missing unless you add them.
- Layer 2: create a custom GPT with an Action against the Apify run sync endpoint (OpenAPI stub below), API key in the query as token. The GPT can then trigger the scrape itself. Transcription still needs a separate call or the transcript actor.
- Either way, demand the findings paragraph and the five remix angles, and refuse any number it cannot point at in the file.
openapi: 3.1.0
info: {title: Apify Run, version: '1'}
servers: [{url: https://api.apify.com/v2}]
paths:
/acts/apify~instagram-scraper/run-sync-get-dataset-items:
post:
operationId: scrapeInstagram
requestBody:
content:
application/json:
schema:
type: object
properties:
directUrls: {type: array, items: {type: string}}
resultsType: {type: string}
resultsLimit: {type: integer}
onlyPostsNewerThan: {type: string}Anything with an API (a token and a curl call)
The whole scrape is one POST to the Apify REST API. The token comes from an environment variable in the query string. The response is the dataset items as JSON.
- Export APIFY_TOKEN in your shell.
- Post the input below to the run sync endpoint. It waits for the run and returns the items. For big pulls, use the async run endpoint and fetch the dataset by id afterwards.
- For every item with a videoUrl, send the URL to your transcription API and store the text with the item.
- Run the scoring, tagging and page build script on the file. Rerun monthly with onlyPostsNewerThan set to the last scrape date.
export APIFY_TOKEN=... # env only
cat > input.json <<'J'
{"directUrls": ["https://www.instagram.com/handle1/", "https://www.instagram.com/handle2/"],
"resultsType": "posts", "resultsLimit": 60}
J
curl -s -X POST "https://api.apify.com/v2/acts/apify~instagram-scraper/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
-H 'Content-Type: application/json' -d @input.json > posts.json
# per account: median of videoPlayCount; per post: ratio = videoPlayCount / median
# transcribe each videoUrl with your transcription API, then build the pageFailure modes
Every one of these has happened to me or to someone I set this up for.
| Failure | Fix |
|---|---|
| Sorted by plays and learned who has the biggest audience | Sort by ratio to the account's own median |
| Mean baseline made one account look like it had no winners | Median, always |
| Skipped transcripts and tagged from captions alone | Captions are the CTA; the transcript holds the hook |
| Copied a winning caption with the nouns swapped | Mechanic only; write the words yourself, in your voice |
| Token pasted into a chat to get it working | Environment variable, and rotate the one you pasted |
| Dashboard built as an app with a login nobody used | One static HTML file, noindex, open it from a bookmark |
| Pulled fifteen accounts of million follower creators | Pull accounts whose normal is close to yours; their outliers are copyable |
| Scrape ran once in a burst of enthusiasm | Monthly refresh appending new posts, or it decays into a screenshot folder |
The tools I use for this
| Tool | What it is for here | |
|---|---|---|
| Apify | Runs the Instagram scraper. Profile URLs in, posts with plays, captions and video URLs out. | no link, just use it |
| Deepgram | Transcribes the reels so the hook can be read and tagged without watching. | no link, just use it |
| Supabase | The table the rows live in so a rescrape appends instead of duplicating. | no link, just use it |
| Claude | Runs the pipeline, writes the page, names the mechanics and drafts the remixes. | Open |
| Blotato | Where the remixed posts get scheduled once written. | Get it |
The free skill
It is the competitor swipe file build I run, written for an agent. It takes a list of accounts, runs the Instagram scraper through Apify, transcribes the reels, computes each post's ratio to its account's median plays, tags the hook mechanic from caption and transcript, writes one static HTML page with filters and a transcript modal, and returns the report: per creator medians, the top posts by ratio, and the patterns worth remixing in your own voice.
--- name: instagram-remix-dashboard description: Builds a private competitor swipe file from Instagram: scrapes the last 60 posts per account through Apify, transcribes the reels, scores each post against its own account's median plays, tags the hook mechanic, writes one static HTML page with filters and a transcript modal, and reports the patterns worth remixing in the user's voice. Trigger on "build my remix dashboard", "scrape my competitors", "what are my competitors' best posts", "refresh the creator vault". --- # Instagram Remix Dashboard You are building a scored, transcribed, filterable swipe file. Plays alone measure an account's audience. Plays divided by that account's median measure the post. You remix mechanics, never words. ## Before you start - Accounts: 9 to 15 handles. Prefer accounts whose normal is close to the user's; their outliers are copyable. Ask if not given. - Apify token in the environment (APIFY_TOKEN). Transcription key in the environment. If either is missing, stop and say where to set it. Never accept a token in the chat; if one was pasted, tell the user to rotate it. - A place for rows keyed by post id (a table or a JSON file) so reruns append. ## Step 1: Scrape Actor: apify/instagram-scraper. Input: directUrls = profile URLs, resultsType = posts, resultsLimit = 60 (on refresh add onlyPostsNewerThan = last scrape date). Keep per item: id, shortCode, url, productType, timestamp, videoPlayCount, likesCount, commentsCount, caption, videoUrl, displayUrl. Save the dataset to a file. Do not read a large dataset into the conversation; process it with a script. ## Step 2: Transcribe For every item with a videoUrl, send it to the transcription API and store the text with the post. Note how many came back; report the count. Missing transcripts are reported, never invented. ## Step 3: Score Per account: median of videoPlayCount (median, not mean). Per post: ratio = videoPlayCount / account median. Badges: 5x and above = outlier; 1.5 to 5x = above baseline; else baseline. ## Step 4: Tag one hook mechanic From caption plus transcript, assign exactly one: job killer, free alternative, dashboard reveal, hidden feature, curated number, contrarian, absurd analogy, leaked secret, urgency, platform tactics, story, other. Keyword rules are acceptable for a first pass. State that the tags are rule based. ## Step 5: Write the page One static HTML file, `<meta name="robots" content="noindex, nofollow">`, data baked in, no client side API calls, no login: - spotlight row: each creator's highest ratio post - cards: thumbnail, handle, tag, plays, comments, ratio badge, caption excerpt, permalink - filters: creator, tag; sort: ratio, comments, plays - modal: full caption, transcript, permalink ## Step 6: The report 1. Per creator: post count, median plays, median comments, transcript count, top posts by ratio with both numbers. 2. Of the top 20 by ratio: share ending with a comment keyword, dominant format (screen recording, green screen, talking head, carousel), dominant length. 3. The three mechanics that repeat, with two example posts each. 4. Five remix angles for the user: mechanic named, their topic, their one person, short plain lines, no stats, no borrowed sentence. ## Rules - Never invent a number. A missing field is reported as missing. - Mechanics are shared; sentences are theirs. - Refresh appends by post id and recomputes medians with the new posts included.
What done looks like at thirty days
- Nine or more accounts scraped at sixty posts each with the token in an environment variable
- Every reel has a transcript stored next to the post
- Every post has a ratio to its account's median and one mechanic tag
- One private HTML page with filters, sort and a transcript modal exists and you have opened it more than once
- The findings paragraph is written: keyword share, format, length, repeating mechanics
- Three posts published this month borrowed a mechanic and zero borrowed a sentence
Want the dashboard built on your niche with the remix scripts written?
Inside the AI CEO Lab the Research to Content Engine module runs this scrape on your competitors and turns the top ten into scripts in your voice.