I watch a lot of long-form podcasts and interviews.

A good 90-minute conversation can contain 15 ideas worth keeping: a framework, a story, a counterargument, a throwaway line that explains how someone actually thinks.

But a week later, most of it is gone.

The usual answer is note-taking. That works if you are watching a lecture. It works badly for podcasts.

Podcasts are not just containers for ideas. They are live reasoning traces. The useful part is often not the final answer, but the path that produced it:

• the question that provoked the idea
• the pushback that sharpened it
• the story that made it believable
• the reframe where the original question changed meaning
• the side detail that made the conversation feel situated in a real life

That is the part most summaries lose.

Most tools turn a conversation into something like this:

Episode Topics
+-- Entrepreneurship
+-- India
+-- AI investing
+-- Global companies

That looks organized, but it quietly destroys the thing I cared about.

Take the Nikhil Kamath episode with Martin Escobari. One of the interesting moments is not simply "India and global companies." The interesting moment is that the phrase "global company" gets contested.

Nikhil asks why India has not built a global company. Martin points to Infosys and BPO companies. Nikhil pushes back: that is services and labour arbitrage, not a product company. Martin partially accepts the distinction. The question then becomes more precise: why has India not built a global product brand?

The insight is not one line. It is the movement:

question
  answer
    pushback
      partial concession
        reframe
          revised answer

A topic cluster cannot show that. A generic summary usually compresses it into a sentence. But the whole value is in the conversational shape.

So the agent needed to preserve conversation flow, not just extract topics.

The obvious way to build this is as a classic software pipeline:

YouTube URL
  -> fetch transcript
  -> detect format
  -> induce schema
  -> parse transcript into JSON tree
  -> render markdown / React tree

The interesting step is schema induction. Before parsing, the agent reads the transcript and discovers the conversation move types that exist in that episode: question, answer, pushback, reframe, partial_concede, personal_story, concept_unlocked, and so on.

That idea is still good.

But building it this way, I realized the product was not the schema. The product was the loop.

The thing I actually wanted was:

1. Give the agent a YouTube podcast URL.
2. Get back something readable and useful.
3. Notice what it missed.
4. Update the right skill.
5. Rerun the affected stages.
6. Watch the output get better.

A pure pipeline over-centers the data model. What I actually wanted centers the workflow.

The current setup has two agents in the temp/ folder:

• youtube-podcast-agent runs the podcast processing pipeline.
• youtube-podcast-refine updates the skills when the output is missing, shallow, or structurally wrong.

The main agent produces one markdown file with exactly three sections:

# Context
# Summary
# Conversation Tree

That sounds simpler than a nested JSON tree, but it is much more useful as a daily tool.

The output is not just a summary. It answers three different needs.

The context_extraction skill asks the agent to extract background before it explains the ideas.

This matters because podcasts are full of assumed context. People mention cities, universities, funds, cultural habits, food, media, neighborhoods, and institutions as if every listener shares their world.

I do not want the agent to pretend the listener is a generic internet person. The skill is explicitly written for a reader in Pune, India.

So the Context section includes:

• where the conversation takes place
• who the speakers are
• what their lifestyle looks like
• cultural references that may need explanation from an Indian vantage point
• ambient details like food, setting, objects, background sounds, and the physical feel of the conversation

This turned out to be more important than I expected.

A conversation is easier to remember when it has a world around it. "Martin talked about volatility" is abstract. "Martin, a Bolivian-born investor shaped by hyperinflation, Harvard, Brazil, and General Atlantic, talks about volatility over dinner after a studio conversation in New York" sticks differently.

Context gives the ideas a place to live.

The summarize skill is deliberately strict.

It does not ask for "topics discussed." It asks for 8 to 9 standalone bullets capturing the key ideas, tensions, positions, and disagreements.

The rules are small but important:

• prioritize contested, surprising, counterintuitive, or specific ideas
• use the speaker's own framing when it matters
• if two speakers disagreed, state both positions
• put the episode's most important insight first
• do not produce a bland list of subjects

That last rule is doing a lot of work.

Bad summaries say:

- They discussed entrepreneurship, AI, India, and investing.

Useful summaries say:

- Martin argues that volatility is an investor's friend only if the company is fit enough to survive until a once-in-a-generation opportunity appears.

The second version gives me an idea I can reuse.

The parse_transcript skill builds the actual conversation tree in markdown.

The output uses numbered topics and numbered conversational moves:

## 7. Entrepreneurs, Gangsters, and Trauma

### 7.1 Nikhil: Entrepreneurs are like gangsters
...

### 7.2 Martin: You're the Bollywood of entrepreneurship
...

### 7.3 Nikhil: And if I ask "why" again?
...

Each H2 is a developed topic. Each H3 is a conversational move: a question, an answer, a pushback, a reframe, a partial concession, a personal story, or a digression.

The key rule is simple:

Never collapse a pushback-reframe-concession chain into one block.

That is the whole point.

If the host challenges an answer, that is a node. If the guest accepts part of the challenge and holds ground on the rest, that is a node. If the question changes because of the challenge, that is a node.

The tree is not trying to be elegant. It is trying to be faithful.

This project made the tool vs skill distinction much clearer for me.

A tool is a thing the agent can call:

fetch_transcript(url)
write_output(markdown)

A skill is not a callable function. It is the judgment layer.

The skill says what counts as important, what must not be collapsed, what kind of context the reader needs, what the summary should prioritize, and what "good" output means.

For example, the conversation tree skill says:

Every distinct conversational move gets its own H3:
a question, an answer, a pushback, a reframe,
a partial concession, a personal story, a digression.

That is not code. But it changes the output more than code often does.

The agent becomes useful when the skills encode taste.

The second agent, youtube-podcast-refine, exists because the first output is never perfect.

After reading an output, I might notice:

• the Context section missed food or setting details
• the Summary became too generic
• the Conversation Tree collapsed a disagreement into one paragraph
• a personal story was treated as filler
• a one-minute digression deserved its own topic

The refinement agent does not just "try again."

It asks: which skill caused this failure?

context_extraction.md -> context, lifestyle, Pune-specific notes, ambient details
parse_transcript.md   -> H2/H3 structure, pushback, reframes, sub-threads
summarize.md          -> summary specificity, disagreement, bullet priority

Then it proposes a minimal change to that skill and waits for confirmation.

Only after the change is accepted does it rerun the affected stages and everything downstream.

context skill changed -> rerun context, tree, summary, output
parse skill changed   -> rerun tree, summary, output
summary skill changed -> rerun summary, output

That matters because it keeps improvement inspectable.

The agent does not get better because I make the prompt longer every time. It gets better because each observed failure becomes a small rule in the right place.

The system is best described as a skill-driven podcast processor:

YouTube URL + output name
  -> fetch transcript
  -> extract context
  -> build conversation tree
  -> summarize from the tree
  -> write markdown

feedback
  -> identify responsible skill
  -> propose minimal skill update

Schema induction is still a useful idea, and the Python CLI still contains that pipeline. But the more important lesson is that an agent needs editable judgment.

Hardcoded categories are brittle. A one-shot prompt is brittle. A beautiful UI over a shallow parse is still shallow.

Skills give the system a memory of what "better" means.

The most useful agent architecture is not always the most technically exotic one.

For this project, the winning pieces were boring in the best way:

• markdown skill files
• explicit stages
• cached transcripts
• deterministic output structure
• small refinement diffs
• rerunning only the affected downstream work

The magic is not that the agent can summarize a podcast. Every LLM can summarize a podcast.

The magic is that when I say, "this output missed the wife story," or "this pushback got flattened," the system has somewhere to put that lesson.

It can update parse_transcript.md.

It can rerun the tree.

It can make the next episode better.

That is the difference between a prompt and an agent I can keep using.

The end state is not a perfect universal podcast summarizer.

The end state is a personal knowledge tool with my preferences baked into it:

• preserve friction
• preserve stories
• explain unfamiliar context from my vantage point
• do not flatten disagreements
• do not throw away ambient details
• summarize positions, not topics

That is why this agent is built around skills.

Not because skills are fancy. Because taste has to live somewhere.