knowledge-base-framework/skills/knowledge-base-ingest-resource/SKILL.md

name, description

name

description

knowledge-base-ingest-resource

Ingest an external resource (local file, URL/webpage, PDF, YouTube video, etc.) into this knowledge-base vault (PARA-organized). Copies the original into 03_resources/<resource-name>/raw/, reads the content, breaks it into individual topics, and wires each topic to related notes already in the vault (internal typed markdown links) and to outside sources (URLs). For YouTube, pulls the transcript into raw/. Use when the user says "ingest", "add this resource", "save this link/video/paper", or hands you a file/URL to capture and link.

Ingest a Resource

Capture an external resource into the vault as a reference note, store its original bytes under raw/, decompose it into topics, and link those topics into the knowledge graph. Follow every step.

Obey AGENTS.md and 02_areas/ways-of-working.md (read them if not already loaded this session; if ways-of-working.md does not exist yet, skip it).

The output is a type: note reference note at 03_resources/<resource-name>/index.md, one type: note topic note per distinct topic at 03_resources/<resource-name>/<topic-name>.md, and the untouched original preserved in 03_resources/<resource-name>/raw/ — all linked into the graph with typed markdown links ([title](id){"type":"…","link-info":"…"}).

03_resources/<resource-name>/
  index.md            # the resource note (summary, source, topic index)
  <topic-name>.md     # one note per topic distilled from the resource
  <topic-name>.md
  raw/                # original bytes / transcript / fetched page

1. Identify the resource and its type

The resource comes from the conversation — a file path, a URL, a pasted link, or an attachment. Classify it; this drives how you capture it in step 3:

Type	Signal	Capture into raw/
Local file	a path that exists on disk	copy the file verbatim
YouTube video	host is youtube.com / youtu.be	pull the transcript (+ metadata)
Webpage / link	any other http(s) URL	save the fetched page (markdown/html) + URL
PDF / doc	path or URL ending in .pdf/.docx/...	copy/download the binary verbatim

If the resource type is genuinely ambiguous (e.g. a bare string that could be a slug or a search term), ask the user. Otherwise proceed.

2. Derive the resource name and create the folder

Pick a short kebab-case <resource-name> from the title / filename / video title. Keep it descriptive and unique (e.g. cyclonedx-sbom-spec, oauth2-rfc6749).

ls 03_resources/                      # check the slug is not already taken
mkdir -p 03_resources/<resource-name>/raw

If the slug already exists, either you are re-ingesting (update in place, tell the user) or pick a more specific slug.

3. Copy the original into raw/

Always preserve the source bytes untouched. Pick the branch that matches step 1.

Local file — copy verbatim, keep the original name:

cp "<source-path>" "03_resources/<resource-name>/raw/"

PDF / binary at a URL — download verbatim:

curl -fsSL "<url>" -o "03_resources/<resource-name>/raw/<filename>"

Webpage / link — record the URL and save the readable content. Use WebFetch to retrieve it (request markdown), then write the result and a source.url:

printf '%s\n' "<url>" > "03_resources/<resource-name>/raw/source.url"

Save the fetched body to raw/page.md (write the WebFetch output to that file).

YouTube video — pull the transcript into raw/. Record the URL first, then try, in order:

printf '%s\n' "<url>" > "03_resources/<resource-name>/raw/source.url"

# Preferred: yt-dlp grabs subs + metadata without downloading the video
yt-dlp --skip-download --write-subs --write-auto-subs --sub-lang en \
  --convert-subs srt --write-info-json \
  -o "03_resources/<resource-name>/raw/%(title)s.%(ext)s" "<url>"

# Fallback if yt-dlp is absent: python transcript API
python3 -c "import youtube_transcript_api" 2>/dev/null && \
  python3 -c "from youtube_transcript_api import YouTubeTranscriptApi as A; \
import sys; vid='<VIDEO_ID>'; \
open('03_resources/<resource-name>/raw/transcript.txt','w').write(\
'\n'.join(s['text'] for s in A.get_transcript(vid)))"

If neither tool is installed, tell the user which to install (pip install yt-dlp or pip install youtube-transcript-api) and stop before fabricating a transcript. Never invent transcript text.

4. Read the captured content

Read what you just stored in raw/ (the file, page.md, the transcript). This — not the live URL — is the source of truth for the rest of the steps, so the note matches the bytes we preserved. For long content, read enough to identify every distinct topic, not just the intro.

5. Decompose into topics and find links

Reason about the individual topics inside the resource — the concrete concepts, standards, formats, tools, or entities it discusses. For each topic, decide whether the vault already knows about it, and whether it points outward.

For every candidate topic, search the vault for a matching note:

# free-text over the graph (id / title / path / tags)
jq -r --arg q "<topic>" '.nodes[]
  | select((.id|ascii_downcase|contains($q)) or (.title|ascii_downcase|contains($q)))
  | "\(.id)\t\(.path)"' index/graph.json

# by tag
jq -r --arg t "<topic-tag>" '.nodes[] | select(.tags[]? == $t) | "\(.id)\t\(.path)"' index/graph.json

# full-text fallback across the PARA buckets
grep -rIl "<topic>" 01_projects 02_areas 03_resources 04_archives

Prefer the knowledge-database MCP server when available (graph_search_nodes, graph_neighbors, read_note) — it is the same index. Open promising matches and confirm they are really the same concept before linking.

Classify each topic:

• Internal — the vault has a note for it → link with [<title>](<note-id>){"type":"…"}.
• External — no vault note, but it has an authoritative source (a spec, RFC, vendor page) → record the URL so it can become a note later.
• Unlinked — neither yet → list it as a candidate so a future note can fill it.

Example: a CycloneDX SBOM resource that mentions CPE and purls → search the vault for cpe, purl, sbom; link [text](id){"type":"…"} to whichever notes exist, and capture the CPE/purl spec URLs as external for the rest. A resource describing CycloneDX vendor references → reason about which existing supplier/vendor or component-identity notes it relates to and link those.

6. Write a note per topic

Each distinct topic from step 5 becomes its own note at 03_resources/<resource-name>/<topic-name>.md (kebab-case slug). This is what makes topics first-class, individually linkable graph nodes.

---
id: resource/<resource-name>/<topic-name>
title: <Topic Title>
type: note
status: active
tags: [resource, <topic-name>, <extra-tags...>]
source: <original url or path>
ingested: <today, YYYY-MM-DD>
last_reviewed: <today, YYYY-MM-DD>
---

# <Topic Title>

<What the resource says about this topic — a few sentences, in your words.>

## In this resource
From [<resource title>](resource/<resource-name>){"type":"part-of","link-info":"<how the topic appears / why it matters here>"}

## Related
- [<title>](<existing-vault-note-id>){"type":"related","link-info":"<what to expect there>"}   # internal
- <https://authoritative-spec-url>   # external — spec / RFC / vendor page

Links are markdown links whose target is the note id, with an optional {"type":"…","link-info":"…"} brace. type must be one of the controlled vocabulary (the pre-commit hook rejects others): related, supersedes, superseded-by, depends-on, part-of, parent, derived-from, see-also, source. Omit the brace for a plain related link. link-info is what the source needs from the destination — "what to expect there".

• One topic = one file. Skip trivia; write a note only for topics worth recalling.
• If the vault already has a note for the topic (an internal match from step 5), do NOT duplicate it — link it from the resource note instead and, if useful, add a backlink (step 7). Create a topic note here only for topics the vault does not yet cover.
• Link each topic note back to its resource with [<resource title>](resource/<resource-name>){"type":"part-of"}, and outward to internal notes (same link form) and external sources (URLs).
• Legacy [[wiki links]] are rejected by make validate-graph — never write them.

7. Write the resource note (index)

Filename: 03_resources/<resource-name>/index.md. This is the hub that ties the raw source to the topic notes.

---
id: resource/<resource-name>
title: <Human Title>
type: note
status: active
tags: [resource, <topic-tags...>]
source: <original url or path>
ingested: <today, YYYY-MM-DD>
last_reviewed: <today, YYYY-MM-DD>
---

# <Human Title>

## Summary
<2–5 sentences: what this resource is and why it is worth keeping.>

## Source
- Type: <local file | webpage | youtube | pdf | ...>
- Origin: <url or original path>
- Raw: `03_resources/<resource-name>/raw/`

## Topics
- [<Topic>](resource/<resource-name>/<topic-name>){"type":"part-of","link-info":"<one line>"}   # new topic note
- [<Topic>](<existing-note-id>){"type":"related","link-info":"<one line>"}                       # vault already had it
- **<Topic>** — <one line> → <https://external-spec-url>                                          # external only
- **<Topic>** — <one line> (no note yet)                                                          # candidate

## Related
- [<title>](<related-note-id>){"type":"related","link-info":"<what to expect there>"}

• ingested and last_reviewed = today (the currentDate from session context).
• type must stay within the vault taxonomy enum (hub|project|area|guideline|policy|feature|roadmap|decision|note|meta|reference), which generate-graph.py --validate enforces — use note; the resource tag is what marks it.
• Markdown link targets resolve by id, rel-path, title, or unique stem — always use the id as the target, e.g. [Human title](resource/foo/bar).
• Do not leave a section empty; write None. if a section truly has nothing.

8. Wire backlinks

So the resource is reachable in the graph, not just reaching out:

• Add a [<title>](resource/<resource-name>) (or a relevant topic note) link from the most relevant existing notes (the strong internal topics from step 5) — e.g. under a ## Resources or ## Related section, creating it if absent. Add a {"type":"…","link-info":"…"} brace where the relationship is more specific than related. Use individual edits.
• If several resources accumulate, maintain 03_resources/index.md as a hub that lists them, and link the new note from there (create the hub if it does not exist, with type: hub, tags: [resource, hub]).

9. Regenerate and validate the graph

make graph

This regenerates index/graph.json and validates links and edge relationship types. If validation reports unresolved links, fix the targets (typos, wrong id, or a link to a note that does not exist yet — either create it or downgrade it to plain text / external URL); if it reports an unknown edge type, correct it to a value in the controlled vocabulary. Re-run until clean. The pre-commit hook rejects a stale or invalid graph, so this must pass.

10. Stage (do not commit unless asked)

Per ways-of-working.md, NEVER use git add -A or git add .. Stage each touched path individually:

git add 03_resources/<resource-name>/        # note + raw/
git add index/graph.json
git add <any-existing-note-you-backlinked>.md

Only commit or open a PR if the user explicitly asks. Changes go via small, clear pull requests.

11. Report

Summarize: the resource id/path, type, what landed in raw/, the topic notes you created (<topic-name>.md), which topics were linked to existing vault notes (typed markdown links) vs. captured as external vs. left as candidates, files touched, and the graph-validation result.