Files
personamanagmentlayer__pcl/stdlib/tools/document-processing-expert/SKILL.md
ANGX 4e07a6a874 feat(stdlib)!: Agent Skills v1.0 conformance, resolver fix, and 17 new skills (#43)
* chore(stdlib): add conformance tooling for the skills library

The library had no gate that could express what a valid skill is, so
malformed frontmatter, over-budget files and credential literals went
unnoticed. These scripts define and enforce that contract.

- validate-skills.py: extend with Agent Skills v1.0 conformance -
  naming rules and directory agreement, an explicit activation trigger
  in the description, canonical layout, the 500-line SKILL.md budget,
  relative-link integrity, and two security rules (no unrestricted
  Bash(*) grant, no credential literals). Structural checks now read a
  skill as a whole so material moved into references/ still counts.
- split-skill-references.py: move the heaviest sections of an oversized
  SKILL.md into references/. Verifies line by line that every source
  line lands in exactly one output and refuses to write otherwise.
- generate-skill-inventory.py: derive SKILLS_INVENTORY.md and
  DIRECTORY_STRUCTURE.md from the tree so they cannot drift.
- generate-skill-catalog.py: index the canonical layout only, count
  references/ toward a skill's size, and stop truncating descriptions
  at 200 characters - the catalog is the discovery surface and must
  carry the activation trigger.
- .gitignore: Python bytecode from these scripts, and local .pcl state.

Reviewed-by: Security Analyst, Compiler Engineer, DevX Engineer

* fix(stdlib): bring all 174 skills to Agent Skills v1.0 conformance

No skill in the library was conformant. 41 of 175 failed the existing
validator outright, and none satisfied the spec. This pass fixes the
tree and records the result. See stdlib/CHANGELOG.md for the per-skill
versioning and migration notes.

BREAKING CHANGE: one skill removed, five moved, seven with a capability
withdrawn. Library version 2.0.0 -> 3.0.0.

Structure
- Convert 71 skills stored as flat files to <category>/<name>/SKILL.md,
  the layout the spec requires and the only one able to carry companion
  resources.
- Rename 5 directories that disagreed with their declared name:, so
  resolution by path and by name stop giving different answers. The
  repository docs already referenced the corrected paths.
- Merge the duplicate legal-technology skills into legaltech-expert and
  delete legal-tech-expert, carrying over everything the survivor
  lacked.

Metadata
- Repair 20 unparseable frontmatters: an H1 preceded the --- block and
  the key was skill_id: instead of name:, so any parser reading
  frontmatter at offset 0 saw no metadata at all. The objectives,
  prerequisites and outcome they carried become real body sections.
- Add the missing version: to 21 skills, fold keywords into tags, move
  expertise_level under metadata, and align 10 category: values with
  their directory.

Discoverability
- Give all 174 descriptions an explicit "Use when ..." trigger, built
  from each skill's own tags and section headings. 152 previously said
  only what the skill was about, leaving an agent nothing to match on.

Progressive disclosure
- Split 127 oversized skills, moving bulk material into references/ read
  on demand and leaving an index behind. A SKILL.md now averages 277
  lines instead of 644 and none exceeds 500. Verified lossless line by
  line against the previous revision.

Security
- Withdraw the unrestricted Bash(*) grant from 7 skills; none of the six
  non-financial ones contained a single shell command. finance-expert
  documents code that charges cards and reads bank accounts, so it now
  grants no shell at all.
- Remove credential literals: Stripe keys, two JWT signing keys, an
  Express session secret, a Redis password and WiFi credentials, all
  inlined in examples. They load from the environment now.
- finance-expert: idempotency keys were documented but absent, so a
  retry double-charged; webhook signature failures were not caught, so a
  forged event raised uncaught instead of being rejected;
  int(amount * 100) truncated fractions of a cent. All fixed, plus a
  money-movement guardrails section.
- trading-expert: place_order routed without any pre-trade risk check
  although a RiskManager sat unused in the next section. Risk validation
  now runs before routing and raises rather than returning a boolean a
  caller can ignore; live routing is opt-in.
- Correct an Ansible Vault example that showed plaintext under an
  "encrypted" label.

Also fixes 8 broken documentation links and a prematurely closed code
fence in skill-creator-expert that leaked template content into prose.

Per the repository file-organization policy, four stale progress reports
(SKILLS_MANIFEST, COMPLETE_SKILLS_LIBRARY, COMPLETE_LIBRARY_MANIFEST,
PHASE_3_DATA_ANALYTICS) move out of stdlib/ into the gitignored
.roadmap/status/stdlib/; they claimed 31 of 100 skills complete. They
appear here as deletions because .roadmap/ is not tracked.

Reviewed-by: Security Analyst, Language Designer, DevX Engineer,
             Documentation Specialist

* fix(skills): resolve standard library skills from their real layout

No stdlib skill was resolvable. SkillResolver looked for
`<stdlibDir>/<name>.md` with stdlibDir defaulting to `stdlib/skills`, a
directory that does not exist, and the path ignored the category level
the library actually uses. The two LSP providers had the same wrong
directory and scanned only one level deep, so neither completion nor
go-to-definition could see a skill either.

- skill-discovery.ts: enumerate skill entry points across the shapes in
  use - a flat `<name>.md`, `<name>/SKILL.md`, and
  `<category>/<name>/SKILL.md` - while skipping references/, scripts/
  and assets/, which hold supporting material rather than entry points.
- skill-resolver.ts: point stdlibDir at `stdlib`, and look the name up
  as a directory under any category as well as the two legacy shapes, so
  existing custom stdlib directories keep working. The name is used as a
  single path segment and rejected if it contains a separator or
  traversal, so a reference cannot escape the stdlib root.
- Rewire both LSP providers onto the shared discovery helper.

Tests: 13 new cases covering the layouts, ordering, the references/
exclusion, path-traversal rejection, and an integration test that
resolves real shipped skills by bare name.

Reviewed-by: Compiler Engineer, Runtime Architect, Security Analyst

* feat(stdlib): add 17 skills covering the library's largest gaps

Selection was driven by cross-referencing coverage against observed usage
rather than by intuition: Stack Overflow and GitHub Octoverse data for
which technologies matter, and install rankings on public agent-skill
registries for the shape of skills people actually use.

That comparison found a mismatch of form, not only of coverage. The
most-installed skills in the ecosystem are procedures - tdd, refactoring,
frontend design, browser automation - while all 174 existing skills were
domain knowledge shaped as <domain>-expert. The new workflows/ category
closes that gap and introduces the <activity>-workflow convention.

New category workflows/ (4)
- tdd-workflow, refactoring-workflow, debugging-workflow,
  code-review-workflow. code-review-workflow complements rather than
  replaces code-review-expert: procedure versus judgement. Merge them if
  the split proves confusing in use.

Applied AI (3) - the largest coverage gap
- llm-engineering-expert, rag-expert, agent-engineering-expert.
  The library referenced LLMs in tags but covered none of it: pgvector
  appeared zero times across 174 skills, LangChain once, MCP twice
  despite the repository shipping an MCP integration guide.

Capabilities (2)
- document-processing-expert (PDF, DOCX, XLSX, PPTX in one skill),
  browser-automation-expert (a client against third-party sites,
  distinct from playwright-expert which tests your own application).

Data (3)
- mysql-expert, the second most used database and previously absent
  while rarer stores had skills; analytical-databases-expert;
  stream-processing-expert.

Security (3)
- secrets-management-expert, supply-chain-security-expert (OWASP A03
  2025), identity-access-expert (OWASP A01).

Interface and meta (2)
- accessibility-expert: design/ held only design-expert, which despite
  its name covers software architecture, so WCAG had no home.
- skill-router: find and compose skills across 191, with a
  disambiguation table for overlapping pairs.

Grouping by decision context rather than by technology keeps the growth
to 17 skills for roughly 50 topics. Five vector stores live inside
rag-expert; four document formats inside document-processing-expert.
Per-tool depth sits in references/, which the progressive-disclosure work
in 3.0.0 made cheap.

Four further gaps are better served by extending existing skills -
PyTorch and TensorFlow in ml-expert, OpenTelemetry in monitoring-expert,
Pulumi and Crossplane in terraform-expert - and are not in this change.

Library 3.0.0 -> 3.1.0, additive only. Adds the workflows label to the
inventory generator; catalogue, inventory, structure and CHANGELOG
regenerated.

Reviewed-by: Product Strategist, Security Analyst, DevX Engineer,
             Documentation Specialist

* fix(test): create test directories with mkdtempSync

CodeQL flagged js/insecure-temporary-file (CWE-377) on both suites.
`join(tmpdir(), <predictable name>)` followed by mkdirSync creates a
world-readable directory at a guessable path, so another local user can
pre-create it or plant a symlink between the name being computed and the
directory being created.

mkdtempSync creates the directory atomically with a random suffix and
0700 permissions, which closes both the disclosure and the race.

One occurrence is new in this branch (skill-discovery); the other
pre-dates it in skill-resolver and is fixed in the same pass rather than
left as a known finding in a file this branch already touches.
2026-09-05 04:35:30 +02:00

9.9 KiB

name, version, description, category, tags, allowed-tools, metadata
name version description category tags allowed-tools metadata
document-processing-expert 1.0.0 Read, generate and modify office documents and PDFs from code: PDF extraction and forms, Word documents, Excel workbooks and PowerPoint decks. Use when the user mentions PDF, DOCX, XLSX, PPTX, Word, Excel, PowerPoint or spreadsheets, wants data extracted from documents, needs a report or invoice generated as a file, must fill a form, or when the task involves parsing scanned documents, merging or splitting files, or producing a document a person will open. tools
pdf
docx
xlsx
pptx
documents
extraction
reporting
ocr
spreadsheets
office
Read
Write
Edit
Bash(python:*, python3:*, pip:*, pdftotext:*, qpdf:*, libreoffice:*)
Grep
Glob
author complexity
PCL Standard Library intermediate

Document Processing Expert

Working with the formats people actually exchange. The recurring mistake is treating a document as text: these are structured containers, and the structure is where the meaning lives.

Core Concepts

Read the Format, Not the Bytes

.docx, .xlsx and .pptx are ZIP archives of XML parts. PDF is a page description language, not a document model. Using a library that understands the format is not a convenience — string manipulation on these files produces output that opens corrupted.

Extraction Is Lossy; Decide What Matters

Every extraction discards something: layout, styling, reading order, or the relationship between a heading and its table. Decide up front whether you need text, structure, or fidelity, and choose the tool accordingly.

Generation Is About Templates

Building a document element by element in code produces unmaintainable output that a designer cannot touch. Fill a template that a human authored, in the tool they authored it in.

Untrusted Documents Are Untrusted Input

Documents carry macros, external entities, embedded files and remote references. A parsing library is an attack surface. Treat every uploaded document as hostile.

PDF

Extracting text

import pdfplumber

def extract_pages(path: str) -> list[str]:
    with pdfplumber.open(path) as pdf:
        return [page.extract_text() or "" for page in pdf.pages]

pdfplumber preserves layout well and extracts tables; pypdf is faster for plain text and page manipulation. If extract_text() returns empty on a page that clearly has content, the page is an image — you need OCR, not a different parser.

Extracting tables

with pdfplumber.open(path) as pdf:
    for page in pdf.pages:
        for table in page.extract_tables():
            header, *rows = table
            records = [dict(zip(header, row)) for row in rows]

Table extraction is unreliable on borderless or multi-page tables. Validate the result — column count, expected types, totals that reconcile — rather than trusting it.

Manipulating pages

from pypdf import PdfReader, PdfWriter

def extract_range(src: str, dst: str, start: int, end: int) -> None:
    reader, writer = PdfReader(src), PdfWriter()
    for page in reader.pages[start:end]:
        writer.add_page(page)
    with open(dst, "wb") as fh:
        writer.write(fh)

Note that removing a page does not remove content another page references, and redacting by drawing a black rectangle leaves the text underneath fully extractable. Real redaction removes the content stream — use a tool that does.

Forms

reader = PdfReader("form.pdf")
fields = reader.get_fields()                     # discover names first

writer = PdfWriter(clone_from="form.pdf")
writer.update_page_form_field_values(
    writer.pages[0], {"applicant_name": "A. Dupont", "amount": "1234.50"}
)

Flatten the form after filling if the values must not be editable; otherwise the recipient can change them and the file still looks official.

Excel

Reading

import openpyxl

wb = openpyxl.load_workbook("report.xlsx", data_only=True, read_only=True)
sheet = wb["Sales"]
rows = [dict(zip([c.value for c in sheet[1]], [c.value for c in row]))
        for row in sheet.iter_rows(min_row=2)]

data_only=True returns the last cached formula result; without it you get the formula string. If the file was never opened in Excel, that cache is empty — None for every computed cell. This is the most common surprise in spreadsheet processing.

read_only=True streams rather than loading the workbook into memory; use it for anything above a few thousand rows.

Writing

For data, pandas is the shortest path. For anything a person will read — formatting, formulas, multiple sheets — use openpyxl or xlsxwriter directly.

from openpyxl.styles import Font
from openpyxl.utils import get_column_letter

ws.append(["Product", "Units", "Revenue"])
for cell in ws[1]:
    cell.font = Font(bold=True)

for row in data:
    ws.append([row.product, row.units, row.revenue])

ws.freeze_panes = "A2"
for i, width in enumerate([32, 10, 14], start=1):
    ws.column_dimensions[get_column_letter(i)].width = width

Write numbers as numbers and dates as dates, then apply a number format. A right-aligned string that looks like a total cannot be summed, and every recipient will try.

Word

from docx import Document

doc = Document("template.docx")

for paragraph in doc.paragraphs:
    for key, value in context.items():
        if f"{{{{{key}}}}}" in paragraph.text:
            replace_preserving_runs(paragraph, f"{{{{{key}}}}}", value)

Naive paragraph.text = ... destroys all formatting in the paragraph, because text is split across runs that carry the styling. Word also splits a placeholder across runs unpredictably, which is why {{name}} often fails to match. For anything beyond trivial substitution use docxtpl, which handles runs and supports loops and conditionals in the template.

from docxtpl import DocxTemplate

tpl = DocxTemplate("invoice_template.docx")
tpl.render({"customer": customer, "lines": lines, "total": total})
tpl.save("invoice.docx")

PowerPoint

from pptx import Presentation
from pptx.util import Inches, Pt

prs = Presentation("brand_template.pptx")     # keeps master, theme, fonts
slide = prs.slides.add_slide(prs.slide_layouts[1])
slide.shapes.title.text = "Q3 Results"
slide.placeholders[1].text = "Revenue up 12% year on year"

Always start from the organisation's template so the master and theme are inherited. Populate placeholders from the layout rather than adding free-floating text boxes, or the deck will not survive a theme change.

Scanned Documents and OCR

import pytesseract
from pdf2image import convert_from_path

def ocr_pdf(path: str, dpi: int = 300, lang: str = "eng") -> list[str]:
    return [pytesseract.image_to_string(img, lang=lang)
            for img in convert_from_path(path, dpi=dpi)]

Quality is decided before OCR runs: 300 DPI minimum, deskew, increase contrast, binarise. Set the language explicitly — the default assumes English and mangles accented text. Treat OCR output as low-confidence and validate anything numeric.

Security

  • Never open an untrusted document with macros enabled, and never convert one with an office suite that executes them.
  • Disable external entity resolution when parsing the XML inside these containers — XXE applies to .docx and .xlsx as much as to any XML.
  • Enforce size and page limits. A crafted "zip bomb" or a PDF with recursive references exhausts memory during parse.
  • Parse in a sandbox with no network, a memory limit and a timeout.
  • Strip metadata before distribution. Author, revision history, tracked changes and comments routinely leak information the sender did not intend.
  • Redact by removing content, not by drawing over it.
def safe_load(path: str, max_bytes: int = 50_000_000, max_pages: int = 2000):
    if os.path.getsize(path) > max_bytes:
        raise ValueError("document exceeds size limit")
    reader = PdfReader(path)
    if len(reader.pages) > max_pages:
        raise ValueError("document exceeds page limit")
    return reader

Best Practices

  • Validate after extraction, always. Check field presence, types, and domain invariants such as line items summing to the stated total.
  • Keep provenance — file, page, cell — so any extracted value can be traced back and shown to a reviewer.
  • Stream large files. Loading a 200 MB workbook into memory to read column A is avoidable.
  • Generate from templates authored by whoever owns the layout.
  • Test with real documents, including the ugly ones: multi-page tables, merged cells, right-to-left text, and files produced by older software.
  • Prefer a native library to a headless office suite. LibreOffice conversion is a heavyweight fallback, not a default.

Anti-Patterns

  • Regex over PDF bytes — works on one file and fails on the next.
  • Assuming extracted text is in reading order — multi-column layouts are not.
  • Writing numbers as strings into spreadsheets.
  • Rebuilding a branded document in code instead of filling a template.
  • Trusting data_only=True without checking the cache is populated.
  • Black rectangles as redaction — the text is still there.
  • Ignoring encoding — mojibake in a generated invoice reaches the customer.

Reference Documentation

  • Library Selection and Recipes — comparison of the Python and JavaScript libraries per format, with complete recipes for reporting, merging, conversion and batch processing

Resources