mirror of
https://github.com/mvanhorn/last30days-skill.git
synced 2026-09-14 16:11:48 +08:00
main
9 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Matt Van Horn
|
b15e2eb46a |
feat(x): X search over the official X API, plus the Grok Bot X connector (#1124)
Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> |
||
|
Matt Van Horn
|
f9a3c919bc |
feat(discover): judged topic names, junk gate, angles, topic queue (#852)
* feat(discover): add topic_shape name distiller and junk-shape classifier * feat(discover): stage-1 judge pass - short names, junk flags, worthiness-blended ranking * feat(discover): add angle and topic-queue fields to DiscoveryTopic, bump export schema to 1.1 * feat(discover): junk-shape floor gate counts seed sources; weak_signal prefers non-junk failures * feat(discover): stage-2 angle pass renders podcast and X-article hooks on trend cards * feat(discover): persistent topic queue - surfacing memory, covered tracking, queue CLI * feat(discover): SKILL.md trending trigger, relay contract for angle/pipeline lines, queue interactions * chore(discover): drop unused query import in rerank * refactor(discover): dedupe judge/angle plumbing, reuse shared normalizers, fix Pipeline line wording * fix(review): guard queue hook (P0), two-phase annotation, domain-preserving upsert, queue-list message, ordinal tests, SKILL.md queue fast path (#1 #4 #6 #8 #9) * fix(review): dict-payload guard, extract discovery_judge module, multi-token collision disambiguation (#2 #3 #5) * fix(review): covered status survives judge naming drift - fuzzy-matched covered priors born covered (#7) * docs(changelog): fill PR link for discovery content pipeline entries --------- Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> |
||
|
Matt Van Horn
|
d31d5f3b3a |
feat(discover): rebuild --discover as nominate -> enrich -> floor; add global trending (#816)
* feat(discover): U1 nominate stage with keyword-gate toggle for global vs domain Extract the discovery fetch+normalize sweep into a reusable nominate_candidates() and add a keyword_gate param to _fetch_discovery_source. Domain discovery keeps the gate on; global trending (no domain) turns it off so the river feeds' own hot ranking is the signal. run_discover now delegates its fetch loop to nominate_candidates(keyword_gate=True), preserving existing behavior. * feat(discover): U2 nominate_topics - named, seed-ranked candidate topics Extract the cluster/rank/name block from run_discover into nominate_topics(), returning Nomination objects (name, seed velocity score, cluster items, leader summary). This is the contract between the nominate stage and the upcoming enrichment fan-out: names deduped casefold, ranked by cheap seed velocity, never padded past the evidence. run_discover consumes nominations with identical output to before. * feat(discover): U3 enrichment fan-out - full pipeline pass per nomination enrich_nominations() runs the real research pipeline (run() with internal_subrun=True, the comparison-mode lane) on each nominated topic in a bounded ThreadPoolExecutor against a wall-clock batch budget. A topic whose sub-run raises is kept as nomination-only with the error recorded; topics unfinished at budget expiry are likewise dropped to nomination-only. The batch never raises and preserves nomination order. This is what buys discovery the whole multi-source corpus (Reddit+comments, X, YouTube, Techmeme, arXiv, HN, Polymarket, web) with zero bespoke fetch code. * feat(discover): U4 confidence floor - 'nothing solid' beats ranked noise THE bug fix for the 2026-07-12 junk-trend regression (five 1-like tweets ranked as a 'sports' trend list). passes_discovery_floor() gates every topic: an absolute engagement floor kills junk outright, then a topic clears via independent cross-source confirmation OR a genuinely strong single-source spike (a 1,600-point HN thread is a story; a 30-upvote meme is not). run_discover gains enrich= (full pipeline pass per nomination via U3) and judges each topic on its enriched corpus when available, seed evidence otherwise. Zero floor-clearing topics -> DiscoveryReport.outcome 'nothing-solid' with the strongest sub-floor name in weak_signal, exported through to_discovery_export. Regression test: the junk corpus now returns an honest empty result. * feat(discover): U5 global trending mode - bare --discover with no domain --discover DOMAIN is now nargs-optional: bare --discover sweeps every river feed's own hot list (r/all, HN front page, Digg) with the keyword gate off - the feeds' hot ranking is the signal and the confidence floor culls junk. X sits out of the global nominate stage (its search lane needs a keyword) and joins per-topic at enrichment. Dispatch keys on 'flag present' so the empty domain is never mistaken for 'flag absent'. Discovery now enriches by default (full research pass per nomination); --discover-shallow is the fast escape hatch that ranks on listing evidence only, still floored. Global saves slug as 'trending'. * feat(discover): U6 rich trend cards + honest nothing-solid rendering DiscoveryTopic gains top_comment (strongest verbatim community comment from the enriched corpus, per-platform vote-normalized, with attribution) and corroboration_count; both exported in to_discovery_export. Renderer upgrades: '# Trending now' header for global runs, a 'confirmed across N sources' badge on the momentum line, a '**Community voice:**' quote line on enriched cards, and an explicit nothing-solid empty state naming the closest weak signal instead of the old generic 'no clusters survived' line. * feat(discover): U7 Techmeme + arXiv reach discovery through enrichment Enrichment sub-runs get requested_sources=None by default, so every available source - Techmeme, arXiv, YouTube, Polymarket, community comments - joins each nominated topic's research pass despite having no river feed of its own. An explicit user --search boundary now holds through enrichment too (the CLI threads the original pre-narrowed list as enrich_requested_sources). The keyword-source nominate gap (X/Techmeme/arXiv have no front-page lane) is documented at the planner. * docs(discover): U8 route global vs domain trending; document two-stage flow SKILL.md discovery branch now routes bare trending asks (/last30days trending, 'what's hot') to bare --discover and domain asks to --discover DOMAIN, documents the nominate-enrich timing (10-minute Bash timeout), the --discover-shallow fast path, and pins 'Nothing solid this window' as a valid verbatim pass-through outcome (never retry or fabricate around it). CONFIGURATION.md and the JSON export reference document the new flag surface and contract fields (outcome, weak_signal, top_comment, corroboration_count). * fix(discover): strip quote chars from community-voice bodies to avoid doubled quotes Live global-trending verification surfaced comments whose bodies start with a quote character rendering as doubled quotes inside the card's wrapping quotes. * fix(discover): address PR #816 review - real budget enforcement + shallow-flag guard P1: enrichment workers now run as daemon threads behind a semaphore instead of a ThreadPoolExecutor. Executor threads are non-daemon and joined at interpreter shutdown, so one hung sub-run could keep the process alive long after its topic was downgraded to nomination-only. Daemon workers make the wall-clock budget real; abandonment is safe because internal_subrun passes write nothing to disk and every fetch layer carries its own timeout. P2: --discover-shallow without --discover now errors (exit 2) instead of silently no-opping into a full research run. --------- Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> |
||
|
Matt Van Horn
|
3b0b021e2c |
feat(sources): local corpus source - your own files as a ranked signal (#808)
* feat(sources): local corpus source - your own files as a ranked signal * fix: address self-review findings * fix: address round-2 residual (surgical round) * fix: defang corpus sentinels, matching-window snippets, exclusion-aware hosted gate, traversal notes, bounded discovery * fix: keep absolute local paths out of corpus notes and coverage diagnostics * fix(corpus): keep raw exception text out of coverage notes OS and subprocess errors embed the failing absolute path in str(exc), and scan/cache notes flow into source_status detail rendered in coverage diagnostics outside the private corpus block. Notes now carry the error's strerror (or class name) instead, so a permission failure or file race can no longer leak a local path from a private run. --------- Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> |
||
|
Matt Van Horn
|
9c020b1393 |
feat(grounding): per-claim freshness verdicts (--verify-freshness) (#805)
* feat(grounding): per-claim freshness verdicts (--verify-freshness) * fix: address self-review findings * fix: address round-2 residual (surgical round) * fix: prefer active markets in mixed-event verification, plan-only cookies for cached verification * fix: verify event identity on slug-fallback refetch before assigning verdicts * test: use a real outcome datum key in slug-identity test * fix(freshness): carry values in stale detail and announce zero-claim passes A stale verdict's machine-readable detail said only that the value moved; the rendered table already showed was/now, so agent-JSON consumers saw less than readers. Compose the detail as "moved: <original> -> <current>", formatting each value the way the claim text does (percent for Polymarket/StockTwits, thousands separators for stars). A verification pass that extracts zero claims used to complete silently, which misled a live post-hoc run. _verify_report_set now prints one stderr note when no verified report produced any verdict, aggregated across main and entity reports so multi-report passes note it once; the drill path routes through the same helper. * fix(freshness): verify star facts attached by candidate enrichment A GitHub-flavored run produced zero verdicts live: star enrichment attaches metadata["github_stars"] to candidates after reranking, but claim extraction read only item-level engagement, and those candidates' primary items are typically non-GitHub sources. Extraction now emits one repo-keyed star claim per enriched repo (skipping repos already claimed at item level). These claims cannot ride the item-source dispatch - they would land unsupported as "reddit"/"x" claims - so verify_report routes the repo-slug shape straight to the GitHub refetcher, bypassing the grounding-item lookup and the per-source outcome gate (the datum came from enrichment, not the github search source). refetch_datum accepts an owner/repo slug as the datum key, and the snapshot cache keys these claims by repo so multiple candidates citing one repo share one request. * fix(freshness): review fixes - honest evidence, joinable export, rendered detail Four review findings on the hardening pass: - Unsupported verdicts no longer fabricate evidence: evidence_url and evidence_timestamp stay empty when no fresh evidence was obtained; provenance remains on source_url/source_timestamp. - Agent export results now carry candidate_id so consumers can join freshness_verdicts to the result they annotate (schema_version 1.2, golden snapshot and docs updated). - The rendered verdict table shows the verifier's detail - formatted movement on stale rows, the un-checkable reason on unsupported rows - instead of reconstructing raw values. - Item-level star dedup is scoped per candidate: another candidate's item-level claim no longer suppresses an enriched candidate's own verdict and inline flag; both share one repo snapshot. * fix(freshness): require event-id identity on Polymarket slug fallback The slug-fallback refetch verified the response slug but accepted a slug-matched event whose id differed from the cached item's event id, so a re-created event reusing a slug could produce current/stale verdicts from another market's prices. When the cached item carries a real numeric event id, the fallback now requires the response id to match it (synthetic PM<N> parse ids carry no identity and skip the check); mismatches raise and degrade to an unsupported verdict. * fix(freshness): fail closed when the slug fallback has no event identity A cached item with neither metadata.event_id nor a real numeric item id (the synthetic PM<N> parse fallback) let the slug fallback accept whatever event currently owns the slug, so a recreated or loosely matched event could still drive a current/stale verdict. With no identity to verify against, the refetch now raises before any request and the claim degrades to unsupported. --------- Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> |
||
|
Matt Van Horn
|
298310ca34 |
feat(pipeline): --discover mode for topic-less trending discovery (#801)
* feat(pipeline): add --discover mode for topic-less trending discovery * fix: address self-review findings * fix: address round-2 residual (surgical round) * fix: boundary-aware category match, additive-only discovery totals, preserve discovery intent through onboarding * fix: visible default-search fallback, non-plural domain anchors, clean X backend fallback * fix(discover): apply domain filter to Digg listing clusters The Digg source is an AI-focused leaderboard feed, so a discovery sweep on a non-AI domain surfaced off-domain stories (observed live: a crypto sweep returning model-release coverage). Filter parsed Digg clusters with the same title-level _matches_discovery_domain check the adjacent Hacker News branch already applies; an all-filtered result stays a clean no-results outcome. * fix(discover): surface the producing X backend's own error as partial Failing over past a dead backend is a clean outcome, but when the backend that actually produced items also returned an error, the sweep is degraded; returning a clean state hid that from source_status and strict-exit runs. Keep the producing backend's error; earlier fallback errors remain stderr observability only. * fix(discover): hold the configured source boundary instead of widening When the configured source filter (LAST30DAYS_DEFAULT_SEARCH or --search) contains no discovery-capable source, the sweep silently widened to all discovery feeds with only a warning, querying sources the user had filtered out. Discovery now exits 2 with the unsupported sources and the discovery-capable set named, for both the config and explicit-flag paths. --------- Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> |
||
|
Matt Van Horn
|
0270c12352 |
feat(eval): research-quality regression harness with scored fixtures in CI (#799)
* feat(eval): research-quality regression harness with scored fixtures in CI * fix: address self-review findings * fix: harden eval replay (coherence floor, metadata-merge replay, offline star enrichment) * fix: value-scrub module fixture recordings, per-fixture floors, pinned overlap predicate --------- Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> |
||
|
Matt Van Horn
|
7abcb9d108 |
feat(schema): versioned agent JSON export profile (--emit=json v1.0, raw fallback) (#810)
* feat(schema): versioned agent JSON export profile with --json-profile raw fallback * fix: address self-review findings * fix: exclude author-reach fields from headline engagement (stocktwits followers) * fix: fail loudly when the evaluator receives the agent profile instead of raw Report --------- Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> |
||
|
Matt Van Horn
|
aecab67c81 |
feat(setup): restore the v3.0.0 first-run NUX wizard (#661)
* feat(setup): restore the v3.0.0 first-run NUX wizard on the consent-driven foundation Step 0 now has two branches. Claude Code (and any host with AskUserQuestion) gets the restored original guided NUX: welcome message, Auto/Manual/Skip setup modal, cookie-consent modal, ScrapeCreators signup offer, TikTok/Instagram INCLUDE_SOURCES opt-in, and a first-topic picker. Hosts without modals (OpenClaw, Codex, Cursor, Gemini CLI) get the equivalent Non-Modal Prose Flow. Builds on #659 (first-run gate) and #660 (consent-driven prose, key-persist, Full Disk Access remediation) - all of that is preserved, not reverted. Additive: the source inventory is current, not the v3.0.0 set. - Digg threaded into the install messaging everywhere yt-dlp/YouTube appears (welcome list, Auto-setup option, manual guide). Install already existed; this is the copy. - ScrapeCreators credit count restored to "10,000 free calls". - Hard "ALWAYS execute Step 0 BEFORE Step 1" gate restored to resist re-erosion. - Manual-setup guide refreshed to the current source matrix (Digg, youtube comments, SC Reddit/YouTube backups, Perplexity, Bluesky) with append-only .env safety rules. - Threads/Pinterest intentionally not surfaced in onboarding (power-user INCLUDE_SOURCES only). - tests/test_onboarding_contract.py rewritten for the two-branch structure and to lock the flow against silent re-erosion. - Docs synced: CONFIGURATION.md, AGENTS.md, CHANGELOG.md; original wizard captured at docs/reference/old-nux-wizard-v3.0.0.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q37ombFQdv9uLbKm2y8vBD * fix(setup): resolve Greptile findings on the restored NUX wizard - Skip+Skip path now writes SETUP_COMPLETE: picking "Skip for now" at the setup choice wrote no .env flag, so the first-run gate re-fired on every invocation. The skip branch now persists SETUP_COMPLETE=true and goes to the topic picker. - Modal step labels (Step 1/2/3) now match the sequence descriptor; the body jumped from Step 1 to Step 4. - Non-Modal Prose Flow now honors an existing BROWSER_CONSENT=true (skip re-asking when consent was granted in a prior session). - Contract test: symmetric Full Disk Access assertion on the prose branch + a guard that the Skip path writes SETUP_COMPLETE. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q37ombFQdv9uLbKm2y8vBD * fix(setup): restore masked-key retrieval hint on persisted:false path When setup --github succeeds but the key write fails, both flows told the user to add SCRAPECREATORS_API_KEY=<key> manually but dropped the hint on how to obtain the value (the raw key is masked in output). Restored the parenthetical: re-run setup --github or retrieve it from scrapecreators.com. (Greptile) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Q37ombFQdv9uLbKm2y8vBD --------- Co-authored-by: Matt Van Horn <455140+mvanhorn@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |