kesslerio_finance-news/docs/PREMIUM_SOURCES.md

Premium Source Authentication

Contents

• Overview
• Option 1: Keep It Simple (Recommended)
• Option 2: Use Premium Sources (Advanced)
• Troubleshooting
• Alternative: Use APIs Instead
• Recommendation

Overview

WSJ and Barron's are premium financial news sources that require subscriptions. This guide explains how to authenticate and use premium sources with the finance-news skill.

Recommendation: For simplicity, we recommend using free sources only (Yahoo Finance, CNBC, MarketWatch). Premium sources add complexity and maintenance burden.

If you have subscriptions and want premium content, follow the steps below.

---

Option 1: Keep It Simple (Recommended)

Use free sources only. They provide 90% of the value without authentication complexity:

• ✅ Yahoo Finance (free, reliable)
• ✅ CNBC (free, real-time news)
• ✅ MarketWatch (free, broad coverage)
• ✅ Reuters (free via Yahoo RSS)

To disable premium sources:

1. Edit config/config.json (legacy: config/sources.json)
2. Set "enabled": false for WSJ/Barron's entries
3. Done - no authentication needed

---

Option 2: Use Premium Sources (Advanced)

Prerequisites

• Active WSJ or Barron's subscription
• Browser with active login session (Chrome/Firefox)
• Option B only: Install requests library if needed:

  pip install requests
  

Step 1: Export Cookies from Browser

Chrome:

1. Install extension: EditThisCookie
2. Navigate to wsj.com (logged in)
3. Click EditThisCookie icon → Export → Copy JSON

Firefox:

1. Install extension: Cookie Quick Manager
2. Navigate to wsj.com (logged in)
3. Right-click page → Inspect → Storage → Cookies
4. Copy relevant cookies (see format below)

Step 2: Create Cookie File

Create config/cookies.json (this file is gitignored):

{
  "feeds.a.dj.com": {
    "wsjgeo": "US",
    "djcs_session": "YOUR_SESSION_TOKEN_HERE",
    "djcs_route": "YOUR_ROUTE_HERE"
  },
  "www.barrons.com": {
    "wsjgeo": "US",
    "djcs_session": "YOUR_SESSION_TOKEN_HERE"
  }
}

Important: Cookie domain must match feed URL domain:

• WSJ feeds use feeds.a.dj.com (not wsj.com)
• Barron's feeds use www.barrons.com
• Check config/config.json for actual feed URLs

Note: Cookie names/values vary by site. Export from browser to get actual values.

Step 3: Pass Cookies to fetch_news.py

Option A: Modify fetch_news.py (not officially supported)

Add cookie loading to fetch_rss() function (maintains existing signature):

import json
import urllib.request
from pathlib import Path
from urllib.parse import urlparse

def fetch_rss(url: str, limit: int = 10) -> list[dict]:
    """Fetch and parse RSS feed with optional cookie authentication."""
    
    # Load cookies if they exist
    cookie_file = Path(__file__).parent.parent / "config" / "cookies.json"
    cookies = {}
    if cookie_file.exists():
        with open(cookie_file) as f:
            all_cookies = json.load(f)
            # Extract domain from URL (e.g., feeds.a.dj.com)
            domain = urlparse(url).netloc
            cookies = all_cookies.get(domain, {})
    
    # Fetch with cookies and User-Agent
    req = urllib.request.Request(url, headers={'User-Agent': 'OpenClaw/1.0'})
    if cookies:
        cookie_header = "; ".join([f"{k}={v}" for k, v in cookies.items()])
        req.add_header("Cookie", cookie_header)
    
    # ... rest of function (unchanged)

Note: This is a doc-only suggestion, not officially supported by the skill.

Option B: Use requests library instead of urllib

Replace urllib with requests for easier cookie handling (maintains API signature):

import requests

def fetch_rss(url: str, limit: int = 10, cookies_dict: dict = None) -> list[dict]:
    response = requests.get(url, cookies=cookies_dict, timeout=10)
    response.raise_for_status()
    # ... parse with feedparser

Step 4: Security Considerations

Critical: Do NOT commit cookies to git

1. .gitignore already includes cookie files:

   • config/cookies.json
   • *.cookie
   • No action needed (already configured)

2. Set restrictive file permissions:

   chmod 600 config/cookies.json
   

3. Set restrictive file permissions:

   chmod 600 config/cookies.json
   

4. Rotate cookies regularly:

   • Browser session cookies expire (usually 7-30 days)
   • Re-export cookies when authentication fails

5. Never share cookie files:

   • Cookies grant full account access
   • Treat like passwords

---

Troubleshooting

"HTTP 403 Forbidden" errors

Cause: Cookies expired or invalid

Fix:

1. Log in to WSJ/Barron's in browser
2. Re-export cookies
3. Update config/cookies.json

"Paywall detected" in articles

Cause: RSS feed doesn't require auth, but full article does

Fix:

• Premium sources often provide headlines/snippets in RSS (no auth needed)
• Full articles require subscription + cookie auth
• If you only need headlines → no cookies needed

Cookies not working

Debug checklist:

• Correct domain in cookies.json:
  • WSJ: Use feeds.a.dj.com (not wsj.com)
  • Barron's: Use www.barrons.com (not barrons.com)
• Check config/config.json for actual feed URLs
• Cookie values copied completely (no truncation)
• Browser session still active (test by visiting site)
• File permissions correct (chmod 600)

---

Alternative: Use APIs Instead

Some premium sources offer APIs:

• WSJ API: Not publicly available
• Barron's API: Part of Dow Jones API (enterprise only)
• Bloomberg API: Enterprise only

Conclusion: Cookie-based auth is the only practical option for individual users.

---

Recommendation

For most users: Stick with free sources. They're reliable, no auth needed, and provide comprehensive market coverage.

For premium subscribers: Follow Option 2, but be prepared to maintain cookie files and handle expiration.