udiedrichsen_stock-analysis/TODO.md

# Stock Analysis - Future Enhancements

## Roadmap Overview

### v4.0.0 (Current) - Geopolitical Risk & News Sentiment
✅ 8 analysis dimensions with Fear/Greed, short interest, VIX structure, put/call ratio
✅ Safe-haven indicators (GLD, TLT, UUP) with risk-off detection
✅ Breaking news alerts via Google News RSS
✅ Geopolitical risk mapping (Taiwan, China, Russia, Middle East, Banking)
✅ Sector-specific crisis flagging with confidence penalties
✅ 1h caching for shared indicators (Fear/Greed, VIX structure, breaking news)
✅ Async parallel sentiment fetching (5 indicators with 10s timeouts)

### v5.0.0 (Current) - Portfolio & Crypto
✅ Portfolio management (create, add, remove, show assets)
✅ Cryptocurrency support (Top 20 by market cap)
✅ Portfolio analysis with --portfolio flag
✅ Periodic returns (--period daily/weekly/monthly/quarterly/yearly)
✅ Concentration warnings (>30% single asset)
✅ Crypto fundamentals (market cap, category, BTC correlation)

### v4.1.0 - Performance & Completeness
✅ Full insider trading parsing via edgartools (Task #1)
✅ Market context caching with 1h TTL (Task #3b)
🔧 SEC EDGAR rate limit monitoring (Task #4 - low priority)

### Future (v6.0+)
💡 Research phase: Social sentiment, fund flows, on-chain metrics

---

## Sentiment Analysis Improvements

### 1. Implement Full Insider Trading Parsing
**Status**: ✅ DONE
**Priority**: Medium
**Effort**: 2-3 hours

**Current State**:
- ✅ `get_insider_activity()` fetches Form 4 filings via edgartools
- ✅ SEC identity configured (`stock-analysis@clawd.bot`)
- ✅ Aggregates buys/sells over 90-day window
- ✅ Scoring logic: strong buying (+0.8), moderate (+0.4), neutral (0), moderate selling (-0.4), strong (-0.8)

**Tasks**:
- [ ] Research edgartools API for Form 4 parsing
- [ ] Implement transaction aggregation (90-day window)
- [ ] Calculate net shares bought/sold
- [ ] Calculate net value in millions USD
- [ ] Apply scoring logic:
  - Strong buying (>100K shares or >$1M): +0.8
  - Moderate buying (>10K shares or >$0.1M): +0.4
  - Neutral: 0
  - Moderate selling: -0.4
  - Strong selling: -0.8
- [ ] Add error handling for missing/incomplete filings
- [ ] Test with multiple tickers (BAC, TSLA, AAPL)
- [ ] Verify SEC rate limit compliance (10 req/s)

**Expected Impact**:
- Insider activity detection for 4th sentiment indicator
- Increase from 3/5 to 4/5 indicators typically available

---

### 2. Add Parallel Async Fetching
**Status**: ✅ DONE (sentiment indicators)
**Priority**: High
**Effort**: 4-6 hours

**Current State**:
- ✅ Sentiment indicators fetched in parallel via `asyncio.gather()`
- ✅ 10s timeout per indicator
- Main data fetches (yfinance) still sequential (acceptable)

**Tasks**:
- [ ] Convert sentiment helper functions to async
  - [ ] `async def get_fear_greed_index()`
  - [ ] `async def get_short_interest(data)`
  - [ ] `async def get_vix_term_structure()`
  - [ ] `async def get_insider_activity(ticker)`
  - [ ] `async def get_put_call_ratio(data)`
- [ ] Update `analyze_sentiment()` to use `asyncio.gather()`
- [ ] Handle yfinance thread safety (may need locks)
- [ ] Add timeout per indicator (10s max)
- [ ] Test with multiple stocks in sequence
- [ ] Measure actual runtime improvement
- [ ] Update SKILL.md with new runtime (target: 3-4s)

**Expected Impact**:
- Reduce runtime from 6-10s to 3-4s per stock
- Better user experience for multi-stock analysis

---

### 3. Add Caching for Shared Indicators
**Status**: ✅ DONE (sentiment + breaking news)
**Priority**: Medium
**Effort**: 2-3 hours

**Current State**:
- ✅ Fear & Greed Index cached (1h TTL)
- ✅ VIX term structure cached (1h TTL)
- ✅ Breaking news cached (1h TTL)
- ✅ Market context (VIX/SPY/QQQ/GLD/TLT/UUP) cached (1h TTL)

**Tasks**:
- [ ] Design cache structure (simple dict or functools.lru_cache)
- [ ] Implement TTL (time-to-live):
  - Fear & Greed: 1 hour
  - VIX structure: 1 hour
  - Short interest: No cache (per-stock)
  - Insider activity: No cache (per-stock)
  - Put/Call ratio: No cache (per-stock)
- [ ] Add cache invalidation logic
- [ ] Add verbose logging for cache hits/misses
- [ ] Test multi-stock analysis (e.g., `BAC TSLA AAPL`)
- [ ] Measure performance improvement
- [ ] Document caching behavior in SKILL.md

**Expected Impact**:
- Multi-stock analysis faster (e.g., 3 stocks: 18-30s → 10-15s)
- Reduced API calls to Fear/Greed and VIX data sources
- Same-session analysis efficiency

---

### 4. Monitor SEC EDGAR Rate Limits
**Status**: Not Started
**Priority**: Low (until insider trading implemented)
**Effort**: 1-2 hours

**Current State**:
- SEC EDGAR API has 10 requests/second rate limit
- No rate limit tracking or logging
- edgartools may handle rate limiting internally

**Tasks**:
- [ ] Research edgartools rate limit handling
- [ ] Add request counter/tracker if needed
- [ ] Implement exponential backoff on 429 errors
- [ ] Add logging for rate limit hits
- [ ] Test with high-volume scenarios (10+ stocks in quick succession)
- [ ] Document rate limit behavior
- [ ] Add error message if rate limited: "SEC API rate limited, try again in 1 minute"

**Expected Impact**:
- Robust handling of SEC API limits in production
- Clear user feedback if limits hit
- Prevent API blocking/banning

---

## Stock Analysis 4.0: Geopolitical Risk & News Sentiment

### What's Currently Missing

The current implementation captures:
- ✅ VIX (general market fear)
- ✅ SPY/QQQ trends (market direction)
- ✅ Sector performance

What we **don't** have yet:
- ❌ Geopolitical risk indicators
- ❌ News sentiment analysis
- ❌ Sector-specific crisis flags

---

### 7. Geopolitical Risk Index
**Status**: ✅ DONE (keyword-based)
**Priority**: High
**Effort**: 8-12 hours

**Proposed Approach**:
Option A: Use GPRD (Geopolitical Risk Daily Index) from policyuncertainty.com
Option B: Scan news APIs (NewsAPI, GDELT) for geopolitical keywords

**Tasks**:
- [ ] Research free geopolitical risk data sources
  - [ ] Check policyuncertainty.com API availability
  - [ ] Evaluate NewsAPI free tier limits
  - [ ] Consider GDELT Project (free, comprehensive)
- [ ] Design risk scoring system (0-100 scale)
- [ ] Implement data fetching with caching (4-hour TTL)
- [ ] Map risk levels to sentiment scores:
  - Low risk (0-30): +0.2 (bullish)
  - Moderate risk (30-60): 0 (neutral)
  - High risk (60-80): -0.3 (caution)
  - Extreme risk (80-100): -0.5 (bearish)
- [ ] Add to sentiment analysis as 6th indicator
- [ ] Test with historical crisis periods
- [ ] Update SKILL.md with geopolitical indicator

**Expected Impact**:
- Early warning for market-wide risk events
- Better context for earnings-season volatility
- Complement to VIX (VIX is reactive, geopolitical is predictive)

**Example Output**:
```
⚠️ GEOPOLITICAL RISK: HIGH (72/100)
   Context: Elevated Taiwan tensions detected
   Market Impact: Risk-off sentiment likely
```

---

### 8. Sector-Specific Crisis Mapping
**Status**: ✅ DONE
**Priority**: High
**Effort**: 6-8 hours

**Current Gap**:
- No mapping between geopolitical events and affected sectors
- No automatic flagging of at-risk holdings

**Proposed Risk Mapping**:

| Geopolitical Event | Affected Sectors | Example Tickers |
|-------------------|------------------|-----------------|
| Taiwan conflict | Semiconductors | NVDA, AMD, TSM, INTC |
| Russia-Ukraine | Energy, Agriculture | XLE, MOS, CF, NTR |
| Middle East escalation | Oil, Defense | XOM, CVX, LMT, RTX |
| China tensions | Tech supply chain, Retail | AAPL, QCOM, NKE, SBUX |
| Banking crisis | Financials | JPM, BAC, WFC, C |

**Tasks**:
- [ ] Build event → sector → ticker mapping database
- [ ] Implement keyword detection in news feeds:
  - "Taiwan" + "military" → Semiconductors ⚠️
  - "Russia" + "sanctions" → Energy ⚠️
  - "Iran" + "attack" → Oil, Defense ⚠️
  - "China" + "tariffs" → Tech, Consumer ⚠️
- [ ] Add sector exposure check to analysis
- [ ] Generate automatic warnings in output
- [ ] Apply confidence penalty for high-risk sectors
- [ ] Test with historical crisis events
- [ ] Document in SKILL.md

**Expected Impact**:
- Automatic detection of sector-specific risks
- Clear warnings for exposed holdings
- Reduced false positives (only flag relevant sectors)

**Example Output**:
```
⚠️ SECTOR RISK ALERT: Semiconductors
   Event: Taiwan military exercises (elevated tensions)
   Impact: NVDA HIGH RISK - supply chain exposure
   Recommendation: HOLD → downgraded from BUY
```

---

### 9. Breaking News Check
**Status**: ✅ DONE
**Priority**: Medium
**Effort**: 4-6 hours

**Current Gap**:
- No real-time news scanning before analysis
- User might get stale recommendation during breaking events

**Proposed Solution**:
- Scan Google News or Reuters RSS before analysis
- Flag high-impact keywords within last 24 hours

**Tasks**:
- [ ] Choose news source (Google News RSS, Reuters API, or NewsAPI)
- [ ] Implement news fetching with 24-hour lookback
- [ ] Define crisis keywords:
  - **War/Conflict**: "war", "invasion", "military strike", "attack"
  - **Economic**: "recession", "crisis", "collapse", "default"
  - **Regulatory**: "sanctions", "embargo", "ban", "investigation"
  - **Natural disaster**: "earthquake", "hurricane", "pandemic"
- [ ] Add ticker-specific news check (company name + keywords)
- [ ] Generate automatic caveat in output
- [ ] Cache news check results (1 hour TTL)
- [ ] Add `--skip-news` flag for offline mode
- [ ] Test with historical crisis dates
- [ ] Document in SKILL.md

**Expected Impact**:
- Real-time awareness of breaking events
- Automatic caveats during high volatility
- User protection from stale recommendations

**Example Output**:
```
⚠️ BREAKING NEWS ALERT (last 6 hours):
   "Fed announces emergency rate hike"
   Impact: Market-wide volatility expected
   Caveat: Analysis may be outdated - rerun in 24h
```

---

### 10. Safe-Haven Indicators
**Status**: ✅ DONE
**Priority**: Medium
**Effort**: 3-4 hours

**Current Gap**:
- No detection of "risk-off" market regime
- VIX alone is insufficient (measures implied volatility, not capital flows)

**Proposed Indicators**:
- Gold (GLD) - Flight to safety
- US Treasuries (TLT) - Bond market fear
- USD Index (UUP) - Dollar strength during crisis

**Risk-Off Detection Logic**:
```
IF GLD +2% AND TLT +1% AND UUP +1% (all rising together)
THEN Market Regime = RISK-OFF
```

**Tasks**:
- [ ] Fetch GLD, TLT, UUP price data (5-day change)
- [ ] Implement risk-off detection algorithm
- [ ] Add to market context analysis
- [ ] Apply broad risk penalty:
  - Risk-off detected → Reduce all BUY confidence by 30%
  - Add caveat: "Market in risk-off mode - defensive positioning recommended"
- [ ] Test with historical crisis periods (2008, 2020, 2022)
- [ ] Add verbose output for safe-haven movements
- [ ] Document in SKILL.md

**Expected Impact**:
- Detect market-wide flight to safety
- Automatic risk reduction during panics
- Complement geopolitical risk scoring

**Example Output**:
```
🛡️ SAFE-HAVEN ALERT: Risk-off mode detected
   - Gold (GLD): +3.2% (5d)
   - Treasuries (TLT): +2.1% (5d)
   - USD Index: +1.8% (5d)
   Recommendation: Reduce equity exposure, favor defensives
```

---

## General Improvements

### 11. Add Social Sentiment (Future Phase)
**Status**: Deferred
**Priority**: Low
**Effort**: 8-12 hours

**Notes**:
- Requires free API (Twitter/Reddit alternatives?)
- Most sentiment APIs are paid (StockTwits, etc.)
- Research needed for viable free sources

### 12. Add Fund Flows (Future Phase)
**Status**: Deferred
**Priority**: Low
**Effort**: 6-8 hours

**Notes**:
- Requires ETF flow data
- May need paid data source
- Research free alternatives

---

## Implementation Priorities

### v4.1.0 Complete
- ✅ Task #1 - Insider trading parsing via edgartools
- ✅ Task #3b - Market context caching (1h TTL)
- 🔧 Task #4 - SEC EDGAR rate limits (low priority, only if hitting limits)

### Completed in v4.0.0
- ✅ Task #2 - Async parallel fetching (sentiment)
- ✅ Task #3 - Caching for shared indicators (sentiment + news)
- ✅ Task #7 - Geopolitical risk (keyword-based)
- ✅ Task #8 - Sector-specific crisis mapping
- ✅ Task #9 - Breaking news check
- ✅ Task #10 - Safe-haven indicators

---

## Version History

- **v5.0.0** (2026-01-16): Portfolio management, cryptocurrency support (Top 20), periodic analysis
- **v4.1.0** (2026-01-16): Full insider trading parsing via edgartools, market context caching
- **v4.0.0** (2026-01-15): Geopolitical risk, breaking news, safe-haven detection, sector crisis mapping
- **v3.0.0** (2026-01-15): Sentiment analysis added with 5 indicators (3-4 typically working)
- **v2.0.0**: Market context, sector performance, earnings timing, momentum
- **v1.0.0**: Initial release with earnings, fundamentals, analysts, historical