mims-harvard__tooluniverse/docs/dev_docs/VALIDATION_GUIDE.md

Documentation Validation Guide

This guide explains how to use the documentation validation system to ensure documentation quality before committing changes.

Overview

The validation system includes:

1. Pre-commit hook (.githooks/pre-commit-docs) - Automatic validation on git commit
2. Manual validation - Scripts you can run independently
3. CI/CD validation - Automatic checks on pull requests

Quick Start

Enable Pre-Commit Hook

# One-time setup
./setup_precommit.sh

# Or manually
git config core.hooksPath .githooks

Run Validation Manually

# Validate documentation changes
./.githooks/pre-commit-docs

# Or run specific validation scripts
cd docs
python validate_examples.py  # Validate code examples
python doc_sync_tool.py      # Check API/docs sync

Pre-Commit Hook Features

The .githooks/pre-commit-docs hook automatically checks:

1. RST Syntax Validation

What it checks:

• Title/underline length mismatches
• Common reStructuredText syntax errors

Example issue:

My Title
========  <-- ❌ Wrong length (9 chars vs 8)

My Title
========  <-- ✅ Correct (8 chars = 8)

How to fix:

# The hook will show you the mismatch:
⚠️  Line 42: Title/underline length mismatch in docs/quickstart.rst
  Title: 'My Title' (length: 8)
  Underline: '=========' (length: 9)

# Fix by matching lengths exactly

2. Broken Internal References

What it checks:

• :doc: references to non-existent files
• Missing cross-reference targets

Example issue:

See :doc:`installation` for details.  <-- ✅ OK (file exists)
See :doc:`setup_guide` for details.   <-- ❌ File not found

How to fix:

# Use correct path
See :doc:`installation` for details.

# Or use relative paths
See :doc:`../guide/installation` for details.

# Or use explicit labels
.. _my-label:

See :ref:`my-label` for details.

3. TODO/FIXME Markers

What it checks:

• Uncommitted TODO/FIXME comments in documentation

Example:

.. TODO: Update this section  <-- ⚠️ Warning
.. FIXME: Broken link          <-- ⚠️ Warning

Why it matters:

• Documentation should be complete before committing
• TODO markers indicate unfinished work

How to fix:

# Either complete the TODOs:
# "Update this section" → Actually update it

# Or create GitHub issues:
# Convert TODO to tracked issue
# Remove TODO from docs

4. Long Lines

What it checks:

• Lines longer than 120 characters (excluding code blocks and URLs)

Why it matters:

• Improves readability
• Better diffs in version control

How to fix:

# ❌ Bad: Long line
This is a very long line that goes on and on and on and contains too much information on a single line making it hard to read.

# ✅ Good: Broken into multiple lines
This is a long line that has been broken into multiple shorter lines,
making it much easier to read and maintain in version control.

5. Auto-Generated Files

What it checks:

• Manual edits to files in docs/tools/ and docs/api/

Example warning:

⚠️  Warning: You are modifying auto-generated files:
 - docs/tools/uniprot_tools.rst
 - docs/api/tooluniverse.core.rst

How to fix:

# Don't edit these files directly!
# Instead, edit source files and regenerate:

# For tool documentation
cd docs
python generate_config_index.py
python generate_tool_reference.py

# For API documentation
# Edit Python docstrings in src/tooluniverse/
cd docs
sphinx-apidoc -f -o api ../src/tooluniverse

6. Tool Count Consistency

What it checks:

• Inconsistent tool counts across documentation (e.g., "600 tools" vs "750 tools")

Standard: Use 1000+ tools consistently

How to fix:

# Search for tool count mentions
grep -r "tools" docs/ --include="*.rst" | grep -E "[0-9]+"

# Update to standard format (replace any stale counts with 1000+)
sed -i 's/[0-9]*+ tools/1000+ tools/g' docs/quickstart.rst

Manual Validation Scripts

validate_examples.py

Purpose: Validates Python code examples in documentation

cd docs
python validate_examples.py

What it checks:

• Python syntax errors
• Import availability
• Common code issues

Example output:

✅ quickstart.rst: All 5 code blocks valid
❌ advanced.rst: Syntax error in block 3
  Line 12: unexpected indent

doc_sync_tool.py

Purpose: Ensures documentation matches source code

cd docs
python doc_sync_tool.py

What it checks:

• Docstring/documentation sync
• API signature changes
• Missing documentation

Example output:

⚠️  API change detected in tooluniverse.core
  Method `load_tools()` signature changed
  → Update docs/api/tooluniverse.core.rst

doc_analytics.py

Purpose: Analyzes documentation quality metrics

cd docs
python doc_analytics.py

What it provides:

• Page statistics (word count, code blocks, links)
• Quality recommendations
• Coverage analysis

Validation Workflow

Before Committing

# 1. Make documentation changes
vim docs/quickstart.rst

# 2. Regenerate auto-generated content if needed
cd docs
python generate_config_index.py
python generate_tool_reference.py

# 3. Build locally to test
./quick_doc_build.sh

# 4. Run validation (automatic on commit)
git add docs/quickstart.rst
git commit -m "docs: update quickstart"  # Validation runs automatically

# If validation fails, fix issues and try again

Dealing with Warnings

Warnings won't block your commit, but you should fix them:

# Commit with warnings (allowed but not recommended)
git commit -m "docs: update quickstart"
⚠️  Validation completed with 2 warning(s)
You can still commit, but consider fixing these issues

# Fix warnings and amend commit
vim docs/quickstart.rst  # Fix issues
git add docs/quickstart.rst
git commit --amend --no-edit

Dealing with Errors

Errors will block your commit:

git commit -m "docs: update quickstart"
❌ Validation failed with 1 error(s)

# Fix the error
vim docs/quickstart.rst
git add docs/quickstart.rst
git commit -m "docs: update quickstart"
✅ Documentation validation passed!

CI/CD Validation

GitHub Actions automatically validates documentation on pull requests:

# .github/workflows/deploy-docs.yml
- name: Validate documentation
 run: |
   ./.githooks/pre-commit-docs

What happens:

• Validation passes → PR can be merged
• Validation fails → PR blocked until fixed
• ️ Warnings → PR can be merged but review recommended

Common Issues & Solutions

Issue: "Title/underline length mismatch"

Cause: RST heading underline doesn't match title length

Solution:

# Count characters carefully
Installation Guide
==================  (18 characters = 18 equals signs)

Tip: Use an editor with column indicators or run:

echo "My Title" | wc -c  # Get exact length

Issue: "Possibly broken reference"

Cause: :doc: or :ref: points to non-existent file

Solution:

# Check if file exists
ls docs/installation.rst

# Use correct path
:doc:`installation`          # For docs/installation.rst
:doc:`guide/loading_tools`   # For docs/guide/loading_tools.rst

# Or use explicit references
.. _my-section:

Some content here

Reference it: :ref:`my-section`

Issue: "Modifying auto-generated files"

Cause: Editing files in docs/tools/ or docs/api/ directly

Solution:

# Don't edit generated files!
# Edit source instead:

# For tool docs:
vim src/tooluniverse/data/uniprot_tools.json
cd docs && python generate_config_index.py

# For API docs:
vim src/tooluniverse/core_tool.py  # Edit docstrings
cd docs && sphinx-apidoc -f -o api ../src/tooluniverse

Issue: "Inconsistent tool counts"

Cause: Different tool counts mentioned across docs

Solution:

# Use standard format everywhere
grep -r "[0-9]\+ tools" docs/ --include="*.rst"

# Replace with "1000+ tools"
# See DOCUMENTATION_STANDARDS.md for details

Bypassing Validation (Not Recommended)

In rare cases, you may need to bypass validation:

# Skip pre-commit hooks (not recommended)
git commit --no-verify -m "docs: emergency fix"

When to bypass:

• Emergency hotfixes
• False positives in validation
• Documented exceptions

Always:

• Document why you bypassed validation in commit message
• Create follow-up issue to fix properly
• Get code review before merging

Best Practices

1. Validate Early and Often

# Don't wait until commit time
# Validate as you write

# After making changes
./quick_doc_build.sh  # Build and check
./.githooks/pre-commit-docs  # Validate

2. Fix Warnings Proactively

# Don't accumulate warnings
# Fix them immediately

# Each warning is a small issue
# Small issues → big problems

3. Use Auto-Generated Headers

All auto-generated files must include header:

.. AUTO-GENERATED - DO NOT EDIT MANUALLY
.. Generated by: docs/generate_config_index.py
.. Last updated: 2026-02-05 10:30:00
.. 
.. To modify, edit source files and regenerate.

This is now automatically added by generation scripts.

4. Keep Documentation in Sync

# When changing code, update docs
git diff src/tooluniverse/core_tool.py  # Changed code
# → Also update docs/api/ or relevant guides

# Use doc_sync_tool.py to check
python docs/doc_sync_tool.py

5. Test Code Examples

# All code examples should work
# Test them regularly

cd docs
python validate_examples.py

# Or run examples manually
python -c "$(sed -n '/.. code-block:: python/,/^$/p' quickstart.rst | tail -n +3)"

Troubleshooting

Hook not running

# Check hook path configuration
git config --get core.hooksPath
# Should be: .githooks

# Reconfigure if needed
git config core.hooksPath .githooks

# Make hook executable
chmod +x .githooks/pre-commit-docs

Validation script errors

# Check Python environment
python --version  # Should be 3.8+

# Install dependencies
pip install sphinx sphinx-rtd-theme

# Check script permissions
ls -la docs/*.py
chmod +x docs/*.py  # If needed

False positives

# Report false positives as issues
# Include:
# - File being validated
# - Warning/error message
# - Why it's a false positive

# Example:
# File: docs/advanced.rst
# Warning: "Broken reference :doc:`custom_page`"
# Reason: custom_page.rst exists but validation doesn't find it

Contributing

Want to improve validation? See:

• .githooks/pre-commit-docs - Main validation script
• docs/validate_examples.py - Code example validator
• docs/doc_sync_tool.py - API/docs sync checker
• docs/DOCUMENTATION_STANDARDS.md - Documentation standards

Additional Resources

• Documentation Standards Guide
• Utility Scripts Guide
• Sphinx Documentation
• reStructuredText Primer

---

Last updated: 2026-02-05
Maintained by: ToolUniverse Documentation Team