OpenAI's completion_tokens already includes reasoning_tokens as a subset,
so adding them again was incorrectly inflating token counts by ~2x for
all reasoning models (o1, o3, o4-mini, etc.).
This aligns with other OpenAI-compatible providers (OpenRouter, Vercel,
Cerebras, Groq) which use completion_tokens directly.
Note: This is different from Google Gemini where thinking_tokens ARE
reported separately and need to be added.
Fixes#4065
Fix indentation so structured-output request executes even when add_schema_to_system_prompt is false; add regression tests for choices=null in both structured and non-structured paths.
Avoid indexing response.choices[0] when proxies return choices=null/empty; raise a clear ModelProviderError (with base_url hint) and use the validated first choice for content/finish_reason
Fixes#3970
The LLM was typing literal placeholder names (e.g., 'user_name') instead
of wrapping them in <secret> tags (e.g., '<secret>user_name</secret>').
Changes to _get_sensitive_data_description():
- Added clear 'SENSITIVE DATA' header to draw attention
- Changed placeholder list format from Python list to bullet points
- Added explicit 'MUST' instruction emphasizing the required format
- Added concrete example using the first actual placeholder name
- Added explanation that the system will replace tags with actual values
This makes it much clearer to the LLM that it needs to wrap placeholder
names in <secret> tags for the automatic replacement mechanism to work.
- restricted read_long_content file access to only paths in available_file_paths or browser_session.download_files to mitigate injection
- made PDF page selection truncate pages that exceed char budget instead of dropping them entirely
- fixed truncation hint in file_system.py to reference read_long_context instead of search_page
Saurav Panda