fix(agent): allow underscores in variable_ref_patt component_id (#16758) (#16792)

## Summary

`ComponentBase.variable_ref_patt` (and its duplicate in
`agent.canvas.Graph.get_value_with_variable`) is the regex the canvas
runtime uses to find `cpn_id@var_nm` template refs in component prompts.

The `cpn_id` half was constrained to `[a-zA-Z:0-9]+`, which silently
dropped underscores. Component ids emitted by the frontend all contain
underscores (`userfillup_abc`, `retrieval_xyz`, `llm_0`, `message_0`,
…), so any template ref like `{userfillup_abc@line}` failed to match.
The placeholder then leaked through to the LLM verbatim, and the Agent
answered only its system-prompt directive.

This is exactly the "unconsidered await response" symptom in #16758:

```
Begin(Task) -> Await response -> Agent -> Message
```

Widen `cpn_id` from `[a-zA-Z:0-9]+` to `[a-zA-Z0-9_]+`. Bare `{line}`
(no cpn_id) remains unrecognised so it stays literal until the user
wires it up — matching the existing `VARIABLE_REF_PATTERN` shape used by
`agent.dsl_migration` for the same purpose.

## Changes

- `agent/component/base.py` — fix `variable_ref_patt` class attribute.
- `agent/canvas.py` — same fix applied to the inline regex inside
`Graph.get_value_with_variable` (kept as the literal regex to avoid
coupling the two unrelated sites).
-
`test/testcases/test_web_api/test_canvas_app/test_variable_ref_pattern_unit.py`
— new regression test pinning both the regex shape and end-to-end
resolution.

## Regression coverage

```
test_variable_ref_patt_matches_underscored_component_ids     PASSED
test_variable_ref_patt_still_matches_legacy_ids              PASSED
test_get_input_elements_from_text_resolves_underscored_id    PASSED
test_string_format_substitutes_underscored_ref                PASSED
test_variable_ref_patt_does_not_match_bare_var_name          PASSED
```

All five regression tests fail against the pre-fix regex (verified via
`git stash` round trip — drop fix, tests fail, restore fix, tests pass).

The two targeted existing tests in the same directory
(`test_fillup_unit.py`, `test_iterationitem_unit.py`) continue to pass.

## Repro before the fix

```python
import re
patt = r"\{* *\{([a-zA-Z:0-9]+@[A-Za-z0-9_.-]+|sys\.[A-Za-z0-9_.]+|env\.[A-Za-z0-9_.]+)\} *\}*"
list(re.finditer(patt, "{userfillup_abc@line}"))
# => []   # <-- bug
```

## Repro after the fix

```python
import re
patt = r"\{* *\{([a-zA-Z0-9_]+@[A-Za-z0-9_.-]+|sys\.[A-Za-z0-9_.]+|env\.[A-Za-z0-9_.]+)\} *\}*"
list(re.finditer(patt, "{userfillup_abc@line}"))
# => [<re.Match object; span=(0, 24), match='{userfillup_abc@line}'>]
```

Fixes #16758

## Test plan

- [x] New unit tests pass
- [x] Reverse-apply the fix and confirm the regression tests fail (they
do)
- [x] `test_fillup_unit.py` (existing sibling suite) still passes
- [x] `test_iterationitem_unit.py` (existing sibling suite) still passes
- [ ] Project CI green

---------

Co-authored-by: Taranum01 <taranum01@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>

agent/canvas.py

@@ -200,7 +200,9 @@ class Graph:
         return self._tenant_id
 
     def get_value_with_variable(self, value: str) -> Any:
-        pat = re.compile(r"\{* *\{([a-zA-Z:0-9]+@[A-Za-z0-9_.-]+|sys\.[A-Za-z0-9_.]+|env\.[A-Za-z0-9_.]+)\} *\}*")
+        # Reference the canonical pre-compiled regex from ComponentBase so
+        # the source-pattern and the runtime-pattern can never drift apart.
+        pat = ComponentBase.variable_ref_patt_re
         out_parts = []
         last = 0
 

agent/component/base.py

@@ -351,8 +351,14 @@ class ComponentParamBase(ABC):
 class ComponentBase(ABC):
     component_name: str
     thread_limiter = asyncio.Semaphore(int(os.environ.get("MAX_CONCURRENT_CHATS", 10)))
-    variable_ref_patt = r"\{* *\{([a-zA-Z:0-9]+@[A-Za-z0-9_.-]+|sys\.[A-Za-z0-9_.]+|env\.[A-Za-z0-9_.]+)\} *\}*"
+    # Match `cpn_id@var_nm` / `sys.var_nm` / `env.var_nm` style template refs.
+    # `cpn_id` allows underscores (frontend ids like `userfillup_abc`,
+    # `retrieval_xyz`) and colons (legacy DSL ids like `UserFillUp:CateInput`,
+    # `Retrieval:KBSearch`).
+    variable_ref_patt = r"\{* *\{([a-zA-Z0-9_:]+@[A-Za-z0-9_.-]+|sys\.[A-Za-z0-9_.]+|env\.[A-Za-z0-9_.]+)\} *\}*"
+    variable_ref_patt_re = re.compile(variable_ref_patt, flags=re.IGNORECASE | re.DOTALL)
     iteration_alias_patt = r"\{* *\{(item|index|result)\} *\}*"
+    iteration_alias_patt_re = re.compile(iteration_alias_patt, flags=re.IGNORECASE | re.DOTALL)
 
     def __str__(self):
         """
@@ -481,7 +487,7 @@ class ComponentBase(ABC):
                 resolved = self._canvas.get_variable_value(v)
                 self.set_input_value(var, resolved)
                 _logger.debug("[Base]   var '%s': resolved ref '%s' -> %s", var, v, json.dumps(resolved, ensure_ascii=False, default=str)[:200])
-            elif isinstance(v, str) and re.search(self.variable_ref_patt, v):
+            elif isinstance(v, str) and self.variable_ref_patt_re.search(v):
                 elements = self.get_input_elements_from_text(v)
                 kv = {k: e.get("value", "") for k, e in elements.items()}
                 self.set_input_value(var, self.string_format(v, kv))
@@ -517,7 +523,7 @@ class ComponentBase(ABC):
 
     def get_input_elements_from_text(self, txt: str) -> dict[str, dict[str, str]]:
         res = {}
-        for r in re.finditer(self.variable_ref_patt, txt, flags=re.IGNORECASE | re.DOTALL):
+        for r in self.variable_ref_patt_re.finditer(txt):
             exp = r.group(1)
             # Use maxsplit=1 to be defensive: although `exp` here comes
             # from `variable_ref_patt` (which constrains `var_nm` to
@@ -531,7 +537,7 @@ class ComponentBase(ABC):
                 "_retrieval": self._canvas.get_variable_value(f"{cpn_id}@_references") if cpn_id else None,
                 "_cpn_id": cpn_id,
             }
-        for r in re.finditer(self.iteration_alias_patt, txt, flags=re.IGNORECASE | re.DOTALL):
+        for r in self.iteration_alias_patt_re.finditer(txt):
             exp = r.group(1)
             if exp in res:
                 continue