## What kind of change does this PR introduce?
Feature. The last remaining piece of this PR's original scope, rebased
onto current `master`.
## What is the current behavior?
A BigQuery pipeline replicates every published table into a flat
destination table. There is no way to say how those tables should be
laid out in BigQuery, so partitioning and clustering have to be applied
by hand after the fact, and are lost whenever a destination table is
reset.
The rest of this PR's original scope has since merged separately:
#49841, #49842, #49843, #49844 and #49845. The branch now carries only
the table layout work, and adds nothing to `packages/api-types`. It
consumes the `table_options` contract that #49844 already brought in.
## What is the new behavior?
The BigQuery destination form gains a "Table layout" section under
Advanced settings. Every table in the selected publication appears as a
row, and expanding one reveals optional partitioning and clustering:
- Time-column partitioning, by hour, day, month or year
- Integer-range partitioning, with start, end and interval
- Ingestion-time partitioning
- Clustering, up to four columns
Rows are always present, so nothing implies a table can be excluded
here. A collapsed row summarises what is applied: `Not configured`,
`Daily by created_at`, `Integer range by id`, plus clustering counts.
`Not configured` is dimmed a step further so configured rows are what
the eye lands on in a long publication.
Notes on behavior:
- Partition columns are resolved from the published table's real
columns, filtered to the types BigQuery accepts for each partition mode.
- A row you expand but leave empty is dropped from the payload rather
than saved half-configured. Choosing a partition mode without a column
is a validation error, not a silent drop.
- Clear returns a row to `Not configured` and keeps it in the list.
Remove is reserved for stale configuration whose table has left the
publication.
- Updates send `null` to clear previously stored table options, since
omitting the property leaves the stored value unchanged.
- Layout applies when a destination table is first created or reset,
matching the backend.
Roughly 40% of the diff is tests. The bulk of the rest is four new files
under `DestinationForm/BigQuery`, which are all one feature. The three
commits are readable in order: types and payload builders, then the
columns query, then the UI.
## To test
Open a project's Database > Replication, then create or edit a BigQuery
pipeline.
1. Expand Advanced settings. "Table layout" lists every table in the
selected publication.
2. Expand a row, set Partition by to Time column and pick a column.
Collapse. The row reads `Daily by <column>`. Hit Clear. The row returns
to `Not configured` and stays in the list.
3. Set Partition by to Time column, leave Partition column empty,
collapse, and Save. The row explains "Select a partition column" in red
rather than saving and silently dropping it.
4. Expand a row and add clustering columns. The fifth is refused.
5. Switch publications and watch the loading state. The row list should
not jump size when it resolves.
6. Narrow the sheet. The integer range Start, End and Interval fields
should reflow rather than stay in three columns.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* BigQuery replication destinations now support per-table layout
settings.
* Configure partitioning by time column, integer range, or ingestion
time.
* Configure up to four clustering columns per table.
* View available columns and validation feedback while editing table
layouts.
* Table settings are preserved when editing destinations and cleaned up
when publications change.
* **Bug Fixes**
* Improved handling of invalid, unavailable, or removed table and column
configurations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Danny White <3104761+dnywh@users.noreply.github.com>
Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
## What kind of change does this PR introduce?
Studio data-layer refactor.
## What is the current behavior?
Pipeline creation, editing, and validation build similar destination and
pipeline payloads separately. The duplicated mappings rely on type
assertions and can drift between actions.
## What is the new behavior?
Uses shared typed builders for create, update, and validation payloads
across the existing destinations. Update payloads continue to omit blank
secrets, while create payloads preserve their current values. This PR
does not add table partitioning configuration.
## To test
This is a data-layer refactor. No visible behaviour should change.
1. Open **Database > Replication** and click **Start a new pipeline**.
2. Select **BigQuery**, or any other enabled destination.
3. Edit a few non-secret fields and expand **Advanced settings**.
4. Confirm the form remains usable and no runtime errors appear.
Create, update, validation, and secret-handling behaviour is covered by
the focused tests and CI. Deploy previews and fresh local projects do
not have the existing destinations or credentials needed to exercise
those paths manually.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved replication destination configuration handling during
creation, updates, and validation.
* Applied consistent configuration mapping across supported destination
types.
* Ensured blank secret values are omitted during updates while retained
when creating destinations.
* Standardized table synchronization defaults when no specific setting
is provided.
* **Tests**
* Added coverage for BigQuery configuration mapping and secret handling.
* Updated DuckLake tests for destination updates.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
---------
Co-authored-by: Joshen Lim <joshenlimek@gmail.com>
## Context
> [!IMPORTANT]
> Changes in this PR only apply to the local environment - there should
not be any changes to staging (nor production)
Given that read replicas currently sit under database replication, the
UI currently waits for replication destinations to load before rendering
the page. However for local development, setting up of the ETL API isn't
necessary nor applicable for everyone so this indirectly adds friction
if we just want to work with read replicas.
## Changes involved
- Opting to skip retrying fetching ETL related requests if the error
returned is "replication API URL is not configured"
- This is indicative that the local platform isn't set up for ETL yet
- ^ Database replication page will hence not wait for ETL requests to
succeed before finally rendering the UI
- Node diagram will also then render properly (just read replicas)
- Add a small admonition to visualize this
<img width="1079" height="301" alt="image"
src="https://github.com/user-attachments/assets/32bd5d2f-a76e-417e-bedf-9a04de3bb305"
/>
## To test
- Will only be able to test locally - basically just head over to the
database replication page (unless you somehow already have ETL API set
up locally)
- But can also verify that there's no changes on staging preview
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved error messaging when ETL is not configured in local
development environments
* Enhanced error handling for replication API failures with better
non-retryable error detection
* **Improvements**
* Refined replication diagram rendering based on destination setup state
* Updated dropdown menu interactions for read replica management
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Riccardo Busetti