Heterogeneous GPU Dispatch
What it is
Section titled “What it is”Heterogeneous Dispatch is the planner’s ability to run parts of a query on
whatever computing fabric is fastest for the workload. When the node has a GPU or
neural processing unit available, operations like vector similarity, large
GROUP BY, and tensor-heavy expressions can be compiled to device-native
kernels and executed on the accelerator. The data motion to and from the device
is handled by the engine; your SQL does not change.
Dispatch is decided per query, not per session. The planner weighs the transport overhead of moving data to the device against the savings from running the work in parallel there. Small queries stay on CPU because the round-trip would outweigh any benefit; big tensor multiplications and high-cardinality groupings shift to the accelerator.
Why it matters
Section titled “Why it matters”Many analytical clusters already have GPUs next to the database for ML inference. Heterogeneous Dispatch can put that hardware to work for SQL too: vector-similarity searches that would saturate the CPU complete far faster on the GPU, and large aggregations finish sooner. Because the planner is conservative about when to use the accelerator, you do not pay transport cost on small workloads.
- Transparent — no CUDA, OpenCL, or device flags in your SQL.
- Cost-aware — the planner dispatches only when projected savings beat transport overhead.
- Mixed fabrics — CPU and GPU work co-operate within the same query plan.
- Vendor flexible — targets NVIDIA, AMD, and standards-based WebGPU stacks.
How you would use it
Section titled “How you would use it”No special syntax is required. You run the same SQL and the planner picks the
target; EXPLAIN shows what it decided.
-- Find the 10 closest products to a query embedding.SELECT id, TENSOR_COSINE(vec, ?) AS sim FROM products ORDER BY sim DESC LIMIT 10;Inspect the plan to see whether GPU dispatch was chosen:
EXPLAIN SELECT id, TENSOR_COSINE(vec, ?) FROM products ORDER BY 2 DESC LIMIT 10;> Plan:> GpuKernel(cosine) device=cuda:0 rows=10000000 est_ms=4> CubeScan(products.vec)You can pin a CPU plan for comparison or benchmarking:
-- Force a CPU plan (rare; for benchmarking).SET dispatch_target = 'cpu';The plan output names the device in use, the rows in play, and the projected runtime, so you can confirm the planner’s choice on the workloads you care about.
Key concepts
Section titled “Key concepts”| Concept | What it means |
|---|---|
| Cost analysis | The planner stage that decides whether device transport pays off for this query. |
| Kernel codegen | Generation of a device-native compute kernel from the query’s operators. |
| Transfer window | The slice of data moved to (or shared with) the device. |
| Unified memory | On hardware that supports it, the explicit copy is skipped entirely. |
flowchart LR
classDef a fill:#1f2640,stroke:#7c5cff,color:#e8eaf0
classDef b fill:#1f2640,stroke:#00d4ff,color:#e8eaf0
classDef g fill:#1f2640,stroke:#4ade80,color:#4ade80
classDef d fill:#1f2640,stroke:#fbbf24,color:#fbbf24
Q[SQL query]:::a --> P[Planner]:::a
P --> COST{Cost<br/>analysis}:::b
COST -->|small / row-shaped| CPU[CPU plan]:::g
COST -->|tensor / large GROUP BY| KGEN[GPU kernel]:::b
KGEN --> DEV[Device<br/>execute]:::g
DEV --> R[Result rows]:::g
CPU --> R