Multimodal Fabric

MultimodalEmbeddingsVisionSpeech

Roadmap Planned · not yet usable · aspirational AI ingest pipeline; verify availability before relying on it

What it is

In a traditional database a media file is opaque — a blob you can only use by handing it to a separate service. The Multimodal Fabric removes that boundary. When you insert an image, audio clip, or video into a MULTIMODAL column, the engine runs the appropriate model and stores both the original bytes and the extracted meaning: vector embeddings, detected objects, transcripts, OCR text.

The extracted metadata is materialized as proper typed structures — tensor columns for embeddings, a graph relation linking media to the entities it contains, and indexed text for transcripts — so queries like “find every video where a red backpack is visible” are ordinary SQL.

flowchart LR
  classDef in fill:#1f2640,stroke:#9ca3af,color:#e8eaf0
  classDef m  fill:#1f2640,stroke:#7c5cff,color:#e8eaf0
  classDef out fill:#1f2640,stroke:#4ade80,color:#4ade80

  MEDIA[(Media INSERT<br/>image · audio · video)]:::in
  MEDIA --> ROUTE[Route by content type]:::m
  ROUTE --> VIS[Vision: objects · OCR · embedding]:::m
  ROUTE --> AUD[Speech: transcript]:::m
  VIS --> EMB[TENSOR embedding column]:::out
  VIS --> GR[Graph: media → entities]:::out
  AUD --> TXT[Indexed transcript column]:::out

Why it matters

Normally a media file is a dead weight in the database — a blob you can store but not query, useful only after you hand it to a separate vision or speech service. The Multimodal Fabric extracts the meaning on the way in and materializes it as proper typed structures, so “find every video with a red backpack” or “search the transcripts” is ordinary SQL over columns and relations you already know how to query.

• Media becomes queryable — embeddings, detected objects, transcripts, and OCR text are extracted at ingest, not deferred to a downstream service.
• Typed, not opaque — the extracted meaning lands as tensor columns, graph relations, and indexed text — first-class things SQL can filter and join.
• One system — no separate CV/ASR pipeline to wire up and keep in sync with the rows.

Fast at query time

The model cost is paid once, at ingest; queries afterward touch only the materialized structures, so searching by object, transcript, or similarity is a normal indexed SQL query — not a per-request model call.

Roadmap

The Multimodal Fabric is a Roadmap capability — check availability before relying on it.

How you use it

Declare a column with a MULTIMODAL type. From there, inserts are normal SQL, and queries can reference the auto-generated columns and graph relations directly.

-- Declare a multimodal video column.
CREATE TABLE surveillance_feed (
  id        BIGINT,
  camera_id BIGINT,
  captured  TIMESTAMP,
  video     MULTIMODAL('H.264')
);

-- INSERT a clip. The engine extracts frames, runs the vision model, and
-- writes objects + embeddings + graph edges automatically.
INSERT INTO surveillance_feed VALUES
  (1, 42, NOW(), load_file('/uploads/clip-001.mp4'));

-- "Find every clip where a man with a red backpack appears."
-- visual_match encodes the query phrase once and compares against the
-- per-frame embeddings stored alongside the row.
SELECT id, camera_id, video_offset
FROM surveillance_feed
WHERE visual_match(video, 'man with a red backpack') > 0.78
ORDER BY captured DESC;

-- "Show every video that contains a Person entity, with its timestamp."
SELECT sf.id, e.label, sf.video_offset
FROM surveillance_feed sf
  JOIN _sys_media_entities e ON e.media_id = sf.id
WHERE e.label = 'Person';

Key concepts

Concept	What it means
MULTIMODAL column	Stores raw media bytes alongside auto-generated derivatives (embeddings, transcripts).
Graph auto-bind	Extracted entities become graph nodes connected to the source media.
visual_match	A similarity function that encodes a text phrase once and compares against per-row embeddings.

Notes & caveats

Swap in domain models

The default vision and speech models are general-purpose. For a domain like medical imaging or retail product matching, replace them with a fine-tuned model at deploy time — the rest of the pipeline doesn’t change.

Ingest is heavier than a plain blob

Multimodal inserts run model inference inline, so commit latency is dominated by model cost (tens to hundreds of milliseconds per image). For high-throughput streams, use the async ingest path so commits don’t block on inference.

Vector Search →

How the generated embeddings power kNN.

Cognitive Kernel →

The in-process model runtime behind the models.

AI Quality Constraints →

Ingest-time intelligence for numeric data.

Compute →

The in-database execution model.