Semantic NLP & Text Encoding

Text EncodingTokenizationNLP

Beta Works · surface still evolving · works today; the introspection surface may still change

What it is

When you write to a TEXT or VARCHAR column, AetheriusDB doesn’t only store the raw bytes — it encodes repeating values through a per-column dictionary (a lexicon) and stores compact integer tokens. A repeated label like 'ACTIVE' becomes one dictionary entry plus a small token per row instead of the full string each time.

This is invisible to you as a query author. You still write WHERE status = 'ACTIVE' and you still get a TEXT value back from SELECT. The SQL surface is unchanged; only the storage and the speed differ.

flowchart LR
  classDef a fill:#1f2640,stroke:#7c5cff,color:#e8eaf0
  classDef b fill:#1f2640,stroke:#00d4ff,color:#e8eaf0
  classDef g fill:#1f2640,stroke:#4ade80,color:#4ade80

  Raw["INSERT ... VALUES ('Red', 'Ford', 'Mustang')"]:::a
  Tok["Lexicon lookup"]:::b
  IDs["Stored as compact tokens"]:::g
  Lex["Per-column lexicon dictionary"]:::a

  Raw --> Tok --> IDs
  Tok <--> Lex
  IDs -.->|on SELECT, map back to TEXT| Lex

Why it matters

Most analytical work on text is repeating-value work: country names, status enums, category labels. Storing those as full strings means slow joins (string compare), slow group-bys (hashing bytes), and wasted space. Encoding them up front flips each of those costs:

• Smaller on disk — repeated strings collapse to one dictionary entry plus a small token per row.
• Faster joins & group-bys — equality on a token is far cheaper than a byte-by-byte string compare.
• Range queries — an ordered lexicon lets WHERE status BETWEEN 'A' AND 'K' resolve as an integer range check rather than a per-row string compare.

How you use it

You don’t, directly — encoding is automatic. The only thing you’ll notice is that text-heavy queries get faster. You write ordinary SQL:

CREATE TABLE orders (
  id       BIGINT,
  status   TEXT,
  region   TEXT,
  customer TEXT,
  amount   DOUBLE
);

INSERT INTO orders VALUES (1, 'ACTIVE', 'US', 'acme', 42.0);

SELECT region, COUNT(*)
FROM   orders
WHERE  status BETWEEN 'A' AND 'K'   -- resolves to an integer range check
GROUP BY region;                    -- group key is a compact token

Key concepts

Concept	What it means
Lexicon	A per-column dictionary mapping text values to compact tokens.
Sorted lexicon	A lexicon whose tokens preserve lexical order, so range conditions become range checks.
Token stream	The encoded sequence stored for a text column — what queries actually scan.

Notes & caveats

Best on repeating values

Columns dominated by unique, high-entropy strings (UUIDs, raw URLs) benefit least — the dictionary grows as fast as the data. Keep those as text. The big wins come from columns with repeating values: enums, categories, country codes, customer tiers.

Tokens are internal — never export them

The internal tokens are an implementation detail. They are stable within one database but should not be exported, copied between deployments, or assumed to mean anything outside the engine. Always join across systems on the original text value, not on any internal id.

Tensor & Vector Types →

Pair encoded text with embedding columns.

Cognitive Kernel →

Natural-language queries over your tables.

Define Data →

The typed-schema model that makes encoding work.