`yggdrasil.databricks.ai.vector_search`¶

vector_search ¶

Databricks Vector Search service wrappers.

VectorSearchDefaults `dataclass` ¶

VectorSearchDefaults(
    endpoint_name: Optional[str] = None,
    endpoint_type: str = "STANDARD",
    index_type: str = "DELTA_SYNC",
    pipeline_type: str = "TRIGGERED",
    embedding_model_endpoint_name: Optional[str] = None,
    wait: WaitingConfig = DEFAULT_VS_WAIT,
)

Default configuration for :class:VectorSearch.

Set once on the service and every subsequent call inherits these values unless overridden inline::

from dataclasses import replace
client.ai.vector_search.defaults = replace(
    client.ai.vector_search.defaults,
    endpoint_name="rag-endpoint",
    embedding_model_endpoint_name="databricks-bge-large-en",
)

Attributes¶

endpoint_name Default endpoint name used by :meth:VectorSearch.index / :meth:VectorSearch.create_delta_sync_index / :meth:VectorSearch.create_direct_access_index when none is passed inline. endpoint_type Endpoint type used by :meth:VectorSearchEndpoint.create / :meth:VectorSearchEndpoint.ensure_created when the caller does not specify one. "STANDARD" (cheap, shared) by default; "STORAGE_OPTIMIZED" is the larger-billing tier for cold storage workloads. Accepts the SDK enum or its string name. index_type Default :class:VectorIndexType for :meth:VectorSearchIndex.create. "DELTA_SYNC" is the most common path — the index is driven by a UC Delta source table; "DIRECT_ACCESS" lets the caller upsert / delete rows directly. pipeline_type Default :class:PipelineType for delta-sync indexes. "TRIGGERED" is cheaper (sync on demand); "CONTINUOUS" keeps the index in lockstep with the source Delta table. embedding_model_endpoint_name Name of a serving endpoint that produces embeddings (e.g. "databricks-bge-large-en"). When set, delta-sync indexes created via the managed-embedding shape route source columns through this endpoint automatically — callers no longer have to pre-compute and store the embedding vector column. wait :class:~yggdrasil.dataclasses.WaitingConfig carrying the budget for long-running endpoint / index operations (wait.timeout) and the polling cadence (wait.interval). Defaults to :data:DEFAULT_VS_WAIT (20 minutes / 5 seconds). Override per-call by passing wait= — anything :meth:WaitingConfig.from_ accepts works (seconds, timedelta, deadline, dict, full WaitingConfig).

VectorSearchEndpoint ¶

VectorSearchEndpoint(
    service: "VectorSearch",
    endpoint_name: str,
    *,
    details: "Optional[EndpointInfo]" = None
)

Bases: DatabricksResource

A Databricks Vector Search endpoint.

Endpoints are the compute layer — one endpoint hosts one or more :class:VectorSearchIndex instances. Provisioning is asynchronous; use :meth:wait_online (or pass wait=True to :meth:create / :meth:ensure_created) when the next step needs the endpoint to be serving queries.

sql `property` ¶

sql: 'SQLEngine'

Shorthand for self.service.client.sql — the active :class:SQLEngine.

infos `property` ¶

infos: 'EndpointInfo'

Cached :class:EndpointInfo — fetched on first access.

state `property` ¶

state: Optional[str]

Lifecycle state from the endpoint status (ONLINE / PROVISIONING / …).

refresh ¶

refresh() -> 'VectorSearchEndpoint'

Re-fetch :attr:infos from the API.

exists ¶

exists() -> bool

True when the endpoint exists in the workspace.

create ¶

create(
    *,
    endpoint_type: Any = None,
    budget_policy_id: Optional[str] = None,
    target_qps: Optional[int] = None,
    wait: WaitingConfigArg = None,
    missing_ok: bool = True
) -> "VectorSearchEndpoint"

Create this endpoint.

Parameters¶

endpoint_type Endpoint type. Accepts the SDK enum or its string name. Defaults to :attr:VectorSearchDefaults.endpoint_type. budget_policy_id Optional budget-policy id to attribute usage to. target_qps Optional target queries-per-second the endpoint should scale to. Only applicable to STANDARD endpoints. wait Per-call wait budget. None (the default) returns as soon as the SDK accepts the create request — endpoint provisioning typically takes 5-10 minutes and most callers don't want to block. Pass True to wait with :attr:VectorSearchDefaults.wait, or anything :meth:WaitingConfig.from_ accepts (seconds, timedelta, deadline, dict, full WaitingConfig). missing_ok When True (the default), an existing endpoint with the same name is treated as success and the cached :attr:infos is refreshed instead of raising.

ensure_created ¶

ensure_created(
    *,
    endpoint_type: Any = None,
    budget_policy_id: Optional[str] = None,
    target_qps: Optional[int] = None,
    wait: WaitingConfigArg = None
) -> "VectorSearchEndpoint"

Create this endpoint when missing, otherwise return self.

delete ¶

delete(*, missing_ok: bool = False) -> None

Delete this endpoint.

Parameters¶

missing_ok When True, a missing endpoint is treated as success.

wait_online ¶

wait_online(*, wait: WaitingConfigArg = None) -> 'VectorSearchEndpoint'

Block until the endpoint reaches ONLINE (or the budget elapses).

index ¶

index(index_name: str) -> 'VectorSearchIndex'

Return a :class:VectorSearchIndex handle bound to this endpoint.

indexes ¶

indexes() -> Iterator['VectorSearchIndex']

Iterate over indexes hosted on this endpoint.

VectorSearchIndex ¶

VectorSearchIndex(
    service: "VectorSearch",
    index_name: str,
    *,
    endpoint_name: Optional[str] = None,
    details: "Optional[VectorIndex]" = None
)

Bases: DatabricksResource

A Databricks Vector Search index.

Indexes are UC-governed three-part identifiers (catalog.schema.name); callers refer to them by that full name. The :attr:endpoint_name is required for create / sync / pagination operations but is read back from :attr:infos for queries — the SDK only needs the index name for :meth:query.

Two index types are supported:

DELTA_SYNC — driven by a UC Delta source table. Create via :meth:create_delta_sync (the source table's primary key is the index PK, and the embedding either comes from a precomputed vector column or via a managed-embedding endpoint).
DIRECT_ACCESS — caller-managed rows. Create via :meth:create_direct_access, then :meth:upsert / :meth:delete_rows.

sql `property` ¶

sql: 'SQLEngine'

Shorthand for self.service.client.sql — the active :class:SQLEngine.

endpoint_name `property` ¶

endpoint_name: Optional[str]

Endpoint this index is hosted on.

Resolves from the most informative source available: an explicit constructor arg, the cached :attr:_details, then the service default. The infos-roundtrip path is left for the explicit :meth:refresh so attribute access stays free of API calls.

infos `property` ¶

infos: 'VectorIndex'

Cached :class:VectorIndex — fetched on first access.

refresh ¶

refresh() -> 'VectorSearchIndex'

Re-fetch :attr:infos from the API.

create_delta_sync ¶

create_delta_sync(
    *,
    source_table: str,
    primary_key: str,
    embedding_source_columns: Optional[
        Sequence[Union[str, "EmbeddingSourceColumn"]]
    ] = None,
    embedding_vector_columns: Optional[
        Sequence[Union[Mapping[str, Any], "EmbeddingVectorColumn"]]
    ] = None,
    embedding_model_endpoint_name: Optional[str] = None,
    pipeline_type: Any = None,
    columns_to_sync: Optional[Sequence[str]] = None,
    embedding_writeback_table: Optional[str] = None,
    endpoint_name: Optional[str] = None,
    wait: WaitingConfigArg = None,
    missing_ok: bool = True
) -> "VectorSearchIndex"

Create a DELTA_SYNC index backed by a UC Delta source table.

Pick exactly one embedding shape:

Managed embeddings — pass embedding_source_columns (a list of column names) and either embedding_model_endpoint_name or the service default. The endpoint embeds each source value automatically.
Self-managed embeddings — pass embedding_vector_columns (each entry being a mapping {"name": "<col>", "embedding_dimension": <int>} or a fully-built :class:EmbeddingVectorColumn). The vector column must already exist on the source table.

create_direct_access ¶

create_direct_access(
    *,
    primary_key: str,
    schema: "Optional[SchemaLike]" = None,
    schema_json: Optional[str] = None,
    embedding_source_columns: Optional[
        Sequence[Union[str, "EmbeddingSourceColumn"]]
    ] = None,
    embedding_vector_columns: Optional[
        Sequence[Union[Mapping[str, Any], "EmbeddingVectorColumn"]]
    ] = None,
    embedding_model_endpoint_name: Optional[str] = None,
    endpoint_name: Optional[str] = None,
    wait: WaitingConfigArg = None,
    missing_ok: bool = True
) -> "VectorSearchIndex"

Create a DIRECT_ACCESS index that the caller upserts into.

schema is the row schema in any of the shapes :func:_schema_to_json accepts:

:class:yggdrasil.data.Schema — the canonical surface; field names, types, and nullability stay in lockstep with the curated table backing the index.
:class:pyarrow.Schema — the Arrow-native shortcut.
Mapping[str, str] — the Databricks-native flat {"<col>": "<sql_type>"} shape.

schema_json is the legacy raw JSON string, kept for callers already speaking the wire format. Pass exactly one of schema / schema_json.

Provide either embedding_source_columns (managed embeddings) or embedding_vector_columns (self-managed) the same way as :meth:create_delta_sync.

delete ¶

delete(*, missing_ok: bool = False) -> None

Delete this index.

sync ¶

sync() -> 'VectorSearchIndex'

Trigger a sync for a DELTA_SYNC index (no-op for direct-access).

wait_ready ¶

wait_ready(*, wait: WaitingConfigArg = None) -> 'VectorSearchIndex'

Poll :attr:infos until status.ready is True (or budget elapses).

upsert ¶

upsert(rows: 'RowsLike') -> Any

Upsert rows into a DIRECT_ACCESS index.

rows accepts every shape the rest of the project speaks:

:class:pyarrow.Table — preferred when the data comes from the cast registry / Delta / curated tables. Conversion to the wire's row-dict shape happens in one to_pylist hop at the boundary (vector-search upserts are a genuine row endpoint — see CLAUDE.md).
:class:polars.DataFrame — routed through Arrow so dtype intent survives.
:class:pandas.DataFrame — to_dict(orient='records').
Sequence[Mapping] — already row-shaped, used as-is.

The vector column is either pre-computed (self-managed embeddings) or filled in server-side by the managed embedding endpoint.

delete_rows ¶

delete_rows(primary_keys: Sequence[str]) -> Any

Delete rows by primary key from a DIRECT_ACCESS index.

scan ¶

scan(
    *, num_results: Optional[int] = None, last_primary_key: Optional[str] = None
) -> Any

Scan rows (direct-access maintenance) returning the raw response.

query ¶

query(
    *,
    columns: Sequence[str],
    query_text: Optional[str] = None,
    query_vector: Optional[Sequence[float]] = None,
    num_results: Optional[int] = 10,
    filters: Optional[Union[str, Mapping[str, Any]]] = None,
    query_type: Optional[str] = None,
    columns_to_rerank: Optional[Sequence[str]] = None,
    reranker: "Optional[RerankerConfig]" = None,
    score_threshold: Optional[float] = None,
    target_schema: "Optional[Schema]" = None
) -> "VectorSearchQueryResult"

Run a similarity / hybrid / full-text query against this index.

Parameters¶

columns Result columns to return. The score column (__db_score) is appended automatically by Databricks. query_text Natural-language query — embedded by the index's managed embedding endpoint. Mutually exclusive with query_vector. query_vector Pre-computed embedding vector. Mutually exclusive with query_text. num_results Maximum number of results to return. filters Optional metadata filter. Either a JSON string (the SDK's filters_json) or a mapping that is serialised to JSON via :mod:yggdrasil.pickle.json. query_type "ANN" (default), "HYBRID", or "FULL_TEXT". columns_to_rerank Columns whose values should be considered by the reranker. reranker Optional :class:RerankerConfig. score_threshold Optional minimum score; rows below the threshold are dropped server-side. target_schema Optional :class:yggdrasil.data.Schema pinned on the returned :class:VectorSearchQueryResult. When set, the result's :meth:VectorSearchQueryResult.to_arrow_table casts each result column through the yggdrasil cast registry to honour the caller's dtype / nullability / timezone intent (matches the CastOptions(target_field=...) pattern used by :class:StatementResult and Genie).

VectorSearchQueryResult ¶

VectorSearchQueryResult(
    index: VectorSearchIndex,
    response: "QueryVectorIndexResponse",
    *,
    target_schema: "Optional[Schema]" = None
)

Wrapper around a :class:QueryVectorIndexResponse.

Carries the column manifest, the raw row data (as the API returned it — :class:list of :class:list of :class:str), and the pagination token. Materialises the result as Arrow / Polars / pandas through the registered cast paths.

next_page ¶

next_page() -> Optional['VectorSearchQueryResult']

Fetch the next page, or None when none is available.

iter_pages ¶

iter_pages() -> Iterator['VectorSearchQueryResult']

Yield self then every subsequent page.

to_arrow_table ¶

to_arrow_table(*, target_schema: 'Optional[Schema]' = None) -> 'pa.Table'

Materialise the result as a :class:pyarrow.Table.

Each column is cast to the Arrow type resolved from its type_text; unknown / complex types stay as strings so the original byte payload is preserved.

When target_schema is set (either inline or via the :attr:target_schema carried from :meth:VectorSearchIndex.query), the assembled Arrow table is run through the yggdrasil cast registry so the caller's dtype / nullability / timezone intent is honoured — the same path :class:StatementResult and :class:GenieAnswer already use.

to_polars ¶

to_polars(*, target_schema: 'Optional[Schema]' = None) -> 'pl.DataFrame'

Materialise the result as a :class:polars.DataFrame.

to_pandas ¶

to_pandas(*, target_schema: 'Optional[Schema]' = None) -> 'pd.DataFrame'

Materialise the result as a :class:pandas.DataFrame.

to_dicts ¶

to_dicts() -> list[dict[str, Any]]

Return the rows as a list of dicts keyed by column name.

Genuine row endpoint: vector-search query results are typically consumed as [{"id": …, "text": …, "score": …}, ...] payloads handed straight to a downstream RAG prompt / JSON response.

VectorSearch ¶

VectorSearch(client=None, defaults: Optional[VectorSearchDefaults] = None)

Bases: DatabricksService

High-level wrapper around Databricks Vector Search APIs.

Attributes¶

defaults :class:VectorSearchDefaults — service-wide configuration. Replace via client.ai.vector_search.defaults = replace(...) the same way :class:~yggdrasil.databricks.genie.Genie does.

wheels `property` ¶

wheels: 'Wheels'

Wheel registry service (shorthand for client.wheels).

environments `property` ¶

environments: 'Environments'

Base-environment service (shorthand for client.environments).

tables `property` ¶

tables: 'Tables'

Collection-level Unity Catalog table service (shorthand for client.tables).

views `property` ¶

views: 'Tables'

Alias for :attr:tables — :class:Table covers both managed/external tables and view-shaped securables.

catalogs `property` ¶

catalogs: 'Catalogs'

Collection-level Unity Catalog hierarchy service (shorthand for client.catalogs).

schemas `property` ¶

schemas: 'Schemas'

Collection-level Unity Catalog schema service (shorthand for client.schemas).

volumes `property` ¶

volumes: 'Volumes'

Collection-level Unity Catalog volume service (shorthand for client.volumes).

genie `property` ¶

genie: 'Genie'

Genie service (shorthand for client.genie).

ai `property` ¶

ai: 'DatabricksAI'

Databricks AI umbrella service (shorthand for client.ai).

default_tags ¶

default_tags(update: bool = True) -> dict[str, str]

Return default resource tags for Databricks assets.

Returns:

Type	Description
`dict[str, str]`	A dict of default tags.

endpoint ¶

endpoint(endpoint_name: Optional[str] = None) -> VectorSearchEndpoint

Return a :class:VectorSearchEndpoint handle.

endpoint_name defaults to :attr:VectorSearchDefaults.endpoint_name.

list_endpoints ¶

list_endpoints() -> Iterator[VectorSearchEndpoint]

Iterate over vector-search endpoints visible in this workspace.

find_endpoint ¶

find_endpoint(*, name: Optional[str] = None) -> Optional[VectorSearchEndpoint]

Return the endpoint with this name, or None when missing.

index ¶

index(
    index_name: str, *, endpoint_name: Optional[str] = None
) -> VectorSearchIndex

Return a :class:VectorSearchIndex handle.

endpoint_name resolves to the explicit arg, then :attr:VectorSearchDefaults.endpoint_name, then the value carried by the cached :class:VectorIndex infos on first :meth:VectorSearchIndex.refresh.

list_indexes ¶

list_indexes(
    *, endpoint_name: Optional[str] = None
) -> Iterator[VectorSearchIndex]

Iterate over indexes hosted on a given endpoint.

yggdrasil.databricks.ai.vector_search¶

vector_search ¶

VectorSearchDefaults dataclass ¶

Attributes¶

VectorSearchEndpoint ¶

sql property ¶

infos property ¶

state property ¶

refresh ¶

exists ¶

create ¶

Parameters¶

ensure_created ¶

delete ¶

Parameters¶

wait_online ¶

index ¶

indexes ¶

VectorSearchIndex ¶

sql property ¶

endpoint_name property ¶

infos property ¶

refresh ¶

create_delta_sync ¶

create_direct_access ¶

delete ¶

sync ¶

wait_ready ¶

upsert ¶

delete_rows ¶

scan ¶

query ¶

Parameters¶

VectorSearchQueryResult ¶

next_page ¶

iter_pages ¶

to_arrow_table ¶

to_polars ¶

to_pandas ¶

to_dicts ¶

VectorSearch ¶

Attributes¶

wheels property ¶

environments property ¶

tables property ¶

views property ¶

catalogs property ¶

schemas property ¶

volumes property ¶

genie property ¶

ai property ¶

default_tags ¶

endpoint ¶

list_endpoints ¶

find_endpoint ¶

index ¶

list_indexes ¶

`yggdrasil.databricks.ai.vector_search`¶

VectorSearchDefaults `dataclass` ¶

sql `property` ¶

infos `property` ¶

state `property` ¶

sql `property` ¶

endpoint_name `property` ¶

infos `property` ¶

wheels `property` ¶

environments `property` ¶

tables `property` ¶

views `property` ¶

catalogs `property` ¶

schemas `property` ¶

volumes `property` ¶

genie `property` ¶

ai `property` ¶