Stamp dbt_spellbook provenance and explicit public/visible on spell properties

Set dune.created_by=dbt_spellbook, dune.public, and dune.visible in the
four shared property macros (mark_as_spell, expose_spells, hide_spells,
expose_dataset) so that every spell lands with an explicit provenance
tag and visibility state on the Dune catalog service. The baseline
mark_as_spell applies to every model via the project-level post-hook;
expose_spells/hide_spells/expose_dataset override as before.

Also document the dbt Cloud and CI runner profile changes required for
the catalog service migration, since they cannot be configured via this
repo.

Towards DWH-317

Author: a-monteiro

dbt_macros/dune/config_trino_properties.sql

@@ -14,7 +14,9 @@
   {%- endif -%}
   {%- if target.name == 'prod' -%}
     {%- set properties = {
+            'dune.created_by': 'dbt_spellbook',
             'dune.public': 'true',
+            'dune.visible': 'true',
             'dune.data_explorer.blockchains':  blockchains | as_text,
             'dune.data_explorer.category': 'abstraction',
             'dune.data_explorer.abstraction.type': spell_type,
@@ -37,7 +39,9 @@
 {% macro hide_spells() %}
   {%- if target.name == 'prod' -%}
     {%- set properties = {
+            'dune.created_by': 'dbt_spellbook',
             'dune.public': 'false',
+            'dune.visible': 'false',
             'dune.data_explorer.category': 'abstraction',
             'dune.vacuum': '{"enabled":true}'
           } -%}

dbt_macros/dune/expose_dataset.sql

@@ -1,7 +1,9 @@
 {% macro expose_dataset(blockchains, contributors) %}
   {%- if target.name == 'prod' -%}
     {%- set properties = {
+            'dune.created_by': 'dbt_spellbook',
             'dune.public': 'true',
+            'dune.visible': 'true',
             'dune.data_explorer.blockchains':  blockchains | as_text,
             'dune.data_explorer.category': 'third_party_data',
             'dune.data_explorer.contributors': contributors | as_text,

dbt_macros/dune/mark_as_spell.sql

@@ -1,9 +1,20 @@
 {% macro mark_as_spell(this, materialization) %}
   {%- if target.name == 'prod' -%}
     {%- if model.config.materialized == "view" -%}
-      {%- set properties = { 'dune.data_explorer.category': 'abstraction' } -%}
+      {%- set properties = {
+              'dune.created_by': 'dbt_spellbook',
+              'dune.public': 'true',
+              'dune.visible': 'false',
+              'dune.data_explorer.category': 'abstraction'
+            } -%}
     {%- else -%}
-      {%- set properties = { 'dune.data_explorer.category': 'abstraction', 'dune.vacuum': '{"enabled":true}' } -%}
+      {%- set properties = {
+              'dune.created_by': 'dbt_spellbook',
+              'dune.public': 'true',
+              'dune.visible': 'false',
+              'dune.data_explorer.category': 'abstraction',
+              'dune.vacuum': '{"enabled":true}'
+            } -%}
     {%- endif -%}
     {%- set deprecated_at = model.config.get('deprecated_at', none) -%}
     {%- if deprecated_at -%}

docs/general/catalog_service_migration.md

@@ -0,0 +1,64 @@
+# Catalog service migration (DWH-317)
+
+Spellbook's underlying Trino catalogs have moved from Dune's self-hosted Hive Metastore (HMS) to the Dune catalog service. The migration is transparent for most spells, but a few things changed that you should be aware of when reading / contributing to the repo.
+
+## What changed in this repo
+
+Every spell now emits an additional set of Dune table properties on each run. The shared post-hook macros (`mark_as_spell`, `expose_spells`, `hide_spells`, `expose_dataset`) have been updated to always set:
+
+- `dune.created_by='dbt_spellbook'` — provenance tag so the catalog service can distinguish dbt-authored spells from rows written by other producers (sqlmesh, `dunectl migrate`, etc.).
+- `dune.public` — explicit `'true'` for spells created via `mark_as_spell` and `expose_spells`, `'false'` for spells created via `hide_spells`. This used to be implicit and relied on HMS defaults.
+- `dune.visible` — explicit `'false'` by default, `'true'` for spells exposed via `expose_spells` / `expose_dataset`, `'false'` in sandpit regardless. This drives Data Explorer discoverability.
+
+You do not need to change existing models. The baseline post-hook chain in every `dbt_project.yml` (`set_trino_session_property` → `optimize_spell` → `mark_as_spell`) now lands these properties for every spell automatically. Per-model `post_hook='{{ expose_spells(...) }}'` / `'{{ hide_spells() }}'` calls override the baseline as before.
+
+## Per-model opt-in patterns (reminder)
+
+| Macro | `dune.public` | `dune.visible` | Use when |
+| --- | --- | --- | --- |
+| `mark_as_spell` (default, automatic) | `true` | `false` | Intermediate / chain-specific tables, default |
+| `expose_spells(...)` (per-model `post_hook`) | `true` | `true` | Sector spells surfaced in Data Explorer |
+| `hide_spells()` (per-model `post_hook`) | `false` | `false` | Internal spells that must stay private |
+| `expose_dataset(...)` (per-model `post_hook`) | `true` | `true` | Third-party datasets |
+
+`dune.public='true'` means anyone can query the table. `dune.visible='true'` means it shows up in Data Explorer. They are independent flags.
+
+## What changed outside this repo
+
+Some configuration cannot be updated via this repo because it lives in dbt Cloud job definitions and CI runner secrets. The following items were changed out-of-band as part of the migration and are documented here for reproducibility.
+
+### dbt Cloud profile (`dunesql`)
+
+The `dunesql` profile is defined in the dbt Cloud account connection (not in any `profiles.yml` in this repo). Its Trino connection was updated to target the migrated clusters:
+
+- `database`: `hive_catalog_svc` (was `hive` on the pre-migration cluster).
+- `host`: one of the migrated spellbook clusters depending on the job (`trino-spellbook-cd.prod.internal.dunetech.io`, `trino-spellbook-daily.prod.internal.dunetech.io`, etc.).
+- `http_headers.X-Trino-Client-Tags`: `routingGroup=spellbook-<cluster>` matching the job's cluster.
+
+On the migrated clusters, the Trino catalogs `hive` and `delta_prod` are aliases for the same catalog-service-backed catalog. Both names resolve to the same rows. Models can continue to reference sources and refs without schema qualification changes.
+
+### CI runner `~/.dbt/profiles.yml`
+
+The `dunesql` profile used by GitHub Actions (`.github/workflows/dbt_run.yml` invokes `dbt ... --profile dunesql`) is provisioned into `$HOME/.dbt/profiles.yml` by the runner image / secret at job start. Its `database` and `host` values follow the same pattern as the dbt Cloud profile above. Update both together when cluster names change.
+
+### Sandpit profile (in-repo)
+
+The sandpit profile (`dbt_subprojects/*/profiles.yml`, key `spellbook-sandpit`) was already updated in-repo: `database: hive_catalog_svc` and `X-Trino-Client-Tags: routingGroup=spellbook-sandpit`. No further action needed.
+
+## Verifying a new spell lands correctly
+
+After a dbt run against prod or sandpit, the spell should appear in the catalog service with the expected properties. Query the catalog service directly from any migrated cluster:
+
+```sql
+SELECT extra_properties
+FROM system.metadata.table_properties
+WHERE catalog_name = 'hive_catalog_svc'
+  AND schema_name = '<your_schema>'
+  AND table_name = '<your_table>';
+```
+
+Expected keys on a `mark_as_spell`-only spell: `dune.created_by`, `dune.public`, `dune.visible`, `dune.data_explorer.category`, and (for tables) `dune.vacuum`.
+
+## Context
+
+The migration tracking issue is DWH-317 in Linear. See the arrakis-jobs and core repos for the cluster and plugin-side changes that back this work.