ADiDaPhat Backend V1

Backend for ADiDaPhat - Tay Phuong Cuc Lac, providing the initial Knowledge-to-Media API foundation. V1 uses mock data with persistent SQLite registries; it does not call LLMs, Qdrant, or Obsidian.

Grounded generation registry (07-012 to 07-014, Sprint A)

Synchronous research, comparison, and video-script workflows record a lifecycle of pending, retrieving, generating, validating, then completed, failed, or refused. The registry stores safe structured output, prompt hashes, retrieval chunk IDs, usage and deterministic generation-citation links; it never stores API keys, provider headers, raw HTTP responses, or hidden reasoning.

GET /api/v1/generations (and /query) supports workflow/status/provider/model/template/content/date/search filters, pagination and whitelisted sorting. GET /api/v1/generations/{id}/citations returns the queryable citation relation and /usage-summary returns SQL aggregates. An optional idempotency key is bound to a deterministic request hash; a reused key returns the original generation, while a different request returns 409.

Research and video scripts accept save_as_content=true and a valid topic_id. Only validated completed generations create draft, unpublished content; the generation links to it and content citations are derived from validated chunk links. Registry backups include generations and links. Registry export bundles intentionally exclude generation logs by default.

Examples (always point --db at a non-production database for experiments):

.venv\Scripts\python.exe scripts\run_research.py --db .\scratch.db --query "compassion" --llm-provider fake
.venv\Scripts\python.exe scripts\run_compare.py --db .\scratch.db --question "Compare" --llm-provider fake
.venv\Scripts\python.exe scripts\generate_video_script.py --db .\scratch.db --topic "Compassion" --audience "general" --llm-provider fake
.venv\Scripts\python.exe scripts\list_generations.py --db .\scratch.db --usage-summary

Sprint B retains citation repair and deeper video-duration validators.

Architecture

This is a modular monolith. API routes delegate to focused services, which use SQLite repositories without changing the API layer.

The Topic Registry is the source of truth for topics (TD-0001). Every Content Registry record belongs to an existing topic and receives a topic-local content ID (TD-0001-V01). No orphan video content is allowed.

The Provider Layer exposes one LLMProvider interface with OpenAI API and local OpenAI-compatible stubs. Configuration is read from environment variables, so neither provider nor model is hardcoded.

Source-to-Retrieval foundation

The SQLite registry now distinguishes a Source (the intellectual work), a Document (a concrete PDF/Markdown/DOCX/TXT representation), a Chunk (retrievable text), and a Citation (a Content-to-source claim). The trace chain is Content → Citation → Chunk → Document → Source. New APIs are /sources, /documents, /chunks, and /citations under /api/v1, with list/query, get, patch, and /trace; content also exposes /contents/{id}/citation-health.

POST /api/v1/retrieval remains compatible and now uses SQLite text matching over chunks when data exists. It supports source_ids, document_ids, language, citation_required, and verified_only; BGE-M3, Qdrant, embeddings, and vectors are deliberately not installed or used. Set DOCUMENT_ROOT (default data/documents) and RETRIEVER_PROVIDER=sqlite as needed.

Local metadata registration and prepared-chunk import:

.venv\Scripts\python.exe scripts\register_document.py --source-id SRC-1 --document-id DOC-1 --file .\data\documents\text.txt --document-type txt --title "Text"
.venv\Scripts\python.exe scripts\import_chunks.py --input .\chunks.json --dry-run

Embedding, Qdrant and reconciliation (Completion Sprint A)

memory is the default vector store and is process-local: it is useful for one command or test process, but is not a persistent index. Set VECTOR_STORE_PROVIDER=qdrant to use Qdrant through QdrantVectorStore. Points use deterministic UUID5 identifiers derived from chunk_id (uuid5(URL, "adidaphat:<chunk_id>")); collection dimension and distance (cosine, dot, or euclid) are checked before use. Qdrant client/API-key configuration is never printed.

SQLite is still the registry/source of truth. It holds indexing metadata only, never raw vectors. Startup does not reconcile or mutate an index. Reconciliation is an explicit CLI operation and reports healthy, missing_point, unexpected_point, stale_vector, stale_payload, orphan_point, invalid_payload, and point_id_mismatch. A stale vector is re-embedded when the text hash or embedding text hash no longer agrees; stale payload is safely re-upserted through reindexing. A mismatched ID creates the deterministic point but leaves the wrong point intact. Orphans are deleted only with both --repair and --delete-orphans; invalid payloads that cannot be matched to a SQLite chunk are reported, not deleted.

.venv\Scripts\python.exe scripts\index_chunks.py --db .\data\adidaphat_registry.db --vector-store memory --embedding-provider deterministic --mode all
.venv\Scripts\python.exe scripts\reconcile_vector_index.py --db .\data\adidaphat_registry.db --vector-store qdrant --dry-run
.venv\Scripts\python.exe scripts\reconcile_vector_index.py --db .\data\adidaphat_registry.db --repair --delete-orphans
.venv\Scripts\python.exe scripts\qdrant_collection.py --check --collection adidaphat_chunks --dimension 32 --distance cosine
.venv\Scripts\python.exe scripts\qdrant_collection.py --recreate --confirm-recreate --collection adidaphat_chunks --dimension 32

Sprint B remains the hybrid retrieval milestone; this sprint only establishes deterministic embedding/index mapping and reconciliation.

Semantic and hybrid retrieval (Completion Sprint B)

Retrieval providers are sqlite_text, qdrant_semantic, and hybrid (RETRIEVER_PROVIDER). Semantic retrieval embeds the query, gets Qdrant candidates, then hydrates the current chunk, document, source, and citations from SQLite. Qdrant payload text is never returned as source text. Orphan/invalid points and stale hash candidates are skipped with deterministic warnings; retrieval never reindexes or mutates the vector store.

citation_required=true requires a real trace: citation locator, citation page range, or chunk page range. verified_only=true requires at least one SQLite citation whose verification_status is verified; a source or indexed chunk alone is not verification. Results include citation metadata, choose a verified citation first, and expose citation_complete.

hybrid runs semantic and SQLite text branches with a bounded candidate pool, deduplicates by chunk_id, and uses reciprocal-rank fusion rather than mixing raw scores:

score = semantic_weight / (rrf_k + semantic_rank)
      + text_weight / (rrf_k + text_rank)

Defaults are HYBRID_SEMANTIC_WEIGHT=0.7, HYBRID_TEXT_WEIGHT=0.3, and HYBRID_RRF_K=60. Ordering is stable by score, branch ranks, and chunk ID. If semantic retrieval is unavailable and ALLOW_RETRIEVAL_FALLBACK=true, the response explicitly becomes sqlite_text with fallback_used=true and a warning. Set RETRIEVAL_FALLBACK_PROVIDER=sqlite_text for the lightweight local mode.

# Lightweight local mode: deterministic embeddings and process-local memory
$env:RETRIEVER_PROVIDER='hybrid'
$env:VECTOR_STORE_PROVIDER='memory'

# Inspect configured retrieval without loading BGE-M3 or calling Qdrant
curl http://127.0.0.1:8000/api/v1/retrieval/health

Real BGE-M3 plus Qdrant is optional; tests do not require a GPU, network, or server. This milestone has no reranker, background indexing worker, ChatGPT/OpenAI SDK integration, OpenWebUI, or frontend.

Setup

From backend:

python -m venv .venv

Activate the environment using the command appropriate to your operating system, then install dependencies:

python -m pip install -r requirements.txt

Create .env from .env.example and provide only the values needed for the environment. Do not commit API keys.

Run

python -m uvicorn app.main:app --reload

Open WebUI / OpenAI-compatible API (07-015)

Configure ADIDAPHAT_API_KEYS and start the backend, then point Open WebUI's OpenAI-compatible connection at http://<backend-host>:8000/v1. Use the configured key as its API key and select one of adidaphat-chat, adidaphat-research, adidaphat-compare, or adidaphat-video-script.

curl http://127.0.0.1:8000/v1/models -H "Authorization: Bearer <key>"
curl http://127.0.0.1:8000/v1/chat/completions -H "Authorization: Bearer <key>" -H "Content-Type: application/json" -d '{"model":"adidaphat-research","messages":[{"role":"user","content":"Explain compassion"}]}'

Responses are validated Markdown with backend citations. Compare needs explicit left/right source IDs in metadata; video-script accepts bounded workflow metadata. save_as_content requires topic_id and saves only a draft. See Open WebUI integration.

Content review and publication workflow (07-016)

Content revisions are immutable Markdown snapshots. Use /api/v1/contents/{content_id}/revisions to create a revision, submit it to /reviews, add comments, request changes or approve it, then create /publication-schedules or /publications. Only the currently approved revision can be scheduled or published; schedules and publication records are retained for audit. See workflow architecture.

Test

python -m pytest

Tests construct the application with temporary SQLite files and do not change production data.

Registry database and migration

The default registry database is data/adidaphat_registry.db, resolved from the backend project root rather than the current directory. Set DATABASE_PATH in the environment or .env to override it (for example, a separate local instance or test file). Back up the SQLite file before manual replacement or destructive maintenance.

The legacy JSON files remain migration sources. Import them once with:

python scripts/migrate_json_to_sqlite.py

Use --db, --topics, and --contents to override the corresponding paths. The migration imports topics first, skips existing IDs on repeat runs, and reports skipped invalid content references.

Registry queries and audit history

The existing GET /api/v1/topics and GET /api/v1/contents endpoints retain their {items, total} response contract. Use the opt-in query endpoints for SQL pagination, search, combined filters, and sorting:

curl "http://127.0.0.1:8000/api/v1/topics/query?limit=50&offset=0&q=niem&sort_by=updated_at&sort_order=desc"
curl "http://127.0.0.1:8000/api/v1/contents/query?topic_id=TD-0001&status=draft&content_type=video&limit=20"

Both query endpoints return:

{"items": [], "total": 0, "limit": 50, "offset": 0, "has_more": false}

limit defaults to 50 and must be from 1 through 200; offset defaults to 0. Topic queries support q, status, primary_source, sort_by, and sort_order. Content queries additionally support topic_id, content_type, production_status, publication_status, and source_id. Search is case-insensitive substring matching; list-valued fields are searched as JSON text in this version.

Every successful create or changed update records an audit event in SQLite. Read newest-first history with:

curl "http://127.0.0.1:8000/api/v1/topics/TD-0001/history?limit=50"
curl "http://127.0.0.1:8000/api/v1/contents/TD-0001-V01/history?action=status_changed"

Audit events currently have no authenticated actor. Back up data/adidaphat_registry.db before any manual SQLite maintenance. Run all tests with:

.venv\Scripts\python.exe -m pytest -v --basetemp .test-tmp -p no:cacheprovider

Bulk registry I/O and backup

Import accepts a JSON array or { "items": [...] }. Registry bundles use:

{"format":"adidaphat-registry","version":1,"topics":[],"contents":[]}

Use POST /api/v1/import/topics, /import/contents, or /import/registry. Query parameters are mode=skip|error|upsert, dry_run=true|false, and atomic=true|false; the defaults are skip, false, and true. Every result includes a per-record status plus received/valid/inserted/updated/skipped/error totals. Dry runs never write rows or audit events. Atomic imports validate first and write nothing when any record is invalid; non-atomic imports retain valid records.

Exports are JSON from GET /api/v1/export/topics, /export/contents, and /export/registry. Topic/content export filters mirror their query endpoints; registry export is a versioned bundle with deterministic IDs and counts.

Local CLI usage (PowerShell):

.venv\Scripts\python.exe scripts\import_registry.py --db .\data\adidaphat_registry.db --entity registry --input .\registry.json --dry-run
.venv\Scripts\python.exe scripts\export_registry.py --db .\data\adidaphat_registry.db --entity registry --output .\registry-export.json
.venv\Scripts\python.exe scripts\backup_registry.py --db .\data\adidaphat_registry.db
.venv\Scripts\python.exe scripts\restore_registry.py --source .\data\backups\adidaphat-registry-YYYYMMDDTHHMMSSZ.db --confirm-restore

Exports never overwrite without --force. Backups use SQLite's backup API and run integrity and foreign-key checks; default backups go to data/backups. Restore is deliberately CLI-only, requires --confirm-restore, creates a pre-restore backup, validates the source and uses atomic replacement. Stop the backend before restore. There is no authenticated actor, API restore, scheduled backup, or cloud backup in this version.

API Examples

Grounded LLM workflow validation

Research, comparison and video-script workflows accept only citation keys supplied by retrieval context. Compare keeps left (L-CIT-*) and right (R-CIT-*) evidence separate. Quotes must exactly match cited source text after whitespace normalization. Invalid structured output receives at most one repair attempt; token usage is aggregated and content is saved only after the repaired output validates. Validation errors are safe 422 responses and do not expose provider responses, secrets or hidden reasoning.

curl http://127.0.0.1:8000/api/v1/health
curl -X POST http://127.0.0.1:8000/api/v1/topics -H "Content-Type: application/json" -d '{"title":"Niem Phat","slug":"niem-phat"}'
curl -X POST http://127.0.0.1:8000/api/v1/contents -H "Content-Type: application/json" -d '{"topic_id":"TD-0001","title":"Niem Phat Moi Ngay","slug":"niem-phat-moi-ngay","content_angle":"Daily practice","primary_message":"Maintain mindfulness","target_duration_seconds":600}'
curl -X POST http://127.0.0.1:8000/api/v1/retrieval -H "Content-Type: application/json" -d '{"query":"niem Phat","top_k":3}'

Content workflow security

The content-review/publication write API uses the configured Bearer authentication policy. Workflow actors are server-owned principals, idempotency keys are principal scoped, and client metadata plus publication URLs are structurally validated before persistence. See the content workflow architecture for limits, idempotency behavior, migration safety, and test-only rollback seams.