AI search and Qdrant
Four features use OpenAI; one of them also uses Qdrant. The key is read from the oddly named OpenAI variable (OPENAI_API_KEY as fallback).
| Feature | Endpoint | Model | Code |
|---|---|---|---|
| Meaning-based Bible search and similar verses | POST /rest/bibles/search/, POST /rest/bibles/similar/ | text-embedding-3-large (3072 dims) + Qdrant collection niv | bible_search/services.py |
| Verse finder ("find me a verse about…") | POST /rest/ai/find-verse/ | gpt-4o-mini-2024-07-18, structured output | ai/services.py (+ ai/moderation.py, omni-moderation-latest) |
| Verse audio (read a memory verse aloud) and recitation scoring | POST /rest/generate-audio-file/, POST /rest/analyze-verse-audio/ | tts-1; whisper-1 + gpt-4o (needs ffmpeg / pydub) | EMAW/views/memory_verses.py; audio stored in Azure Blob |
| Email template drafts | /rest/campaigns/templates/ai-drafts/ (Celery job + polling) | AI_TEMPLATE_MODEL (default gpt-4o) | campaigns/ai_templates.py |
All are authenticated and throttled with AIThrottle. The search endpoints are only registered when AI_SEARCH_ENABLED is set (always on in production).
How semantic search works
Embeddings were generated once for the NIV (AI_SEARCH_BASE_VERSION) and stored as a parquet file in blob storage. Any version can be displayed: the references found in NIV are looked up in the requested version's SQLite file.
The clients fall back when offline: the mobile app uses the local SQLite FTS5 table; the web app uses rest/bibles/<version>/search/ keyword search.
Loading Qdrant
Qdrant starts with an empty qdrant_storage volume on every new machine. Load once:
# inside the stack (dev VM, laptop, or production via Portainer console)
docker compose exec django python manage.py load_bible_embeddings
It downloads the parquet from EMBEDDINGS_URL (default: the blob URL) into EMBEDDINGS_DIR (the embeddings_data volume) and upserts into the niv collection in batches (QDRANT_BATCH_SIZE, QDRANT_MAX_RETRIES, QDRANT_BACKOFF_SECONDS). About 500 MB and a few minutes.
From the host with only Qdrant in Docker:
docker compose up -d qdrant
$env:QDRANT_HOST="localhost"; $env:QDRANT_PORT="7002"
.venv\Scripts\python.exe manage.py load_bible_embeddings
Verify:
curl http://127.0.0.1:7002/collections # should list "niv"
curl http://127.0.0.1:7002/collections/niv # points_count > 0
To regenerate embeddings instead of downloading them (costs OpenAI credits): manage.py embed_bible_verses. Full options: qdrant_storage/README.md.
Configuration
OpenAI, AI_SEARCH_ENABLED, AI_SEARCH_BASE_VERSION, QDRANT_HOST, QDRANT_PORT, QDRANT_TIMEOUT, EMBEDDINGS_DIR, EMBEDDINGS_URL, AI_TEMPLATE_ENABLED, AI_TEMPLATE_MODEL. Qdrant runs as qdrant/qdrant:v1.8.0 with REST on 127.0.0.1:7002 (host) / 6333 (compose network).
The comment in EMAW_BACKEND/urls.py saying AI search is "Postgres + pgvector" is stale; it is Qdrant.
Cost and safety notes
- Every search and verse-finder call is an OpenAI request; throttles keep this bounded per user.
- The verse finder runs the user's question through moderation first.
- Recitation analysis uploads user audio to OpenAI for transcription; the mobile app asks for microphone permission for this. Audio files are kept in blob storage.