Bleve
Fast local-disk search capabilities via Bleve
Bleve is a full-text search engine written in Go that runs within the Storyden process. It stores search indexes on local disk and provides significantly better search quality than database search.
SEARCH_PROVIDER=bleve
# A directory path for Bleve to store its index files:
BLEVE_PATH=./data/bleveWhen to use Bleve
A small to medium sized community forum running on a server with persistent disk storage. The community has tens of thousands of Threads and hundreds of Library Pages. Members frequently use search to find relevant content. The deployment uses a single Storyden instance with SQLite or PostgreSQL database.
Bleve indexes content for fast term-based search. Storyden uses a language-agnostic index analyzer that works for many languages:
- All Latin-based languages (English, Spanish, French, German, Italian, Portuguese, Dutch, etc.)
- Slavic languages (Russian, Ukrainian, Polish, Czech, Slovak, Serbian, Croatian, Bulgarian)
- Greek, Turkish, Georgian, Armenian
- Hebrew, Arabic, Persian
- Urdu, Hindi, Punjab, Nepali
- Yoruba, Igbo, Hausa, Akan, Swahili
Currently, Bleve search has limited functionality for Chinese, Japanese, Korean and other languages that do not use whitespace to separate words, such as Thai, Lao, Khmer, Burmese. We would love help from speakers of these languages to make Storyden more useful for more people! Please open an issue if you are interested in helping.
If you run into issues with search in your language, please open an issue and help make Storyden better for everyone on earth!
Upsides:
- Improved search quality and speed over database search
- Runs well on a stateful server with local disk
Considerations:
- Does not handle very high-volume of queries or indexing well as it's single-threaded
- Not suitable for ephemeral containers or scaled replica deployments
- Single-instance only - index cannot be not shared across replicas
Indexing
When performing a full index (indexing all content from scratch) the rate is rather slow at around 10,000 documents in 10 minutes. A million documents would take hours. This may be improved in future versions.
In order to trigger a full re-index of content, see reindexing.
Single-server deployment with Bleve
Small to medium community running on a VPS or dedicated server:
# Database
DATABASE_URL=sqlite:///var/lib/storyden/data.db
# Search
SEARCH_PROVIDER=bleve
BLEVE_PATH=/var/lib/storyden/bleve
SEARCH_INDEX_CHUNK_SIZE=1000
# Asset storage
ASSET_STORAGE_TYPE=local
ASSET_STORAGE_LOCAL_PATH=/var/lib/storyden/assetsThis configuration keeps all data local to the server. Ensure /var/lib/storyden has sufficient disk space and is included in your backup strategy.