vectorizer.py → formerly embed_utils.py
Breakdown
Utility module for embedding text chunks and performing semantic search using sentence-transformers. Useful for lightweight embedding workflows and non-LangChain RAG pipelines.
1. get_model()
def get_model(model_name: str = "all-MiniLM-L6-v2") -> SentenceTransformer:
| Component |
Description |
| Purpose |
Loads and caches the model globally to avoid reloading |
| Default Model |
all-MiniLM-L6-v2 (compact, fast, general-purpose) |
| Optimization |
Uses global _model object to implement a singleton pattern |
- Ideal for reuse across multiple embedding calls without overhead.
2. embed_chunks()
def embed_chunks(chunks: List[str], model_name: str = "all-MiniLM-L6-v2") -> np.ndarray:
| Component |
Description |
| Purpose |
Converts a list of text chunks into vector embeddings |
| Input |
List[str] — text chunks |
| Output |
np.ndarray — dense vector embeddings |
| Model Used |
Cached SentenceTransformer instance |
- Useful for standalone scripts, batch embedding, or interactive querying.
3. search_similar_chunks()
def search_similar_chunks(
query: str,
chunk_embeddings: np.ndarray,
chunks: List[str],
top_k: int = 3,
return_scores: bool = False,
model_name: str = "all-MiniLM-L6-v2"
) -> Union[List[str], List[Tuple[str, float]]]:
| Component |
Description |
| Purpose |
Finds the top-k most semantically similar chunks to a query |
| Method |
Cosine similarity via numpy dot product |
| Return Value |
List of matching chunks or (chunk, score) pairs |
| Fallback |
Returns empty list if similarity fails |
- Lightweight vector search alternative if Chroma is not used.
4. save_embeddings()and load_embeddings()
def save_embeddings(path: str, chunks: List[str], embeddings: np.ndarray)
def load_embeddings(path: str) -> Tuple[List[str], np.ndarray]
| Function |
Description |
| save_embeddings() |
Saves chunk texts and embeddings as .npz |
| load_embeddings() |
Reloads previously saved chunks and embeddings |
| Format |
Uses numpy.savez and numpy.load |
| Use Case |
For avoiding recomputation in pipelines or cold starts |
Summary Table
| Function |
Role |
| get_model() |
Load and cache embedding model |
| embed_chunks() |
Convert chunks to dense vectors |
| search_similar_chunks() |
Retrieve top-k relevant chunks |
| save_embeddings() |
Save chunks and vectors to disk |
| load_embeddings() |
Reload from saved vector file |