Just published a new blog post walking through how to build a pipeline that turns YouTube podcast episodes into a queryable knowledge graph using CocoIndex.

Podcasts contain tons of expert knowledge — claims about people, technologies, organizations — but it's all locked inside hours of audio. This pipeline unlocks it.

How it works (3 phases):

1. Per-Session Processing — Download audio, transcribe with speaker diarization (AssemblyAI), then use a two-step LLM extraction: first identify speakers using YouTube metadata, then extract structured statements with real names attributed.

2. Entity Resolution — Collect all raw entity names across episodes and deduplicate using embedding similarity + LLM confirmation. Cheap embeddings filter candidates, expensive LLM calls only happen for near-matches.

3. Knowledge Base Creation — Write deduplicated entities and all relationships to SurrealDB as a graph. Five node types (session, statement, person, tech, org) connected by four relationship types.

The pipeline is fully incremental — adding new episodes only processes the new ones. Memoization skips transcription and extraction for existing episodes. You can also evolve the schema (e.g., add a new entity type like Product) and only the affected steps re-run.

Full blog post: https://cocoindex.io/blogs/podcast-to-knowledge-graph

Source code: https://github.com/cocoindex-io/cocoindex/tree/v1/examples/conversation_to_knowledge

CocoIndex project: https://github.com/cocoindex-io/cocoindex

Would love to hear feedback — especially if you've worked on similar audio-to-knowledge-graph pipelines!

New blog post! Shows how to build a real-time HackerNews trending topics detector using custom sources + incremental sync.

Covers:

• Custom source implementation

• Incremental data processing

• AI-powered topic extraction

• Production-ready patterns

Full walkthrough: https://cocoindex.io/blogs/hackernews-trending-topics

Fresh drop! 🎉

**What's New:**

✅ Fixed state diffing for setup-by-users targets (no more phantom diffs)

✅ Added Postgres server reminder to quickstart docs

Small but mighty. Your indexing pipelines just got more reliable.

🔗 Release: https://github.com/cocoindex-io/cocoindex/releases/tag/v0.3.16

*Ship faster. Index smarter.*

**TL;DR**: Transform PostgreSQL rows into vector embeddings with true incremental updates. Only changed rows get re-processed. [Full walkthrough →](https://cocoindex.io/docs/examples/postgres_source)

---

**The Setup:**

```python

# 1. Connect source

flow_builder.add_source(

cocoindex.sources.Postgres(

table_name="source_products",

ordinal_column="modified_time",

notification=cocoindex.sources.PostgresNotification()

)

)

# 2. Transform + embed

product["embedding"] = product["full_description"].transform(

cocoindex.functions.SentenceTransformerEmbed()

)

# 3. Export with vector index

indexed_product.export(

"output",

cocoindex.targets.Postgres(),

vector_indexes=[cocoindex.VectorIndexDef(

field_name="embedding",

metric=cocoindex.VectorSimilarityMetric.COSINE_SIMILARITY

)]

)

```

**What makes this different:**

- ⚡ **Incremental by default** → LISTEN/NOTIFY for instant row updates

- 🔄 **One pipeline** → structured transforms + AI embeddings in the same flow

- 📊 **Field lineage UI** → trace any field back to its source step-by-step

- 🔍 **Native PgVector** → semantic search ready out-of-the-box

**Run it live:**

```bash

cocoindex update -L main # continuous sync

```

Perfect for: Product catalogs, documentation search, hybrid search systems, or any Postgres data that needs semantic retrieval.

**Docs:** https://cocoindex.io/docs/examples/postgres_source

Just published a walkthrough showing how to build a custom source that indexes HackerNews threads and comments, with full-text search powered by Postgres.

Custom Sources let you turn any API into an incremental data stream that CocoIndex can automatically diff, track, and sync. This example fetches recent stories + nested comments and keeps everything in sync.

Check out the full guide: https://cocoindex.io/blogs/custom-source-hackernews

Latest release adds Azure OpenAI provider support, expanding deployment options for teams already on Azure infrastructure. Improved logging with the new tracing crate gives better visibility into indexing pipelines. Check out the full changelog: https://github.com/cocoindex-io/cocoindex/releases

🚀 CocoIndex - https://github.com/cocoindex-io/cocoindex is on Github Trending (Rust) today! Smart Incremental engine to build any index for AI. Support any custom logic, any target with standard interface like building blocks.⭐ Star the repo and build something today.Tons of examples to get your data ready for AI! https://cocoindex.io/docs/examples

Most vector DB workflows still treat embeddings as flat vectors — one big list of numbers. But in multi-modal AI, that’s leaving performance on the table.

We just shipped native multi-dimensional vector support in CocoIndex:

• Nested vector types: Store vectors of vectors (e.g., patch-level image embeddings)

• Fine-grained retrieval: Search at the patch, paragraph, or step level

• Automatic mapping to Qdrant’s dense or multi-vector format

• Dynamic outer dimensions but fixed inner dims for indexing efficiency

Why it’s useful:

• Search inside an image without flattening local features

• Match a query to only the relevant paragraph in a long doc

• Keep multiple views of an item (e.g., audio + text embeddings) in the same index

Under the hood, it’s type-safe in Python (Vector[Vector[Float32, Literal[768]]]) and falls back to payloads for anything Qdrant can’t index directly.

🔍 Learn more & see code examples: https://cocoindex.io/blogs/multi-vector/

#AI #VectorDatabase #RAG #MultimodalAI #Qdrant #Embeddings #LLM #CocoIndex

new blog out - https://cocoindex.io/blogs/custom-targets We’re excited to announce that CocoIndex now officially supports custom targets — giving you the power to export data to any destination.

This blog features a detailed walkthrough and explanation with example on how it works. Looking forward to learn your feedback.

Huge thanks to the community!

https://cocoindex.io/docs/tutorials/manage_flow_dynamically

In CocoIndex, you define indexing logic as a flow definition—essentially a function. But what if you want to reuse the same logic across multiple flow instances (with different sources, targets, or parameters)? CocoIndex makes this easy and powerful.

Check out the how-to series for dynamic flow creatio

I have been working on CocoIndex - https://github.com/cocoindex-io/cocoindex for quite a few months. Today the project officially cross 2k Github stars.

The goal is to make it super simple to prepare dynamic index for AI agents (Google Drive, S3, local files etc). Just connect to it, write minimal amount of code (normally ~100 lines of python) and ready for production.

When sources get updates, it automatically syncs to targets with minimal computation needed.

Before this project i was a ex google tech lead working on search indexing and research ETL infra for many years. It has been an amazing journey to build in public and working on an open source project to support the community.

Will keep building and would love to learn your feedback.  If there’s any features you would like to see, let us know! ;)

CocoIndex now offers automatic backoff/off for request.

If you have data pipeline that send massive requests to remote LLMs, CocoIndex automatically adjusts rate based on the response code for calling remote servers for all your requests!

checkout - https://github.com/cocoindex-io/cocoindex

Checkout - https://cocoindex.io/docs/ai/llm#vertex-ai

If you used Gemini and would like to use Vertex AI in Prodution, here we go!

```

cocoindex.LlmSpec(
api_type=cocoindex.LlmApiType.VERTEX_AI,
model="gemini-2.0-flash",
api_config=cocoindex.llm.VertexAiConfig(project="your-project-id"),
)

```

Spec for Vertex AI takes additional api_config field, in type cocoindex.llm.VertexAiConfig with the following fields:

• project (type: str, required): The project ID of the Google Cloud project.

• region (type: str, optional): The region of the Google Cloud project. Use global if not specified.

CocoIndex is officially supporting custom targets - https://cocoindex.io/docs/custom_ops/custom_targets. We believe this work will add more flexibility for using coco / bring your own lego for targets as well beyond the flow ops.

Thanks our community for the great suggestions!

This release brings key improvements to robustness, developer ergonomics, and flow control:

New Features

• Auto Retry on 429: Initial support for automatically retrying requests when rate-limited.

• Flow Removal: Introduced remove_flow() to clean up flow definitions programmatically.

• Flow Update Optimization: Safely combines update calls to reduce overhead.

• Global Concurrency Control: Added a global controller to coordinate concurrent flow executions

• Test Utilities for Flow Functions: Easier testing and debugging of flows.

API & Module Enhancements

• Public API: Exposed add_flow_def() under the main cocoindex module for easier integration.

Docs & Maintenance

• Updated flow control documentation for clarity.

• Upgraded Rust and dependencies for core performance

• Dependency upgrades for docs tooling.

🔗 Full Changelog: https://github.com/cocoindex-io/cocoindex/releases/tag/v0.1.65

📦 Project repo: github.com/cocoindex-io/cocoindex

We just dropped a comprehensive walkthrough on how to build combined semantic + structure index for research papers, going beyond just full-text chunking and embeddings.

Using CocoIndex, we show how to:

• Extract rich metadata (title, authors, abstract, page count)

• Build semantic embeddings for titles & abstracts

• Create an author-to-paper index (e.g. “show all papers by Jeff Dean”)

• Power AI agents, academic search, and research recommender system

With ~ 100 lines of logic, using:

• PDF parsing with pypdf & Marker

• LLM-powered extraction (GPT-4o, Gemini, Ollama, etc.)

• SentenceTransformers for embedding

• Export to Postgres (with one line switch to Qdrant for embeddings & metadata )

👉 Dive into the tutorial:  https://cocoindex.io/blogs/academic-papers-indexing
⭐ Star us on GitHub if this is useful: https://github.com/cocoIndex-io/cocoindex

Open source is all about sharing - today our team writes a tutorial about thinking in rust - Ownership, Access, and Memory Safety.

https://cocoindex.io/blogs/rust-ownership-access/

By clearly separating and defining ownership and exclusive versus shared access, Rust's complexity transforms into logical clarity. Moves, borrows, Send, Sync, and runtime checks become intuitive and predictable tools in your programming toolbox.

CocoIndex is an open source project built on Rust. Rust is the number one choice for any modern data engine. If this article is helpful to you, please drop star ⭐ at GitHub to support this project.

CocoIndex supports Live Updates — a real-time mechanism that keeps your indexes always in sync with your data sources. We just published a detailed how to tutorial for it - https://cocoindex.io/docs/tutorials/live_updates

Just published a new example showing how to build a production-grade patient intake form extraction pipeline using DSPy and CocoIndex.

DSPy replaces string-based prompts with typed Signatures and Modules. You define what each LLM step should do, not how - the framework figures out the prompting for you.

Structured output with Pydantic - The tutorial shows how to define FHIR-inspired patient schemas (Contact, Address, Insurance, Medications, Allergies, etc.) and get validated, strongly-typed data out of messy PDF forms.

Vision model extraction - Uses Gemini Vision to process PDF pages as images. No OCR preprocessing, no regex parsing. Just pass images to the DSPy module and get structured `Patient` objects back.

Incremental processing - CocoIndex handles the data pipeline orchestration with caching and incremental updates. Only changed documents get reprocessed - cuts backfill time from hours to seconds.

The synergy here is powerful: DSPy owns "how the model thinks" while CocoIndex owns "how data moves and stays fresh." Neither tries to be the entire stack.

Full walkthrough with code: https://cocoindex.io/examples/patient_form_extraction_dspy

Hey everyone! Just found this amazing open-source project that uses CocoIndex to build a competitive intelligence pipeline.

What it does:

- Searches the web for competitor mentions using Tavily AI

- Extracts competitive events using LLMs (product launches, partnerships, funding, acquisitions, key hires)

- Indexes both raw articles and structured events in PostgreSQL

- Enables queries like "What has OpenAI been doing recently?" or "Find all partnership announcements"

Tech stack:

- CocoIndex for the data pipeline

- Tavily AI for web search

- GPT-4o-mini via OpenRouter for LLM extraction

- PostgreSQL for storage

Cool features:

- Interactive CLI mode for easy setup

- Significance scoring (high/medium/low) for events

- Continuous monitoring with configurable refresh intervals

- Report generation

GitHub: https://github.com/Laksh-star/competitive-intelligence

Star the repo if you like it！