A RAG ingestion pipeline on clean Markdown

Convert mixed real-world documents to Markdown first, then chunk by headings — the simplest reliable RAG ingestion pattern, shown in Python.

RAG quality is set at ingestion: garbage chunks in, garbage answers out. Converting every source document to Markdown FIRST gives you one clean, heading-structured format to chunk — regardless of whether the source was a PDF, a Word file or scraped HTML.

Pipeline sketch

import pathlib

import httpx

TYPES = {
    ".pdf": "application/pdf",
    ".docx": "application/vnd.openxmlformats-officedocument.wordprocessingml.document",
    ".html": "text/html",
    ".txt": "text/plain",
}

def to_markdown(path: pathlib.Path) -> str:
    resp = httpx.post(
        "https://mdkit.online/v1/convert",
        files={"file": (path.name, path.read_bytes(), TYPES[path.suffix])},
        headers={"X-API-Key": "YOUR_KEY"},
        timeout=120,
    )
    resp.raise_for_status()
    return resp.json()["markdown"]

for doc in pathlib.Path("corpus").iterdir():
    markdown = to_markdown(doc)
    chunks = markdown.split("\n## ")  # chunk on second-level headings
    # …embed + index the chunks…

Scaling up

For a large corpus, switch the loop to the async lane: submit every document up front with a webhook_url, and index each one as its callback arrives — no polling loop, and files up to 50 MB.

Try it

Try it free.