Markdown AdapterStep for chunking by heading

[olirice]

Context

Markdown is a common format for documents ingested into vector systems and has more exploitable structure than simple text.

This task is to create an vecs.adapters.base.AdapterStep that handles chunking markdown by heading.

Ideally it would also accept parameters for

• The maximum number of words in each chunk

e.g.

from vecs.adapters import MarkdownChunker

MarkdownChunker(
  max_tokens=512
)

[jbritain]

Will give this a go if wanted

[jbritain]

According to the Markdown Guide

Markdown applications don’t agree on how to handle a missing space between the number signs (#) and the heading name. For compatibility, always put a space between the number signs and the heading name.

Should we count headings without this space as valid?

###like this

[olirice]

My vote would be to require the space

you could reference https://python.langchain.com/docs/modules/data_connection/document_transformers/text_splitters/markdown_header_metadata for ideas