Feature: SQL Template Matcher

[anisbhsl]

This is a feature PR for SQL based template matcher. Two downstream PRs:

1. Fuzzy SQL Matcher (Tracked by Fuzzy SQL Template Matcher #3 )
2. LLM based SQL Matcher (Tracked by LLM Based SQL Template Matcher #4)

will be branched off this PR.

[NISH1001]

Reiterating: Let's break the PR (draft) into 2:

• first with fuzzy matching SQL template (initial work by @connectwithprakash )
• Second with LLM-based matcher (cc: @anisbhsl supported by @NISH1001 )

[NISH1001]

Possible use case:

• Making the sql agent better by having the template matcher as primary tool?
• Grounding the query-to-sql to reduce possible hallucinations in the SQL.

For LLM-based matcher, we can provide in-context (all-shot prompting?) list of templates (query+sql) and let the LLM give us scored output which we can use.

Usage

Possible usage could be

from larch import SQLTemplate
from larch.search import FuzzySQLTemplateMatcher

templates  = [
    SQLTemplate(
        query="What is the capital of Nepal?",
        sql="select * from country c where c.country=="Nepal";
]

matcher = FuzzySQLTemplateMatcher(templates=templates, ...)

query = "Capital of Nepal"
matched_templates = matcher(query=query, ...)

[NISH1001]

Let's return List[SQLTemplate] instead of List[str].

[anisbhsl]

While the fuzzy matcher might be able to provide response as List[SQLTemplate], it may not be that efficient to do with LLM based matcher. If the number of templates is huge, getting the list of sql templates with pattern and entity substituted sql query might require prompting LLM to provide a list of matching queries. I've not experimented on LLM part so I can't fully support the statement above.

I'll put more context once I get to know how it performs.

[NISH1001]

I mean the return list should still be a subset of all the templates, cut-off by threshold or top_k. So, the correct return type should which SQLTemplate objects are returned. Hence List[SQLTemplate] makes more sense as it gives us idea about what sort of query and intents are also being matched for input query.

[NISH1001]

Re: With llm-based, even if the llm just gives us the SQL query, we can ideally reverse map the original sqltemplate object as well. I think the result that LLM returns could infact be enforced by in-context prompting with SQL templates. Nevertheless, let's just stick with List[SQLTemplate] as return type because we're technically just selecting/matching the input templates.

[NISH1001]

also let's add intent: Optional[str] = None field as well in case we want the intent detection somewhere in future.

[anisbhsl]

Reiterating: Let's break the PR (draft) into 2:

• first with fuzzy matching SQL template (initial work by @connectwithprakash )
• Second with LLM-based matcher (cc: @anisbhsl supported by @NISH1001 )

Yeah the plan is to do the same as explained in the description above.

[NISH1001]

Make this Optional[BaseLanguageModel] and inside the constructor we can do llm = llm or ChatOpenAI(...)

[NISH1001]

@anisbhsl can you also run the code through pre-commit hooks please.
Here's how to do it:

pip install pre-commit

pre-commit install #installs the config from .pre-commit-config.yaml

Once done, you'll be able to run the hooks automatically every time you invoke git commit <...> command.

[NISH1001]

Another thought I have:

By making use of description, intent, query for SQLTemplate we can also possibly do RAG on top of templates to find the best matching candidate, and do the entity substitution via llm for matched query.

[NISH1001]

Closing this for time being as apparently it's already in develop for some rason.