KG-RAG now provides provenance for the generated biomedical text!

[karthiksoman]

This release has two main additions:

1. In this release, KG-RAG extracts node context by making API calls to SPOKE-KG.

2. In this release, KG-RAG provides provenance information associated with the generated biomedical text. The provenance comes directly from the underlying SPOKE KG. Hence, with this addition, biomedical text generated using KG-RAG is not only grounded in established knowledge but also offers insights into the source or provenance of that knowledge.

---

This discussion was created from the release KG-RAG now provides provenance for the generated biomedical text!.

[karthiksoman]

[namin]

Hi Karthik,

Cool that you've released this! I tried the example with Llama13b, and it seems to provide a spurious DOID. See below.

Besides that, it does seem like a good idea to provide more provenance / evidence that just the ontology name, but I guess SPOKE doesn't carry that detailed information?

Thanks, ~n

$ python -m kg_rag.rag_based_generation.Llama.text_generation interactive
Enter your question : Is hypochondrogenesis a monogenic disease?
Press enter for Step 1 - Disease entity extraction using GPT-3.5-Turbo
Processing ...
Calling OpenAI...
Extracted entity from the prompt = 'hypochondrogenesis'

Press enter for Step 2 - Match extracted Disease entity to SPOKE nodes
Finding vector similarity ...
Matched entities from SPOKE = 'hypochondrogenesis'

Press enter for Step 3 - Context extraction from SPOKE
Extracted Context is :
Disease hypochondrogenesis isa Disease osteochondrodysplasia and Provenance of this association is from Disease Ontology. Disease hypochondrogenesis isa Disease monogenic disease and Provenance of this association is from Disease Ontology. Disease hypochondrogenesis associates Gene COL2A1 and Provenance of this association is from DISEASES.

Press enter for Step 4 - Context pruning
Pruned Context is :
Disease hypochondrogenesis isa Disease monogenic disease and Provenance of this association is from Disease Ontology.

Press enter for Step 5 - LLM prompting
Prompting llama
Loading checkpoint shards: 100%|██████████████████| 3/3 [00:04<00:00, 1.62s/it]
[INST]<>
You are an expert biomedical researcher. For answering the Question at the end, you need to first read the Context provided. Then give your final answer with the Provenance information by considering the context. Refer the Example 1 and Example 2 to know how to cite the Provenance.
Example 1:
The genes associated with DiseaseX are Gene1, Gene2 [Provenance: ].
Example 2:
The genes associated with DiseaseX are Gene1, Gene2 [Provenance: ]. The disease localizes in the organ1, organ2, organ3 [Provenance : ].

<>

Context:

Disease hypochondrogenesis isa Disease monogenic disease and Provenance of this association is from Disease Ontology.

Question: Is hypochondrogenesis a monogenic disease?[/INST] Yes, hypochondrogenesis is a monogenic disease. The Provenance of this association can be cited as follows:

[Provenance: Disease Ontology (DO) (2019) Hypochondrogenesis, DOID:5488.]

According to the Disease Ontology (DO), hypochondrogenesis is a monogenic disease caused by mutations in the HCH gene (DOID:5488). This information can be cited using the DOID (Disease Ontology Identifiers) provided by the Disease Ontology consortium.

It is important to note that the Provenance of the information should be provided when sharing or citing the information to ensure the accuracy and credibility of the source.

[karthiksoman]

@namin Hi Nada,

Thank you for the feedback. I have now changed the system prompt and the provenance extraction has improved from what you showed. Please see below:

python -m kg_rag.rag_based_generation.Llama.text_generation
Enter your question : Is hypochondrogenesis a monogenic disease?
Response (Truncated): Yes, hypochondrogenesis is a monogenic disease, as stated in the Provenance of this association,which is Disease Ontology. (Source: Disease Ontology)

Please let me know how it goes with other prompts that you try using KG-RAG.
I totally agree that more granularity to the provenance would be ideal, however, SPOKE has captured the provenance in this fashion. I was thinking of including more evidence attribute to the generated text (like p-value or confidence value from SPOKE) to enrich the generated text. I am excited to hear your thoughts/suggestions on this.

Regards
Karthik Soman