Prompt expansion feature

[rchan26]

Add functionality to obtain variants of prompts in order to better explore the input space.

Can potentially be linked to #82 for "chaining" prompto runs but for this idea rather than creating a new experiment after an initial run, we are maybe doing an initial "expansion" run to obtain a larger number of prompts to send.

[fedenanni]

Starting point, reading this recent NAACL student workshop paper: https://aclanthology.org/2024.naacl-srw.2.pdf

[fedenanni]

Also: https://arxiv.org/pdf/2402.07927

[rchan26]

To start, an early version of a rephrasals pipeline could look something like the one that creates and runs a LLM-as-a-judge evaluation:

• Rephrasal templates are provided (potentially as a .json file of an array of strings?) - unlike the judge, we probably have several rephrasal templates?
• Rephrasal "settings" are provided (i.e. the API and model to send those rephrasals to)

To implement:

• A specific command (prompto_run_rephrasal?) which produces an rephrased input file - it will send the prompts to the same API/model provided in the original input file

We can try this out for a bit before implementing this into prompto_run_experiment

[fedenanni]

Are we expecting that the rephrasal script would generate a new file where for each line you have the rephrased prompt or that you will have the original prompt and in the following line the rephrased one? I think the second one would be more useful for our work, but maybe for general applications the first one is better?

[rchan26]

Is the only difference whether or not the original prompt is included in the new file? We can make that a flag in the command whether or not its duplicated to the new file, with the default option being that it is there