The generated analogies can also be browsed here: http://instructgpt-analogies.herokuapp.com/
std_davinci
has analogies generated by the Davinci model for STD concepts.
saqa.txt
has Target, Source, and Explanation extracted from analogies in saqa_full.txt
. Explanations were used for automatic evaulation for precision since they genereally contain the core part of the analogies, while the full analogies were used for human evaluation for clarity. Target and source is also present in saqa_concepts.txt
.
saqa_<model>
and saqa_<model>_src
contain analogies generated by the MODEL for SAQA dataset concepts in the no_src and wsrc settings.
amt_res
and amt_src_res
contain the results of human evaluation of no_src and wsrc analogies.
non_analogies
contains text generated by no_anlgy prompts.
plm_generator.py
can be used to generate analogies. Prompt type, temperature, and model should be provided as command line arguments.