compare型数据集主观测评

[MHHL-Frank]

想用alpacaeval进行主观评测，参考了eval_subjective_alpacaeval_official.py，
有两个问题：
1、eval_subjective_alpacaeval_official.py文件里只有一个模型chatglm3-6b，还有一个gpt4-turbo作为judgemodel，这指的是chatglm3-6b和got4-turbo进行比较吗，如果不是的话怎么写入两个模型，并用gpt-turbo进行judge；

2、gpt4_judge = dict(
abbr='GPT4-Turbo',
path='gpt-4-1106-preview',
key='', # The key will be obtained from $OPENAI_API_KEY, but you can write down your key here as well
config='weighted_alpaca_eval_gpt4_turbo'
)
这个里面config是什么意思，如果想用别的API模型如qwen-max该怎么写

[bittersweet1999]

首先这里会有三个模型，一个是你需要测试他性能的模型，也就是config里的models变量，在demo的config里是chatglm3-6b。第二个是需要和它比较的模型，在alpacaeval的官方榜单里是某个版本的gpt4，这个是固定的不能更改的，所以在OC里隐藏起来了，不管哪个模型进来都是和它进行比较，如果需要改的话需要更改另一个地方的设置，但是你如果要和官方榜单比较的话不需要管这一点。第三个就是judge model了，利用judge model对刚刚前两个模型的结果进行比较，选出一个好的来，也就是config里的judgemodels。 你的第二个问题，如果你想要替换judge model，那么注意你的结果也是不能和榜单比较的（因为judge model也替换了，官方榜单的judge model是gpt4-turbo，只能变成相当于自己内部建立一个榜单进行比较了），如果要换judge model，直接和写一个model的config是一样的，这样即可

from opencompass.models import Qwen
qwen_max_0428 = dict(
    abbr="qwen-max-0428",
    retry=10,
    type=Qwen,
    # path="qwen-max-0428",
    path="qwen-max",  # identical to qwen-max-0428, for now
    key="",
    generation_kwargs={
        "enable_search": False,
    },
    meta_template=api_meta_template,
    query_per_second=1,
    max_out_len=2048,
    max_seq_len=2048,
    batch_size=8,
)```

[MHHL-Frank]

好的，谢谢，不过qwen_max_0428=这里应该还有个中括号吧[dict(...)]

[bittersweet1999]

你写成judge_models = [qwen_max_0428]就行了

[MHHL-Frank]

我需要调用qwen的API，如下
judgemodel = [dict(
abbr="qwen-max-latest",
retry=10,
type=Qwen,

path="qwen-max-latest",

path="qwen-max-latest", # identical to qwen-max-0428, for now
key="xxx",
generation_kwargs={
"enable_search": False,
},
meta_template=api_meta_template,
query_per_second=1,
max_out_len=2048,
max_seq_len=2048,
batch_size=8,
)]
后面
eval = dict(
partitioner=dict(
type=NaivePartitioner
),
runner=dict(
type=LocalRunner,
max_num_workers=256,
task=dict(type=AlpacaEvalTask, judge_cfg=judgemodel),
)
)
judge_cfg这里是怎么写的，我改成task=dict(type=AlpacaEvalTask, judge_cfg=judgemodel)后会出现问题：
File "/home/jovyan/opencompass/opencompass/tasks/outer_eval/alpacaeval.py", line 95, in init
assert type(judge_cfg) == ConfigDict
AssertionError

[MHHL-Frank]

删掉括号后还是同样的问题

[bittersweet1999]

opencompass/configs/eval_subjective.py

Line 70 in 85a2887

judge_models = [dict(

这个关键字段应该是judge_models，你这里是judgemodel。
还有一个问题忘了跟你说了，你这边用的是qwen做judgemodel的话，走不了这个official的config，因为那个只能用GPT做judge model，alpacaeval那边是这样的，alpacaeval的lc win rate用了logit prob，其他模型不确保能输出这个。所以我推荐你如果要换judge model的话，参考https://github.com/open-compass/opencompass/blob/main/configs/eval_subjective.py 这个config即可。
也就是写成下面这样就能跑了

from mmengine.config import read_base

with read_base():
    from opencompass.configs.datasets.subjective.alpaca_eval.alpacav2_judgeby_gpt4 import alpacav2_datasets
from opencompass.models import HuggingFaceCausalLM, HuggingFace, HuggingFaceChatGLM3, OpenAI
from opencompass.partitioners import NaivePartitioner, SizePartitioner
from opencompass.partitioners.sub_naive import SubjectiveNaivePartitioner
from opencompass.partitioners.sub_size import SubjectiveSizePartitioner
from opencompass.partitioners.sub_num_worker import SubjectiveNumWorkerPartitioner
from opencompass.runners import LocalRunner
from opencompass.runners import SlurmSequentialRunner
from opencompass.tasks import OpenICLInferTask
from opencompass.tasks.subjective_eval import SubjectiveEvalTask
from opencompass.summarizers import SubjectiveSummarizer

api_meta_template = dict(
    round=[
        dict(role='HUMAN', api_role='HUMAN'),
        dict(role='BOT', api_role='BOT', generate=True),
    ]
)

# -------------Inference Stage ----------------------------------------
# For subjective evaluation, we often set do sample for models
models = [your models
]


datasets = [*alpacav2_datasets] # add datasets you want


infer = dict(
    partitioner=dict(type=NaivePartitioner),
    runner=dict(type=LocalRunner, max_num_workers=16, task=dict(type=OpenICLInferTask)),
)
# -------------Evalation Stage ----------------------------------------

## ------------- JudgeLLM Configuration
judge_models = [your qwen]

## ------------- Evaluation Configuration
eval = dict(
    partitioner=dict(type=SubjectiveNaivePartitioner, models=models, judge_models=judge_models,),
    runner=dict(type=LocalRunner, max_num_workers=16, task=dict(type=SubjectiveEvalTask)),
)

summarizer = dict(type=SubjectiveSummarizer, function='subjective')
work_dir = 'outputs/subjective/'

把models和judge_models换成你需要的即可

[MHHL-Frank]

这个也是我的模型与gpt4比较，然后qwen-max判断对吧

[bittersweet1999]

对是的

[MHHL-Frank]

谢谢！