fail to evaluate piqa

[vejaxu]

hello
I am trying to evaluate llama-2-13b-hf with v0.4.7 on dataset piqa but get error

File "/home/xwj/llm/lm-evaluation-harness/lm_eval/api/task.py", line 819, in init
self.download(self.config.dataset_kwargs)
File "/home/xwj/llm/lm-evaluation-harness/lm_eval/api/task.py", line 926, in download
self.dataset = datasets.load_dataset(
File "/usr/local/anaconda3/envs/xwj_transformers/lib/python3.12/site-packages/datasets/load.py", line 2556, in load_dataset
builder_instance = load_dataset_builder(
File "/usr/local/anaconda3/envs/xwj_transformers/lib/python3.12/site-packages/datasets/load.py", line 2265, in load_dataset_builder
builder_instance: DatasetBuilder = builder_cls(
TypeError: 'NoneType' object is not callable

and the command is
lm_eval --model hf --model_args pretrained="/home/llama-2-13b-hf" --tasks piqa --device cuda:0 --batch_size 8

can anyone help please?

[vejaxu]

I seem to solve this problem by replacing
dataset_path: piqa to dataset_path: nthngdy/piqa
in tasks/piqa.yaml

[vejaxu]

but I don't know whether it is correct

[SHUMKASHUN]

Same problem here... This problem affects on hellaswag, piqa, social_iqa...

[LanDisen]

Same problem☹️

[baberabb]

Hi! I can't reproduce this. If it's a network issue then some people have had success with #1634 (comment),
might be other mirrors as well.

[vejaxu]

thanks for your comment!
and maybe one solution is replacing the dataset_path which has data rather than .py file on hug.