You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[2024-12-20 01:25:35,226] [INFO] [real_accelerator.py:203:get_accelerator] Setting ds_accelerator to cuda (auto detect)
[WARNING] async_io requires the dev libaio .so object and headers but these were not found.
[WARNING] async_io: please install the libaio-dev package with apt
[WARNING] If libaio is already installed (perhaps from source), try setting the CFLAGS and LDFLAGS environment variables to where it can be found.
[WARNING] Please specify the CUTLASS repo directory as environment variable $CUTLASS_PATH
[WARNING] sparse_attn requires a torch version >= 1.5 and < 2.0 but detected 2.3
[WARNING] using untested triton version (2.3.1), only 1.0.0 is known to be compatible
/opt/conda/envs/cosyvoice/lib/python3.8/site-packages/torch/nn/utils/weight_norm.py:28: UserWarning: torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.
warnings.warn("torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.")
/opt/conda/envs/cosyvoice/lib/python3.8/site-packages/torch/nn/utils/weight_norm.py:28: UserWarning: torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.
warnings.warn("torch.nn.utils.weight_norm is deprecated in favor of torch.nn.utils.parametrizations.weight_norm.")
/workspace/CosyVoice/train_data/CosyVoice/cosyvoice/dataset/processor.py:24: UserWarning: torchaudio._backend.set_audio_backend has been deprecated. With dispatcher enabled, this function is no-op. You can remove the function call.
torchaudio.set_audio_backend('soundfile')
Traceback (most recent call last):
File "cosyvoice/bin/train.py", line 170, in <module>
main()
File "/opt/conda/envs/cosyvoice/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
return f(*args, **kwargs)
File "cosyvoice/bin/train.py", line 101, in main
configs = load_hyperpyyaml(f, overrides=override_dict)
File "/opt/conda/envs/cosyvoice/lib/python3.8/site-packages/hyperpyyaml/core.py", line 188, in load_hyperpyyaml
hparams = yaml.load(yaml_stream, Loader=loader)
File "/opt/conda/envs/cosyvoice/lib/python3.8/site-packages/yaml/__init__.py", line 81, in load
return loader.get_single_data()
File "/opt/conda/envs/cosyvoice/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 116, in get_single_data
return self.construct_document(node)
File "/opt/conda/envs/cosyvoice/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 120, in construct_document
data = self.construct_object(node)
File "/opt/conda/envs/cosyvoice/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 147, in construct_object
data = self.construct_non_recursive_object(node)
File "/opt/conda/envs/cosyvoice/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 188, in construct_non_recursive_object
for _dummy in generator:
File "/opt/conda/envs/cosyvoice/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 633, in construct_yaml_map
value = self.construct_mapping(node)
File "/opt/conda/envs/cosyvoice/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 429, in construct_mapping
return BaseConstructor.construct_mapping(self, node, deep=deep)
File "/opt/conda/envs/cosyvoice/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 244, in construct_mapping
value = self.construct_object(value_node, deep=deep)
File "/opt/conda/envs/cosyvoice/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 147, in construct_object
data = self.construct_non_recursive_object(node)
File "/opt/conda/envs/cosyvoice/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 183, in construct_non_recursive_object
data = constructor(self, tag_suffix, node)
File "/opt/conda/envs/cosyvoice/lib/python3.8/site-packages/hyperpyyaml/core.py", line 491, in _construct_name
raise ImportError("There is no such entity as %s" % callable_string)
ImportError: There is no such entity as torchaudio.functional.compute_kaldi_pitch
2024-12-20 01:25:38,002 DEBUG Attempting to acquire lock 140253436041872 on /root/.triton/autotune/Fp16Matmul_2d_kernel.pickle.lock
2024-12-20 01:25:38,002 DEBUG Lock 140253436041872 acquired on /root/.triton/autotune/Fp16Matmul_2d_kernel.pickle.lock
2024-12-20 01:25:38,002 DEBUG Attempting to release lock 140253436041872 on /root/.triton/autotune/Fp16Matmul_2d_kernel.pickle.lock
2024-12-20 01:25:38,002 DEBUG Lock 140253436041872 released on /root/.triton/autotune/Fp16Matmul_2d_kernel.pickle.lock
2024-12-20 01:25:38,017 DEBUG Attempting to acquire lock 140253436042448 on /root/.triton/autotune/Fp16Matmul_4d_kernel.pickle.lock
2024-12-20 01:25:38,018 DEBUG Lock 140253436042448 acquired on /root/.triton/autotune/Fp16Matmul_4d_kernel.pickle.lock
2024-12-20 01:25:38,018 DEBUG Attempting to release lock 140253436042448 on /root/.triton/autotune/Fp16Matmul_4d_kernel.pickle.lock
2024-12-20 01:25:38,018 DEBUG Lock 140253436042448 released on /root/.triton/autotune/Fp16Matmul_4d_kernel.pickle.lock
E1220 01:25:38.572536 140642428479296 torch/distributed/elastic/multiprocessing/api.py:826] failed (exitcode: 1) local_rank: 0 (pid: 1248) of binary: /opt/conda/envs/cosyvoice/bin/python3.8
E1220 01:25:38.576262 140642428479296 torch/distributed/elastic/multiprocessing/errors/error_handler.py:136] no error file defined for parent, to copy child error file (/tmp/torchelastic_h0m8d_fd/1986_62c__ac6/attempt_0/0/error.json)
Traceback (most recent call last):
File "/opt/conda/envs/cosyvoice/bin/torchrun", line 8, in <module>
sys.exit(main())
File "/opt/conda/envs/cosyvoice/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
return f(*args, **kwargs)
File "/opt/conda/envs/cosyvoice/lib/python3.8/site-packages/torch/distributed/run.py", line 879, in main
run(args)
File "/opt/conda/envs/cosyvoice/lib/python3.8/site-packages/torch/distributed/run.py", line 870, in run
elastic_launch(
File "/opt/conda/envs/cosyvoice/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 132, in __call__
return launch_agent(self._config, self._entrypoint, list(args))
File "/opt/conda/envs/cosyvoice/lib/python3.8/site-packages/torch/distributed/launcher/api.py", line 263, in launch_agent
raise ChildFailedError(
torch.distributed.elastic.multiprocessing.errors.ChildFailedError:
============================================================
cosyvoice/bin/train.py FAILED
------------------------------------------------------------
Failures:
<NO_OTHER_FAILURES>
------------------------------------------------------------
Root Cause (first observed failure):
[0]:
time : 2024-12-20_01:25:37
host : 78209458e1b2
rank : 0 (local_rank: 0)
exitcode : 1 (pid: 1248)
error_file: /tmp/torchelastic_h0m8d_fd/1986_62c__ac6/attempt_0/0/error.json
traceback : Traceback (most recent call last):
File "/opt/conda/envs/cosyvoice/lib/python3.8/site-packages/torch/distributed/elastic/multiprocessing/errors/__init__.py", line 347, in wrapper
return f(*args, **kwargs)
File "cosyvoice/bin/train.py", line 101, in main
configs = load_hyperpyyaml(f, overrides=override_dict)
File "/opt/conda/envs/cosyvoice/lib/python3.8/site-packages/hyperpyyaml/core.py", line 188, in load_hyperpyyaml
hparams = yaml.load(yaml_stream, Loader=loader)
File "/opt/conda/envs/cosyvoice/lib/python3.8/site-packages/yaml/__init__.py", line 81, in load
return loader.get_single_data()
File "/opt/conda/envs/cosyvoice/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 116, in get_single_data
return self.construct_document(node)
File "/opt/conda/envs/cosyvoice/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 120, in construct_document
data = self.construct_object(node)
File "/opt/conda/envs/cosyvoice/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 147, in construct_object
data = self.construct_non_recursive_object(node)
File "/opt/conda/envs/cosyvoice/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 188, in construct_non_recursive_object
for _dummy in generator:
File "/opt/conda/envs/cosyvoice/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 633, in construct_yaml_map
value = self.construct_mapping(node)
File "/opt/conda/envs/cosyvoice/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 429, in construct_mapping
return BaseConstructor.construct_mapping(self, node, deep=deep)
File "/opt/conda/envs/cosyvoice/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 244, in construct_mapping
value = self.construct_object(value_node, deep=deep)
File "/opt/conda/envs/cosyvoice/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 147, in construct_object
data = self.construct_non_recursive_object(node)
File "/opt/conda/envs/cosyvoice/lib/python3.8/site-packages/ruamel/yaml/constructor.py", line 183, in construct_non_recursive_object
data = constructor(self, tag_suffix, node)
File "/opt/conda/envs/cosyvoice/lib/python3.8/site-packages/hyperpyyaml/core.py", line 491, in _construct_name
raise ImportError("There is no such entity as %s" % callable_string)
ImportError: There is no such entity as torchaudio.functional.compute_kaldi_pitch
The text was updated successfully, but these errors were encountered:
windows11 使用项目Dockerfile 创建docker镜像运行训练脚本 examples\libritts\cosyvoice\run.sh报错。模型是CosyVoice2-0.5B
The text was updated successfully, but these errors were encountered: