You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Setting large min_duration values (say, 100us) with log_torch=True triggers a "Torch timer calibration failed" error:
Python 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] on linux
Type "help", "copyright", "credits" or "license"for more information.
>>> from viztracer import VizTracer
>>> t = VizTracer(min_duration=100, log_torch=True)
STAGE:2024-12-01 10:41:10 307114:307114 ActivityProfilerController.cpp:314] Completed Stage: Warm Up
STAGE:2024-12-01 10:41:10 307114:307114 ActivityProfilerController.cpp:320] Completed Stage: Collection
STAGE:2024-12-01 10:41:10 307114:307114 ActivityProfilerController.cpp:324] Completed Stage: Post Processing
Traceback (most recent call last):
File "<stdin>", line 1, in<module>
File "/home/python_venv/vllm/lib/python3.10/site-packages/viztracer/viztracer.py", line 96, in __init__
self.calibrate_torch_timer()
File "/home/python_venv/vllm/lib/python3.10/site-packages/viztracer/viztracer.py", line 227, in calibrate_torch_timer
raise RuntimeError("Torch timer calibration failed") # pragma: no cover
RuntimeError: Torch timer calibration failed
I'm quite sure this is because the calibration mechanism uses lightweight torch operations, which are however filtered with non-default min_duration setting:
defcalibrate_torch_timer(self, force=False):
ifself.enable:
raiseRuntimeError("You can't calibrate torch timer while tracer is running")
ifself.torch_offsetandnotforce:
returnimportjsonimporttempfileimporttorch# type: ignorefromtorch.profilerimportprofile, supported_activities# type: ignoreverbose=self.verbose# Silent the tracer during calibrationself.verbose=0withprofile(activities=supported_activities()) asprof:
_VizTracer.start(self)
for_inrange(20):
torch.empty(100) # <================ This is filtered_VizTracer.stop(self)
A simple fix would be to temporarily set min_duration to zero during calibration.
The text was updated successfully, but these errors were encountered:
Setting large
min_duration
values (say, 100us) withlog_torch=True
triggers a "Torch timer calibration failed" error:I'm quite sure this is because the calibration mechanism uses lightweight torch operations, which are however filtered with non-default
min_duration
setting:A simple fix would be to temporarily set
min_duration
to zero during calibration.The text was updated successfully, but these errors were encountered: