Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Loss graph lost when training continues on colab #8840

Open
vanquang203546 opened this issue Sep 6, 2023 · 4 comments
Open

Loss graph lost when training continues on colab #8840

vanquang203546 opened this issue Sep 6, 2023 · 4 comments

Comments

@vanquang203546
Copy link

vanquang203546 commented Sep 6, 2023

For example, I trained 1000 batches. Then Colab disconnected and I retrained. The problem is that the previous Loss graph is lost. It only displays Loss from batches 1000 onwards
CLI: !./darknet detector train yolo.data cfg/v4mosaic.cfg backup/v4mosaic_last.weights -dont_show -map

image

@stephanecharette
Copy link
Collaborator

This is fixed on the new darknet/yolo repo: https://github.com/hank-ai/darknet

@Dickoabc123
Copy link

Dickoabc123 commented Nov 8, 2023

I am getting the problem similar to that, where I am only getting the mAP for every 1000 iterations, surely there must be a way to specify how many iterations to show the mAP. Would be better for every Epoch. Does anyone know how to show the mAP sooner than every 1000 epochs?

Id also like to be able to see the validation loss and the training loss.

Hi @stephanecharette , do you know how I could do this ?

@stephanecharette
Copy link
Collaborator

  1. Darknet doesn't work with epochs. 2) You should use the Hank.ai repo instead. https://github.com/hank-ai/darknet#table-of-contents

@Dickoabc123
Copy link

@stephanecharette thank you for that. Does that give the validation losses ?
Since I already have my darker files set up and configured, is it easy enough to transfer across to Hank.ai ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants