move to git-lfs ?

[mickaelaccensi]

Hi all,

I've recently discovered the existence of git-lfs which is a large file system. This could replace the use of the ftp and ww3_from_ftp.sh but it requires to be tested first...

Mickael

[JessicaMeixner-NOAA]

There's been some investigation of that at EMC as well. @KateFriedman-NOAA do you know what the status of looking into git-lfs is at EMC? I wanted to say that there was potential issues with costs, but hopefully Kate can tell us more.

[KateFriedman-NOAA]

So we could use Git-LFS but only if NOAA/NWS paid for Git data-packs. The free version of Git-LFS only allows like 2GB (which is essentially nothing). Should NOAA start paying for Git data-packs that cover our minimum data storage and transfer needs then we can use Git-LFS.

[JessicaMeixner-NOAA]

We have < 2GB of data for WW3:

67M. cases
788M data_regtests
120K docs
4.0K LICENSE.md
22M manual
11M model
4.0K README.md
1.2G regtests
15M smc_docs
365M ww3_from_ftp.v7.12.6.tar.gz

However, I would still be concerned with putting in binary files into a repo, does git-lsf handle that okay? Would that effect the time of cloning (which we already get some complaints from some using WW3 about time to clone and size of repo)? If a netcdf file changed, would that make the size of the WW3 repo explode? (Not an lfs-expert, perhaps this is obvious if I knew the basics).

[KateFriedman-NOAA]

Actually, it's only 1GB total, just double checked. The 1GB includes transfers, which will go over 1GB almost immediately so we can't use Git-LFS without data-packs. Git-LFS should be able to handle binaries but we haven't been able to try it enough to say for 100%. Early on when we set up the NOAA-EMC org I tried using Git-LFS with a fork for the GSI repo (for the GSI fix files) and I went over the 1GB limit immediately and ended up locking the org Git-LFS because of it. I had to work with the GitHub helpdesk to remove the files because they were on GitHub's side (once in Git-LFS) and reset the org-level Git-LFS stuff. I was forced to delete my GSI fork completely to finally remove the files I had added. It was a mess and could have wrecked the auth GSI repo if I hadn't tried in a fork.

So yeah...right now we can't use Git-LFS for NOAA-EMC org without paying for data-packs. Please don't try to use it as a test.

[aliabdolali]

Thanks @KateFriedman-NOAA for the detailed explanation and suggestions/recommendations.

[mickaelaccensi]

thanks a lot @KateFriedman-NOAA for your feedback on it ! it's probably not such a good solution for us !
I can't "close" the discussion but I think it's clear enough that we will keep our good old ftp ;)

[KateFriedman-NOAA]

I'm hopeful that NOAA/NWS will pay for data-packs down the road and we can start using Git-LFS for our data versioning. fingers crossed