offline archive

[jwinarske]

I have a need to support Long Term Support (LTS); automotive and IOT.

I'm proposing a new sub command called archive.

This would cache source types hosted and git into an archive folder.

Then without Internet access you can restore said archive, and build.

I have an implementation in Python that does this today. I am using this with the Yocto build system:
https://github.com/meta-flutter/meta-flutter/blob/scarthgap/tools/pubspec.py

[sigurdm]

We don't have a separate command for this today, but you can already do:

PUB_CACHE=.pub_cache dart pub get

Which would put the cache inside the current directory.
This way you can zip everything up.

Then when extracting somewhere run

PUB_CACHE=.pub_cache dart pub get --offline

To fix the absolute references in the .dart_tool/package_config.json

[jwinarske]

The use case is building 100+ apps simultaneously with LTS support. The proposed archive method dramatically decreases build storage. As there is only one archive of a package, which may show up 100+ times. The compress pub cache approach uses much more disk space at scale. The pub cache restore step takes ~1 second when multi-threaded.

[sigurdm]

Not sure I follow the use case exactly:

You have 100 apps, and now you want to make a single archive with all of them, or one archive for each?

Can you spell out what exactly the proposed command would do?

[jwinarske]

@sigurdm
Archive would not include any redundant packages; only single references for git and hosted artifacts.

1. pub archive
   Referencing the pubspec.lock file fetch hosted tar.gz, create tar.gz.256, and create .cache folder with -advisories.json, and -version.json files. This represents the archive; all the items fetched from the pub server.

2. pub archive restore
   Referencing the pubspec.lock creates a pub cache using archive path. Restores pub cache to path specified by PUB_CACHE.

[sigurdm]

Archive would not include any redundant packages;

Not sure what you mean by this. The procedure I outlined above would only fetch the packages needed.

only single references for git and hosted artifacts.

What are "references" in this context?

[jwinarske]

Not sure what you mean by this. The procedure I outlined above would only fetch the packages needed.

Yes this is true for a single application. If I look at all the artifacts fetched for 100 apps, there are many common packages shared between them. This means the archive folder would only contain one copy of each package version. Using a compressed pub archive duplicates the same package version many times; thus increasing disk space.

What are "references" in this context?

These would be considered a reference:

• hosted/pub.dev/package-x.x.x.tar.gz
  hosted/pub.dev/.cache/package-versions.json
  hosted/pub.dev/.cache/package-advisories.json
• git mirror
  git/cache/repo-sha1

[jwinarske]

Archive listing for flutter-gallery
archive_example.txt

File sizes

Flutter Gallery

• Archive folder - 291M
• tar.gz of PUB_CACHE - 304M
• uncompressed PUB_CACHE - 700M