fix(github): download of files larger than 1MB #57
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
line-o
force-pushed
the
fix/github-file-download
branch
from
1683a32
to
399a8ca
Compare
line-o
force-pushed
the
fix/github-file-download
branch
from
399a8ca
to
321a63b
Compare
When a single commit changes more than 300 changed files than the request is paged. The github module will now handle these.
That would have left the old file in the connected collection dangling. Now every renamed file will be deleted. As another edge-case where a file is _added_ and then renamed would try to remove an non-existing file, `github:incremental-delete` will now treat a not found resource as a successful deletion.
Ignored files are expath-pkg.xml, repo.xml and all .xconf files. Files that would be ignored was already reported by the dry-run but not for the actual run. This allows for clients to act on that information and can, for instance, report that a full installation is necessary or that the index configuration has changed.
When determining the changeset for incremental uploads but the local commit hash cannot be found on the remote the error github:commit-not-found is raised.
- added case is now handled separately as the old entry must be filtered from the del sequence - get rid of github:remove-or-ignore by always filtering lists This should improve readability.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
incremental update reports ignored files
Ignored files are expath-pkg.xml, repo.xml and all .xconf files.
Files that would be ignored was already reported by the dry-run but not for the actual run.
This allows for clients to act on that information and can, for instance, report that a full installation is necessary or that the index configuration has changed.
handle edge-case when a file is modified and then renamed
That would have left the old file in the connected collection dangling.
Now every renamed file will be deleted.
As another edge-case where a file is added and then renamed would try to remove an non-existing file,
github:incremental-deleteincremental updates handles pages
When a single commit changes more than 300 changed files than the request is paged. The github module will now handle these.
download of files larger than 1MB
Files larger than 1 megabyte are not returned in the query for metadata directly. That means we cannot rely on this query alone.
If the content is empty we use the 'download_url' property instead.
For GitHub.com we can construct the file download URL to point to raw.githubusercontent.com and do not even need to query for metadata of a changed file first.