Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make Domains to Cache configurable #127

Merged
merged 1 commit into from
Dec 26, 2023
Merged

Make Domains to Cache configurable #127

merged 1 commit into from
Dec 26, 2023

Conversation

tempusfrangit
Copy link
Contributor

Make domains to cache configurable via ENV. The implements the framework to also limit caching to certain prefixes to the path.

@tempusfrangit
Copy link
Contributor Author

What is still required is some testing and I'd like to revisit the way we talk to the CH hosts.

See: https://github.com/replicate/cluster/pull/1284 <-- This needs work to use current L2 cache, It is an example of how it could work straight to origin.

This allows us to send the URI as:
/r8//path which means we only need to update the map or in some cases the rewrite rule to enable more domains (and it could cache any domain not just GCS with ease).

@tempusfrangit
Copy link
Contributor Author

This PR is missing prefix matching tests. I would also like to consider not setting a default in GetDomtainsToCache. Also GetDomainsToCache is a bad function name, I'd like it to be CacheDomains to match the option.

@tempusfrangit
Copy link
Contributor Author

tempusfrangit commented Dec 26, 2023 •
edited
Loading

Tests still need to be fixed. Additionally work to warm the cache using the new scheme will be needed prior to releasing this

@tempusfrangit tempusfrangit force-pushed the configurable-domains-to-cache branch 3 times, most recently from c8c62c7 to 1849f48 Compare December 26, 2023 21:32
@tempusfrangit tempusfrangit marked this pull request as ready for review December 26, 2023 21:32
@tempusfrangit
Implement consistent-hash domains configurable
f550200 
Implement making consistent-hash domains to cache configurable.
** THIS IS A BREAKING CHANGE **

Consistent-Hash now utilizes a path-based proxy system, prepending
the URI to the path prior to sending to the consistent-hash index
target. This URL manipulation does not change the hash value. Buckets
are maintained.

The prefixes allowed to be cached are provides exclusively as an ENV
var and is in the format of "scheme://host/prefix, scheme://host/prefix, ..."

Eventually these "must only be set via ENV" should move to a config
file.

The consistent hash services already understand the new path-based
proxy.

Include new option to trigger path-based proxy vs pure hostname
proxy.
@tempusfrangit tempusfrangit merged commit 3cecf9f into main Dec 26, 2023
5 checks passed
@tempusfrangit tempusfrangit deleted the configurable-domains-to-cache branch December 26, 2023 23:14
philandstuff added a commit that referenced this pull request Jan 19, 2024
@philandstuff
Restore default cacheable URL prefix
f322616 
This restores to the status quo before #127.
@philandstuff philandstuff mentioned this pull request Jan 19, 2024
Merged
tempusfrangit pushed a commit that referenced this pull request Jan 19, 2024
@philandstuff @tempusfrangit
Restore default cacheable URL prefix
ef7f831 
This restores to the status quo before #127.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@philandstuff philandstuff Awaiting requested review from philandstuff

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
1 participant
@tempusfrangit