Merge pull request #3 from fdocr/crystal

Port to Crystal

.editorconfig

@@ -0,0 +1,9 @@
+root = true
+
+[*.cr]
+charset = utf-8
+end_of_line = lf
+insert_final_newline = true
+indent_style = space
+indent_size = 2
+trim_trailing_whitespace = true

.env.test

@@ -0,0 +1,6 @@
+AASA_APP_IDS="ABCDE12345.com.example.app ABCDE12345.com.example.app2"
+# REDIS_URL="redis://localhost:6379"
+# SAFELIST="fdo.cr github.com"
+# DISABLE_DEFENSE="true"
+# THROTTLE_LIMIT="1000"
+# THROTTLE_PERIOD="3600"

.github/FUNDING.yml

@@ -0,0 +1 @@
+github: [fdocr]

.github/workflows/ci.yml

@@ -1,23 +1,40 @@
 name: CI
+
 on:
   push:
     branches:
       - main
   pull_request:
     branches:
-      - main
+      - "*"
+
 jobs:
-  test:
-    name: Test Suite
+  tests:
+    name: Tests
     runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        safelist:
+          - "fdo.cr github.com"
+          - ""
     steps:
-      - name: Checkout
-        uses: actions/checkout@master
-      - name: Set up Ruby
-        uses: ruby/setup-ruby@v1
+      - name: Download source
+        uses: actions/checkout@v3
+
+      - name: Install Crystal
+        uses: crystal-lang/install-crystal@v1
+
+      - name: Cache shards
+        uses: actions/cache@v3
         with:
-          ruby-version: 3.1.2
-      - name: Install dependencies
-        run: bundle install
+          path: lib
+          key: ${{ runner.os }}-shards-${{ hashFiles('**/shard.lock') }}
+          restore-keys: ${{ runner.os }}-shards-
+
+      - name: Install shards
+        run: shards update
+
       - name: Run tests
-        run: bundle exec rspec
+        run: KEMAL_ENV=test crystal spec --verbose
+        env:
+          SAFELIST: ${{ matrix.safelist }}

.gitignore

@@ -1,3 +1,9 @@
 .env
 .DS_Store
-.byebug_history
+
+# Crystal ignores
+/docs/
+/lib/
+/bin/
+/.shards/
+*.dwarf

CONTRIBUTING.md

@@ -10,14 +10,15 @@ If you implemented a bugfix, a new feature, or updated the docs/tests feel free
 
 1. Fork the repository
 1. Install the dependencies & run locally
-   - `bundle install`
-   - `bundle exec puma -p 4567`
+   - `shards install`
+   - `crystal run src/server.cr`
 1. Create your feature branch
    - `git checkout -b my-new-feature`
 1. Work on your fix/feature
    - Add tests to avoid regressions in the future
 1. Run the tests
-   - `bundle exec rspec`
+   - `KEMAL_ENV=test crystal spec`
+   - `SAFELIST="fdo.cr github.com" KEMAL_ENV=test crystal spec`
 1. Commit your changes
    - `git commit -am 'Added some feature'`
 1. Push to the branch

Dockerfile

@@ -0,0 +1,15 @@
+# Build image
+FROM crystallang/crystal:1.10.1-alpine as builder
+WORKDIR /opt
+# Cache dependencies
+COPY ./shard.yml ./shard.lock /opt/
+RUN shards install -v
+# Build a binary
+COPY . /opt/
+RUN crystal build --static --release ./src/server.cr
+# ===============
+# Result image with one layer
+FROM alpine:latest
+WORKDIR /
+COPY --from=builder /opt/server .
+ENTRYPOINT ["./server"]

README.md

@@ -1,65 +1,87 @@
 # Universal Deep Link (UDL) Server
 
+[![Build Status](https://github.com/fdocr/udl-server/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/fdocr/udl-server/actions/workflows/ci.yml/badge.svg?branch=main)
+
 This is a server that bounces traffic to better leverage Deep Linking in mobile apps.
 
-The project's objectives are to be a simple, effective and lightweight tool that can help any website provide a seamless integration with their mobile apps.
+The project's objectives are to be a simple, effective and lightweight tool that can help any website provide a seamless integration with associated mobile apps.
 
-I used to host a server for public use (free of charge) but [due to _"reasons"_](https://github.com/fdocr/udl-server/issues/19#issuecomment-1536587313) it's not available anymore. You can [self host the project](#self-hosting) on most PaaS hosting providers quite easily. [Create an issue](https://github.com/fdocr/udl-server/issues/new) if you need help or have questions.
+## How it works
 
-## How it works, and why?
+Modern mobile browsers provide developers with [Universal Links (iOS)](https://developer.apple.com/library/archive/documentation/General/Conceptual/AppSearch/UniversalLinks.html) or [Android Intents](https://developer.chrome.com/docs/multidevice/android/intents/) to support deep linking users from a website directly into a mobile app.
 
-It's a dead simple pivot server that will allow for Universal Links to work with you app.
+However, **do you share links to your website on social media or other 3rd party apps?** Your website will likely be browsed on a webview (embedded browser) inside a 3rd party app, i.e. Instagram, TikTok, Reddit, etc. This means iOS Universal Links won't open your app automatically because users are clicking links on the same domain.
 
-Modern mobile browsers provide developers with [Universal Links (iOS)](https://developer.apple.com/library/archive/documentation/General/Conceptual/AppSearch/UniversalLinks.html) or [Android Intents](https://developer.chrome.com/docs/multidevice/android/intents/) to support deep linking users from a website directly into a mobile app. However, Operating Systems currently won't trigger these features when the user clicks a link within the same domain or when the user types the URL directly in the address bar.
+> When a user is browsing your website in Safari and they tap a universal link to a URL in the same domain as the current webpage, iOS respects the user’s most likely intent and opens the link in Safari. If the user taps a universal link to a URL in a different domain, iOS opens the link in your app. ([reference docs](https://developer.apple.com/library/archive/documentation/General/Conceptual/AppSearch/UniversalLinks.html))
 
-These and other edge cases make for a less than ideal experience, if your objective is to allow for a seamlessly transition to your mobile app. A custom banner (in your website) that links to an external site will trigger the deep linking though, and this is where the UDL Server comes in.
+This is how the UDL Server helps make this a smooth experience for you:
 
-![diagram](udl-server-diagram.png)
+![diagram](udl-server-deep-link.png)
 
-## Self-hosting
+It's a dead simple pivot server that will allow for Universal Links to trigger wherever your users are browsing from.
 
-Power users will likely need better reliability and scalability than a free service is able to offer. Self-hosting with Heroku (or similar hosting solutions) is as easy as:
+## Usage
 
-1. Fork this repository
-1. Configure the app to automatically deploy to your Heroku account
-   - Using a [custom domain with Heroku](https://devcenter.heroku.com/articles/custom-domains) is very simple (i.e. `udl.your-domain.com`)
-   - Heroku's default subdomain works too (i.e. `my-app.herokuapp.com`)
-1. Keep up with upstream (this repo) for future updates
-   - Use the **"Sync fork"** feature in your GitHub repo
-   - Or manually with git commands:
-      - `git remote add upstream [email protected]:fdocr/udl-server.git`
-      - `git pull upstream main`
-      - `git push origin main`
-1. Configure `AASA_APP_ID` ENV variable to match your App Id
-   - Use the team ID or app ID prefix, followed by the bundle ID (joined by a dot `.`).
-   - This will allow your UDL Server to directly serve as a Universal Link target for your app and improve the experience
-   - Example: `R9SWHSQNV8.com.forem.app`
+Following the **"reservation"** example app from the diagram above (assuming you support Universal Links already) all you need is to add a button (link) on your website that reads "open in app" requesting a redirect to your target location:
 
-## Throttling, Safelist and Blocklist
+```
+https://udl.fdo.cr/?r=https://reservation.com/restaurants/silvestre
+```
 
-The UDL Server uses [Rack::Attack](https://github.com/rack/rack-attack) to protect itself against abuse. It will respond with a `429` instead of the expected redirect when this happens.
+This will bypass the limitation of Safari (embedded webview) that doesn't allow your Universal Links to trigger. You should now have a working "open in app" UX.
 
-[IP based throttling](https://github.com/rack/rack-attack#throttling) is enabled by default with a limit of 3 requests on a 10 second period, but only if you provide access to a Redis to work as cache (via `REDIS_URL` ENV variable). You can override these values by using `UDL_THROTTLE_LIMIT` and `UDL_THROTTLE_PERIOD`.
+`https://udl.fdo.cr` is a **public (free to use) UDL Server** instance for anyone to try out and use on your own. It has usage limits (throttling), which should be more than enough for most low-medium traffic websites.
 
-You can further restrict if the server will allow or deny a redirect based on passing in a regular expression via `UDL_SAFELIST_REGEXP` or `UDL_BLOCKLIST_REGEXP`. These regular expressions will be checked against the `r` param and will allow or deny the response (respectively). For example:
+If this service adds value to you or your company please consider sponsoring me right here on GitHub. I offer different sponsor tiers too where I will host a private instance without usage limits for ensured reliability. [Read more about this on my profile](https://github.com/sponsors/fdocr) to support the OSS work I do on my free time.
 
-```bash
-# All redirect requests for "tiktok.com" will be safelisted
-# https://github.com/rack/rack-attack#safelisting
-UDL_SAFELIST_REGEXP="^https:\/\/tiktok.com"
-```
+## Self host
+
+The project makes it easy for you to self host a UDL Server. The easiest way to do this is to:
 
-[Read more](https://github.com/rack/rack-attack#how-it-works) about how `Rack::Attack` safelist/blacklist features work.
+1. Fork this repository
+1. Configure a PaaS to automatically deploy from your fork repository
+1. Configure your custom domain/sub-domain
+1. Keep up with upstream (this repo) for future updates
+   - Use the **"Sync fork"** feature in your GitHub repo
+   - Or manually using git
+1. Customize your deployment using ENV variables
+   - `THROTTLE_LIMIT`
+      - Number of requests allowed per `THROTTLE_PERIOD`
+      - i.e. `30` (default is `5`)
+   - `THROTTLE_PERIOD`
+      - Period to track requests to be throttled in seconds
+      - i.e. `60` (default is `30`)
+   - `SAFELIST`
+      - Space separated list of domains for private instance usage
+      - i.e. `"fdo.cr github.com"` (if set it will disable throttling)
+   - `DISABLE_DEFENSE`
+      - Disable throttle/safelist feature
+      - i.e. `"true"`
+   - `AASA_APP_IDS`
+      - Enable activity continuation and other [associated domain features](https://developer.apple.com/documentation/xcode/supporting-associated-domains)
+      - i.e. `ABCDE12345.com.example.app`
+
+Alternatively you can run the lightweight [`fdocr/udl-server`](https://hub.docker.com/repository/docker/fdocr/udl-server/general) Docker container on your own. At the time of this writing the docker image is only about `26.3 MB` in size (`10.87 MB` compressed).
 
 ## Troubleshooting
 
 Some common details to keep in mind in case your redirects aren't working properly:
 
 - Make sure your redirects are all using `https`
-- You will likely need to make this request on a `target="_blank"` anchor tag in order to get Apple's Universal Links to work.
+- If links aren't working try to use `target="_blank"` on your anchor tag
 - Make sure your iOS app has properly configured [Associated Domains](https://developer.apple.com/documentation/safariservices/supporting_associated_domains) for the websites you want to support.
    - There's a chance it won't work in development mode (i.e. only signed with a Production certificate). I suggest releasing to TestFlight in order to properly test everything.
 
+## Performance & Roadmap
+
+The project was originally written using [Ruby](https://www.ruby-lang.org/en/) & [Sinatra](https://sinatrarb.com/). It was a joy to write and worked perfectly fine, but I always wanted this "pivot request" to be as instantaneous as possible. It was doing `~40 ms` response times for `P99` on less than 1 rpm but supporting consistent daily traffic.
+
+The project is now ported to [Crystal](https://crystal-lang.org/) and [Kemal framework](https://kemalcr.com/). I'm now seeing response times drop to microseconds, which is very exciting!
+
+![diagram](nanosecond-response-times.png)
+
+I'm aiming to work on adding a bunch of other features to the project and [share blog posts](https://fdo.cr/blog) with walkthroughs/benchmarks/etc. Feel free to tag along and submit feature requests in the issue tracker.
+
 ## Contributing
 
 Please check out the [Contributing Guide](https://github.com/fdocr/udl-server/blob/main/CONTRIBUTING.md).

config/defense.cr

@@ -0,0 +1,34 @@
+require "defense"
+
+# Use memory store unless REDIS_URL is available in ENV (Redis is default)
+Defense.store = Defense::MemoryStore.new if ENV["REDIS_URL"]?.nil?
+
+# When safelist domains are provided (blank isn't allowed to safelist)
+safelist = ENV.fetch("SAFELIST", "")
+if safelist.presence
+  domains = safelist.split(" ").map { |d| "(#{d})" }.join("|")
+  regex_str = "^https?://#{domains}"
+  safelist_regex = Regex.new(regex_str, Regex::CompileOptions::IGNORE_CASE)
+
+  Defense.throttle("req/ip", limit: 0, period: 3) do |request|
+    # Only throttle redirect path with a target (allow empty param for splash)
+    next unless request.path == "/" && request.query_params["r"]?.presence
+
+    # If there's a safelist match for the target don't throttle at all
+    next unless (safelist_regex =~ request.query_params["r"]?).nil?
+
+    # Not a match means we block that request - Use a single identifier
+    # for all requests to avoid DoS by bloating our Defense cache store
+    "block"
+  end
+else
+  limit = ENV.fetch("THROTTLE_LIMIT", "5").to_i
+  period = ENV.fetch("THROTTLE_PERIOD", "30").to_i
+
+  Defense.throttle("req/ip", limit: limit, period: period) do |request|
+    # Only throttle redirect path
+    next unless request.path == "/"
+
+    request.query_params["r"]?
+  end
+end