Automatic certificates from Let's Encrypt

.vscode/settings.json

@@ -14,6 +14,8 @@
     "cSpell.words": [
         "Atof",
         "autodetected",
+        "caddyserver",
+        "certmagic",
         "certpath",
         "cfhd",
         "chunkdur",

CHANGELOG.md

@@ -12,6 +12,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - new URL parameter `timesubswvtt` provides generated timing wvtt subtitles
 - `timesubsstpp` and `timesubswvtt` now work with SegmentTimeline
 - `continous_1` URL parameter to signal multiperiod continuity
+- Automatic Let's Encrypt certificates for HTTPS for one or more domains via `domains` parameter
 
 ## [0.6.0] - 2023-06-10
 

README.md

@@ -44,40 +44,10 @@ all parameters of [livesim1][1] are implemented, but there are also new
 parameters like the generated subtitles mentioned above.
 The [URL wiki page][urlparams] lists what is available.
 
-## Components
-
 There are two main components, the server `livesim2` and the VoD fetcher
 `dashfetcher`.
 
-### dashfetcher tool
-
-The tool `dashfetcher` fetches DASH VoD assets via HTTP given an MPD URLs.
-Currently it supports MPDs with SegmentTimeline with `$Time$` and
-SegmentTemplate with `$Number$`. The assets must have no explicit `<BaseURL>` elements to
-work properly. With the `-a/--auto` option, the path to the asset is preserved
-as much as possible and adapted to the local path.
-
-Files already downloaded will not be downloaded again, unless `-f/--force` is
-used. As an example, to download a CTA-WAVE asset one can run
-
-```sh
-dashfetcher --auto https://dash.akamaized.net/WAVE/vectors/cfhd_sets/12.5_25_50/t3/2022-10-17/stream.mpd
-```
-
-which will result in a locally stored DASH VoD asset in the directory
-
-```sh
-./vod/WAVE/vectors/cfhd_sets/12.5_25_0t3/2022-10-17/
-```
-
-with an MPD called `stream.mpd` and the segments stored in subdirectories named after their relative
-URLs. The download URL is added to a file `mpdlist.json` which is read by livesim2, to provide
-information about the asset.
-
-One can have multiple MPDs in the same asset directory and they may share some representations.
-That is an easy way to have variants with different representation combinations.
-
-### livesim2 server
+## livesim2 server
 
 The server is configured in one or more ways in increasing priority:
 
@@ -88,9 +58,10 @@ The server is configured in one or more ways in increasing priority:
 
 Major values to configure are:
 
-* the HTTP port being used (default: 8888)
 * the top directory `vodroot` for searching for VoD assets to be used
-* `certpath` and `keypath` is the HTTPS is used
+* the HTTPS `domains` if Let's Encrypt automatic certificates are used
+    - `certpath` and `keypath` if HTTPS is used with manually downloaded certificates
+    - the HTTP `port` if HTTPs is not being used (default: 8888)
 
 Once the server is started, it will scan the file tree starting from
 `vodroot` and gather metadata about all DASH VoD assets it finds.
@@ -109,12 +80,14 @@ that in turn points to:
 * /assets
 * /metrics
 
+and links to the Wiki page for more information.
+
 It is also possible to explore the file tree and play Vod assets by starting at
 
 * /vod/...
 
 Finally, any VoD MPD like `/vod/cfhd/stream.mpd` is available as a live stream by
-replacing `/vod/` with `livesim2`as like `/livesim2/cfhd/stream.mpd`.
+replacing `/vod/` with `livesim2` e.g. `/livesim2/cfhd/stream.mpd`.
 
 ### MPD Restrictions
 
@@ -166,6 +139,25 @@ During development it may be easier to use the usual go commands:
 
 or compile and run directly with `go run .`.
 
+### HTTPS with automatic certificates
+To enable HTTPS in an easy manner, make sure that you have DNS pointing to your machine,
+and that ports 80 and 443 are forwarded. Then use the parameter
+`--domains=your.domain.com,second.domain.com` to automatically fetch TLS certificates
+from Let's Encrypt for your machine.
+
+#### HTTPS with manual certificates
+
+The old-fashioned way of using manually acquiered TLS certificates is also supported.
+Use the two parameters `certpath` and `keypath` to point to the respective files,
+and set the `port` to 443.`
+
+## Content
+
+There are multiple ways to get content to the livesim2 server.
+From start, there is some bundled test content.
+You can also fetch content that was used with [livesim1][1] from
+github at [livesim-content][livesim-content].
+
 ### Bundled test streams with the livesim2 server
 
 A few very short (8s) test assets are bundled with the code.
@@ -194,7 +186,60 @@ Adding longer assets somewhere under the `vodroot` results in longer loops.
 All sources are NTP synchronized (using the host machine clock) with a initial start
 time given by availabilityStartTime and wrap every sequence duration after that.
 
-### Running dashfetcher
+### livesim-content at Github
+
+In the repo [livesim-content][livesim-content], the content that was used for the livesim1
+online service is being gathered to make it easy to reproduce the use cases of livesim1.
+
+All content and features are not yet (2023-08-08) in place, but should be so before end
+of October 2023.
+
+To download and use that content, run
+
+```sh
+$ git clone https://github.com/Dash-Industry-Forum/livesim-content.git
+```
+
+and then set `--vodroot` to the `livesim-content` top directory or include that in
+a bigger file tree.
+
+### dashfetcher tool
+
+The tool `dashfetcher` fetches DASH VoD assets via HTTP given an MPD URLs.
+Currently it supports MPDs with SegmentTimeline with `$Time$` and
+SegmentTemplate with `$Number$`. The assets must have no explicit `<BaseURL>` elements to
+work properly. With the `-a/--auto` option, the path to the asset is preserved
+as much as possible and adapted to the local path.
+
+Files already downloaded will not be downloaded again, unless `-f/--force` is
+used. As an example, to download a CTA-WAVE asset one can run
+
+```sh
+dashfetcher --auto https://dash.akamaized.net/WAVE/vectors/cfhd_sets/12.5_25_50/t3/2022-10-17/stream.mpd
+```
+
+which will result in a locally stored DASH VoD asset in the directory
+
+```sh
+./vod/WAVE/vectors/cfhd_sets/12.5_25_0t3/2022-10-17/
+```
+
+with an MPD called `stream.mpd` and the segments stored in subdirectories named after their relative
+URLs. The download URL is added to a file `mpdlist.json` which is read by livesim2, to provide
+information about the asset.
+
+One can have multiple MPDs in the same asset directory and they may share some representations.
+That is an easy way to have variants with different representation combinations.
+
+#### Running dashfetcher
+
+`dashfetcher` was created to facilitate download of
+DASH assets with lots of small segment files. One particular
+such source was the CTA-WAVE test content. However,
+that content is now also available as zip-files,
+so it is more efficient to download an unzip these
+files instead of making individual downloads of
+the segments.
 
 The `dashfetcher` binary can be found in as `out/dashfetcher` after `make build`.
 
@@ -228,13 +273,6 @@ More information can be found in the `./deployment` directory.
 To get information about the available assets and other information
 access the server's root URL.
 
-## HTTPS and HTTP/2
-
-HTTPS and HTTP/2 are both supported. To enable TLS encryption, the two parameters
-`certpath` and `keypath` must be set to point to a TLS certificate and a private
-key file, respectively. It is also recommended to set the port to the default HTTPS
-port 443. Automatic TLS configuration using Let's Encrypt is a future enhancement.
-
 ## List of functionality and options
 
 The URL parameters are now listed on this project's Wiki page
@@ -257,3 +295,4 @@ See [LICENSE.md](LICENSE.md).
 
 [1]: (https://github.com/Dash-Industry-Forum/dash-live-source-simulator)
 [urlparams]: (https://github.com/Dash-Industry-Forum/livesim2/wiki/URL-Parameters)
+[livesim-content](https://github.com/Dash-Industry-Forum/livesim-content)

cmd/livesim2/app/config.go

@@ -40,8 +40,12 @@ type ServerConfig struct {
 	TimeoutS    int    `json:"timeoutS"`
 	MaxRequests int    `json:"maxrequests"`
 	VodRoot     string `json:"vodroot"`
-	CertPath    string `json:"certpath"`
-	KeyPath     string `json:"keypath"`
+	// Domains is a comma-separated list of domains for Let's Encrypt
+	Domains string `json:"domains"`
+	// CertPath is a path to a valid TLS certificate
+	CertPath string `json:"certpath"`
+	// KeyPath is a path to a valid private TLS key
+	KeyPath string `json:"keypath"`
 	// Scheme should be http or https if set. Otherwise the scheme is auto-detected as far as possible.
 	Scheme string `json:"scheme"`
 	// If Host is set, it will be used instead of autodetected value.
@@ -96,6 +100,7 @@ func LoadConfig(args []string, cwd string) (*ServerConfig, error) {
 	f.String("vodroot", k.String("vodroot"), "VoD root directory")
 	f.Int("timeout", k.Int("timeoutS"), "timeout for all requests (seconds)")
 	f.Int("maxrequests", k.Int("maxrequests"), "max nr of request per IP address per 24 hours")
+	f.String("domains", k.String("domains"), "One or more DNS domains (comma-separated) for auto certificate from Let's Encrypt")
 	f.String("certpath", k.String("certpath"), "path to TLS certificate file (for HTTPS)")
 	f.String("keypath", k.String("keypath"), "path to TLS private key file (for HTTPS")
 	f.String("scheme", k.String("scheme"), "scheme used in Location and BaseURL elements. If empty, it is attempted to be auto-detected")
@@ -154,9 +159,15 @@ func LoadConfig(args []string, cwd string) (*ServerConfig, error) {
 }
 
 func checkTLSParams(k *koanf.Koanf) error {
+	domains := k.String("domains")
 	certPath := k.String("certpath")
 	keyPath := k.String("keypath")
 	switch {
+	case domains != "":
+		if certPath != "" || keyPath != "" {
+			return fmt.Errorf("cannot use certpath and keypath together with Let's Encrypt domains")
+		}
+		return nil
 	case certPath == "" && keyPath == "":
 		return nil // HTTP
 	case certPath != "" && keyPath != "":

cmd/livesim2/main.go

@@ -10,17 +10,15 @@ import (
 	"net/http"
 	"os"
 	"os/signal"
+	"strings"
 	"syscall"
-	"time"
+
+	"github.com/caddyserver/certmagic"
 
 	"github.com/Dash-Industry-Forum/livesim2/cmd/livesim2/app"
 	"github.com/Dash-Industry-Forum/livesim2/pkg/logging"
 )
 
-const (
-	gracefulShutdownWait = 2 * time.Second
-)
-
 func main() {
 	os.Exit(run())
 }
@@ -68,17 +66,17 @@ func run() (exitCode int) {
 		return 1
 	}
 
-	srv := &http.Server{
-		Addr:    fmt.Sprintf(":%d", server.Cfg.Port),
-		Handler: server.Router,
-	}
-
 	go func() {
 		var err error
-		if cfg.CertPath != "" && cfg.KeyPath != "" { // HTTPS
-			err = srv.ListenAndServeTLS(cfg.CertPath, cfg.KeyPath)
-		} else {
-			err = srv.ListenAndServe()
+
+		switch {
+		case cfg.Domains != "":
+			domains := strings.Split(cfg.Domains, ",")
+			err = certmagic.HTTPS(domains, server.Router)
+		case cfg.CertPath != "" && cfg.KeyPath != "":
+			err = http.ListenAndServeTLS(fmt.Sprintf(":%d", server.Cfg.Port), cfg.CertPath, cfg.KeyPath, server.Router)
+		default:
+			err = http.ListenAndServe(fmt.Sprintf(":%d", server.Cfg.Port), server.Router)
 		}
 		if err != nil && err != http.ErrServerClosed {
 			logger.Error().Err(err).Msg("")
@@ -88,17 +86,7 @@ func run() (exitCode int) {
 	}()
 
 	<-stopServer // Wait here for stop signal
-	logger.Info().Msg("Server to be stopped")
-
-	timeoutCtx, cancelTimeout := context.WithTimeout(context.Background(), 5*time.Second)
-	defer func() {
-		logger.Info().Msg("Server stopped")
-		cancelTimeout()
-		time.Sleep(gracefulShutdownWait)
-	}()
+	logger.Info().Msg("Server  stopped")
 
-	if err := srv.Shutdown(timeoutCtx); err != nil {
-		logger.Error().Err(err).Msg("Server shutdown failed")
-	}
 	return exitCode
 }

deployment/README.md

@@ -13,6 +13,9 @@ Make sure that the binary and vod assets are available at the paths used in the
 The binary can also be started and log in more console-friendly formats.
 See the help text provided with `livesim2 -h` to see the options.
 
+For Linode, there is a nice `stackscript` for a full machine setup available at
+[https://cloud.linode.com/stackscripts/1189972](https://cloud.linode.com/stackscripts/1189972).
+
 ## Cross compilation
 
 Cross-compilation can be done like (Linux on Mac example)

deployment/livesim2.service

@@ -0,0 +1,15 @@
+[Unit]
+Description=DASH-IF livesim2 Service
+After=network.target network-online.target
+Wants=network-online.target
+
+[Service]
+User=ec2-user
+Group=ec2-user
+ExecStart=/usr/local/bin/livesim2 --vodroot=/var/media/vod  --maxrequests=500 --domains=my.company.com --logformat=journald
+Restart=on-failure
+RestartSec=10
+
+[Install]
+WantedBy=multi-user.target
+

go.mod

@@ -5,6 +5,7 @@ go 1.19
 require (
 	github.com/Eyevinn/dash-mpd v0.10.0
 	github.com/Eyevinn/mp4ff v0.36.0
+	github.com/caddyserver/certmagic v0.19.1
 	github.com/go-chi/chi/v5 v5.0.8
 	github.com/knadh/koanf v1.5.0
 	github.com/prometheus/client_golang v1.15.1
@@ -23,17 +24,30 @@ require (
 	github.com/fatih/structs v1.1.0 // indirect
 	github.com/fsnotify/fsnotify v1.6.0 // indirect
 	github.com/golang/protobuf v1.5.3 // indirect
+	github.com/klauspost/cpuid/v2 v2.2.5 // indirect
+	github.com/libdns/libdns v0.2.1 // indirect
 	github.com/mattn/go-colorable v0.1.13 // indirect
 	github.com/mattn/go-isatty v0.0.19 // indirect
 	github.com/matttproud/golang_protobuf_extensions v1.0.4 // indirect
+	github.com/mholt/acmez v1.2.0 // indirect
+	github.com/miekg/dns v1.1.55 // indirect
 	github.com/mitchellh/copystructure v1.2.0 // indirect
 	github.com/mitchellh/mapstructure v1.5.0 // indirect
 	github.com/mitchellh/reflectwalk v1.0.2 // indirect
 	github.com/pmezard/go-difflib v1.0.0 // indirect
 	github.com/prometheus/client_model v0.4.0 // indirect
 	github.com/prometheus/common v0.44.0 // indirect
 	github.com/prometheus/procfs v0.10.1 // indirect
-	golang.org/x/sys v0.8.0 // indirect
+	github.com/zeebo/blake3 v0.2.3 // indirect
+	go.uber.org/atomic v1.11.0 // indirect
+	go.uber.org/multierr v1.11.0 // indirect
+	go.uber.org/zap v1.24.0 // indirect
+	golang.org/x/crypto v0.10.0 // indirect
+	golang.org/x/mod v0.11.0 // indirect
+	golang.org/x/net v0.11.0 // indirect
+	golang.org/x/sys v0.9.0 // indirect
+	golang.org/x/text v0.10.0 // indirect
+	golang.org/x/tools v0.10.0 // indirect
 	google.golang.org/protobuf v1.30.0 // indirect
 	gopkg.in/yaml.v3 v3.0.1 // indirect
 )