Replies: 11 comments 6 replies
-
|
I agree zones sound tempting. But I don't think it simplifies things.
This means only one But more importantly none of the services running on jails in zones will be accessible from other devices on the LAN on which the SCALE NAS is located. The user would have to setup port forwarding for this:
But port 443 is already taken by SCALE so there's another hoop to jump through. I think a one time setup of a bridge interface will provide the most versatile and easy to use setup. Reference: https://manpages.debian.org/bookworm/systemd-container/systemd-nspawn.1.en.html |
Beta Was this translation helpful? Give feedback.
-
The way I solve this is by assigning an additional IP to the bridge interface, and having SCALE listed on that IP instead of
I would agree, I found this the easiest, least-conflicting, setup |
Beta Was this translation helpful? Give feedback.
-
Well, that's disappointing. Booooo. 😩
Isn't this the current status quo, though? In both cases you have to dance around 443 being in use.
"What place" you might ask. Presumably from the equivalent of We could debate whether this would be any trickier than existing instructions to pick up and move TrueNAS host networking onto a bridge. I'd start by asserting that it's more discoverable. (The bridge would still usually be necessary; ingress would just be an extension of that procedure and would focus an opportunity to document how/why for both parts.) In principle: what I hear from forum people is that they're coming in with a mindset of long-running Docker containers. They assume up front that there will be no conflicts or interactions with the TrueNAS host, nor other nodes or infrastructure. To expose ports explicitly is a known/anticipated feature of that mindset. And anything in the way of that mindset is a source of frustration that they'd unfairly blame on Jailmaker itself. I'm not stuck on the idea. 🤷♂️ I'm satisfied just having the clear-eyed discussion. Especially if it leads to more approachability in the docs and templates. Thanks! |
Beta Was this translation helpful? Give feedback.
-
|
As I see it there are three major network options
1 requires no config on host, but you can’t use in use host ports 2 and 3 work with DHCP or static IPs and then there is veth |
Beta Was this translation helpful? Give feedback.
-
|
That sounds remarkably clear-eyed to me, and easy to document. Also FWIW wonderfully approachable in YAML form if that were to be a thing. And none of it stands in the way of daredevils delving into the non-major network options. |
Beta Was this translation helpful? Give feedback.
-
|
Personally I think port forwarding in 2 places (jailmaker and also in docker-compose) would be annoying and means modifying the jail config more often (which would require a jail restart etc.). Not great if you ask me. I toyed with the idea of a default zone instead of the current default of host networking. But this is also a bit difficult to implement nicely because it contradict the systemd-nspawn default of using host networking when no networking related flags are present. So if I were to use a default zone, when no networking flags are present, then how would one specify the desire to use host networking? That would mean inventing another flag to bring back host networking... In summary I rather stay close to nspawn default behavior when I can. If you think docs or onboarding could be improved, all help is welcome! |
Beta Was this translation helpful? Give feedback.
-
I don't get what you mean here. When using bridge networking there aren't any conflicts I am aware of. |
Beta Was this translation helpful? Give feedback.
-
Ah! Keen eye — I think you've uncovered a hidden assumption where we diverge. I suppose I'm thinking of Jailmaker as an "app hosting environment," more than as an "app hosting environment hosting environment." Wrapping Docker and Docker Compose is undeniably these same forum users' ＃1 prime monumentally significant use case for Jailmaker! But another way to say that is that it's an exceptional case. Few other use cases resemble that one; I don't consider it representative of healthy defaults for further jails. It wouldn't be that crazy to teach jlmkr.py how to spin up new jails straight out of container registries, and to automatically apply their exports. It would be a bit more crazy — but arguably more cogent — to spin up pods in the Podman sense. Where a "pod" is a jail with its own network zone, in which it hosts any number of containers, and declares its resources and exposed surface area. (This, too, feels like an exceptional case which could be expressed as a base template with a readme if I'm really so eager to prove that concept.)
Smart. 👍 Just realize that for better or worse, you've introduced an awful lot of people to nspawn who've come from other cultures. And whether you intended to or not, you've already built up a pretty extensive value-add CLI wrapper/ecosystem around it. I worry that this might settle in as the forums' next locus of frustration, if/when the honeymoon ends.
I'm assuming the primary TrueNAS SCALE interface has been moved to a bridge containing a physical interface. A jail parked right in the host networking like this can then interact with TrueNAS data services (yay!) — and trip over TrueNAS and/or the network environment within which it operates (boo.) Mind you: I happen to have been unusually steeped in such conflicts this week, and they just happen to be front-of-mind. Downstream networks, netbooting, service discovery, multicast video, captive proxy, IoT aggregation with isolation… For many users it won't come up. But for any who do — say, anyone puzzling over how to start six popular web apps when four of them try to integrate a certbot in their own ways — I'd offer that it's just easier to declare exports at the point of contact. And by and large, Docker has pointed the way for them. |
Beta Was this translation helpful? Give feedback.
-
Well, I personally have just a single jail with docker inside it. Jails made with jailmaker are system containers (similar to VMs), not application containers. Besides, I can help users better if they use a similar setup I do (which is bridge networking). And in my case I also want my jail to be in the same broadcast domain as my LAN clients. Syncthing works better that way.
I tried this and it's not worth the trouble IMHO. The jail rootfs needs systemd for it to run with nspawn. Most docker containers don't include the systemd init system, so you'd have to customize the image (build a new one). Unless the image is based on alpine (very common)... which doesn't have systemd and therefore just wont work at all with jailmaker. Jailmaker isn't (and shouldn't) be a replacement for docker/podman. But the default is still host networking (without
I can imagine a zone would be helpful in such cases, maybe an improvement for the router template? |
Beta Was this translation helpful? Give feedback.
-
Aha! OK, FYI this is a noteworthy divergence from how our predecessors use "jails" on FreeBSD and TrueNAS Core. Honestly: for just the zero-to-Docker case I'd posit that a single-purpose installer script would help more people faster. On my Core server I currently have separate long-running jails for… MariaDB, PostgreSQL, CouchDB, Elasticsearch, Solr, Nextcloud, Hugo, LL-HLS, Nginx, and Lego. (GitLab in a Debian VM; it came first.) Several of these jails own and curate datasets which the others then mount read-only, whether to serve to each other and/or other machines and/or the public. A couple of them vend Unix domain sockets to each other, rather than going through TCP. They're kept separate not to be spun up and down as externally-packaged geegaws, but so that e.g. my bottom-up Nextcloud installation can use whichever version of PHP that it wants. So that I can update/troubleshoot/re-troubleshoot/keep-troubleshooting that blessed app and its dependencies independently, on its own schedule — without fear of affecting any other services. In Python terms: FreeBSD jails are full user-space I plan to keep my databases in their own jails on the TrueNAS appliance — treating them as just more data storage services like NFS and SMB. (I may soon do that with Syncthing as well.) But I'm working to migrate app-layer compute and front-end services off of the TrueNAS appliance; onto a proper Kubernetes fleet. That's been a long walk, and countless old pets-v-cattle habits to unlearn.
Solid point. When I brought up containers I wasn't yet up-to-speed on where the dual template/config format came from and what could be done with it. I'd still offer that by default all port exposure should be declaratively expressed through that file. No surprises. Any undeclared port forwarding could be tackled dynamically through Meanwhile I wholeheartedly respect that a Docker runtime jail is its own very different creature and wouldn't involve that pattern. |
Beta Was this translation helpful? Give feedback.
-
There's a script, inspired by jailmaker, which does that. But this approach is mostly obsolete in the next release of SCALE which will include docker instead of k3s. Jailmaker isn't a a single-purpose solution so it will remain useful. It's probably a bit opinionated towards my use case and as you may have noticed I have 0 experience with jails on FreeBSD, so it's not a 1-on-1 copy of FreeBSD jails. |
Beta Was this translation helpful? Give feedback.
-
|
Bringing my voice as just another user. The way I see it, at this point of time - what Jailmaker introduced is an easy way to manage a pool hosted OS - including a docker host, on the immutable TrueNAS SCALE. Nspawn is still installed and can be used directly if one would want to maintain it extensively. Documentation is key here, as knowledge is power, and that's where the versatility comes into play. I would venture to promote rootless-podman or NixOS based configurations inside jails as systematic solutions which can allow managing of proper security from within the hosted OS. Is there an overhead? yes - but if we didn't want that overhead, we'd install another Linux distro, and not TrueNAS SCALE. |
Beta Was this translation helpful? Give feedback.
-
Rock on. 😎 And meanwhile, iX can puzzle through their own product decisions for the contours of some near-mid-future(?) integrated Sandboxes offering. (I had kinda hoped that this issue would stick around longer and spark more discussion of far-flung possibilities, but of course they have their own forum for that.) Right here and now: nobody needs some newcomer to second-guess how Jailmaker is organized. It's a brilliant idea serving an extremely helpful, very welcome purpose for which I'm entirely grateful. Cheers! 🍻 |
Beta Was this translation helpful? Give feedback.
-
For that my friend - you should have opened a discussion 🧐 (See #135) @Jip-Hop can you please convert this issue to a discussion? |
Beta Was this translation helpful? Give feedback.
-
Ha! Touché. 🤕 That's an exquisite primer on GitHub's discussions feature. 🏆 Speaking as someone who clearly needed one, or I might never have found it. |
Beta Was this translation helpful? Give feedback.
-
|
To circle back to the original topic. The |
Beta Was this translation helpful? Give feedback.
-
|
Turns out |
Beta Was this translation helpful? Give feedback.
-
Proposition
Should Jailmaker default to creating new jails with
--network-zone=jlmkrunless otherwise specified?Rationale
Theoretically, that would create a
vz-jlmkrbridge on-the-fly when referenced by at least one active jail. It would then dispose of that bridge on-the-fly whenever the active-jail refcount drops back to zero.Out-of-the-box
/usr/lib/systemd/network/80-container-vz.networkon the host sets up DHCP and IPv4/IPv6 NAT forwarding on behalf of participating jails. Importantly: without any loose pre/post scripted side-effects to the host.Meanwhile all participating jails in the same zone could reach each other, and reach the outside world. (I'll say again: theoretically.)
Implications
I'll still need a router/dnsmasq template for my external netboot clients. But that template might simplify down into attaching a hardware interface to a zone and installing/configuring only dnsmasq.
Similarly: multiple bridges might still need scripted host side-effects. That is, unless testing shows us that nspawn respects multiple
--network-zone=arguments producing multiple refcounted zones.But for that new/casual user default… might a standard zone be the smartest/easiest catch-all?
Kinda seems too easy… 🧐
Beta Was this translation helpful? Give feedback.
All reactions