Join efforts with Gluon

[CodeFetch]

Why doesn't Freifunk Berlin try to incorporate their packages into Gluon?
Having separate projects is much double-work. By joining efforts we could build better firmware.

[SvenRoederer]

I've only basic knowledge of Gluon, but the main problem I see is the different network-design.
Gluon is based on L2-switching (B.A.T.M.A.N.), Berlin is based in L3-Routing (sadly still Olsr v1).
But in the recent development-code we added the gluon-packages feed already.

[CodeFetch]

I've seen that you think about switching to another Layer 3 protocol. Gluon supports Babel which can be seen as a successor of OLSR. There is only missing a kernel module I'm developing for IPv4 support. At the moment Gluon's Babel implementation only supports IPv6. I don't know whether you can live with that restriction, but sooner or later it will be a full replacement of OLSR.

[pmelange]

I don't think that it makes a lot of sense to put the freifunk-berlin packages into gluon, but collaboration is still possible in the sense that we could work together to support new hardware and possibly also share patches to OpenWrt.

With regard to new hardware, one of the issues which freifunk-berlin is facing right now is driver support for the ath10k devices. We are migrating to 802.11s and the -ct drivers do not support 802.11s in wave1 devices.

[CodeFetch]

@pmelange Why do you think it does not make sense?

[pmelange]

Perhaps @booo @db4rne @sarumpaet can also give their impressions, but from what has been discussed in meetings and on chat the following points are important.

Some of these points may be incorrect, i'm just copying and pasting what I can quickly find. This includes feedback from other communities outside of Berlin

• gluon is like a "all-in-one" package with lot's of opaque c-stuff
• gluon is pretty centralised
• gluon has a lot of services that depend on a layer2 mesh and most of them are written in c = pain in the ass to debug on a router
• freifunk-berlin's wizard is quite hackable as it's posix sh and lua
• freifunk-berlin's openwifimap is also just a lua script
• freifunk-belrin is excellent for hacking/experiments/modifiying stuff
• the simplicity of the berlin firmware ffwizard.d directory, where you put a sh file and run some uci callback functions on config file, is pretty cool and super flexible

I'm sure I could dig around a bit more.

[SvenRoederer]

I remember a short moment in time, where we used you tunneldigger-package. But for some reason we started an own fork soon. @pmelange might remember the details.

[Akira25]

@CodeFetch I don't understand your question completely. Do you want freifunk-berlin to port its original packages to gluon and to use a common package-base? Do you want freifunk-berlin to switch to gluon? Maybe these two points depend on each other...

For the first point: It sounds like a nice idea. I would agree on that for efficency reasons in developement. But as far as I understood pmlange right, gloun seems not flexible enough to grant berlin-specific adjustments.

For the second point: No.

I do not like gluon for some reasons:

• It is extremly centralised: one server doing wrong can set the whole network out of service (for example a dying DHCP-Server)
• for it is centralised, it needs an admin team. I heard several opinions on that and think, that everything gloun gives the users for carefreeness gets more workload for the admin team. I think most freifunkas in berlin would spend their free time more hapily with other funny things, than carrying that burden to administrate the whole network. Lets mind the concept of single point of failure. It might even rely on humans... ;-)
• gluon can only be configured when triggered by a hardware-botton. Please correct me, if i got it wrong. But that is an unsuitable requirement for the freifunk-berlin net. Core-routers must be configureable all time. Even remote.
• freifunk-berlin would loose (some) of its independency. I suppose there were good reasons to start to devolp an own firmware. Maybe for net-architectural resons, maybe for many others too. If freifunk berlin would switch completely to gluon (wich would be, by the way, a horrible pile of several months work), those might not be met anymore. What is fine in many other cities might be inapplicable in Berlin... In my opinion, an own firmware fulfill the own requirements best..

[SvenRoederer]

regarding a common packages-repo: there is https://github.com/freifunk/openwrt-packages. It's hosted under the flag of the global Freifunk-team.

[CodeFetch]

I see your points. You expect the development effort to increase, but I think in the long-run you wouldn't have any great effort if you don't want to participate in development anymore.
Actually I think if you'd use Gluon as your base and add the functionalities/packages you need the effort would decrease most as there is device support to handle etc.

• gluon is like a "all-in-one" package with lot's of opaque c-stuff

It is a framework and aims to be very customizable. You can build it with basic functionality only. The C-stuff is either something that just did not exist before or for a critical function that would take to much resources if it would have been written in an interpreter language.

• gluon is pretty centralised

We are working on changing that. With Babel everything will be completely decentral and more autonomous. We are also working on a similar solution with Batman, but as you said, you're not interested in layer 2.

• gluon has a lot of services that depend on a layer2 mesh and most of them are written in c = pain in the ass to debug on a router

We are changing these packages to comply with layer 3 mesh protocols atm.

• freifunk-berlin's wizard is quite hackable as it's posix sh and lua

Most of Gluon's scripts that are likely to be altered by beginners are written in Lua, too.
We consider ash messy and thus prefer Lua.

• freifunk-belrin is excellent for hacking/experiments/modifiying stuff

Gluon does not restrict you in any way. Packages that should be merged just need a relatively high coding quality.

• the simplicity of the berlin firmware ffwizard.d directory, where you put a sh file and run some uci callback functions on config file, is pretty cool and super flexible

We consider these things a hack as when you have a lot of those scripts, it is getting a bit messy, but we are working on providing a similarly modular functionality that is likewise easy to use.

• It is extremly centralised: one server doing wrong can set the whole network out of service (for example a dying DHCP-Server)

We are working on changing that. Every node will be able to provide Uplink/VPN or whatever with the upcoming Babel concept. We even have a decentralized DHCP daemon. (DDHCP) for that.

• for it is centralised, it needs an admin team. I heard several opinions on that and think, that everything gloun gives the users for carefreeness gets more workload for the admin team. I think most freifunkas in berlin would spend their free time more hapily with other funny things, than carrying that burden to administrate the whole network. Lets mind the concept of single point of failure. It might even rely on humans... ;-)

That applies to us, too. That's why we are working on the decentralization concept. A supernode should be nothing than a KVM running a Gluon image for supernodes in the end. Little knowledge needed for running the infrastructure then.

• gluon can only be configured when triggered by a hardware-botton. Please correct me, if i got it wrong. But that is an unsuitable requirement for the freifunk-berlin net. Core-routers must be configureable all time. Even remote.

You can configure it remotely using SSH. Do you have a web interface for that? We decided against this approach due to missing proper certificates/TLS. How did you solve the problem?

• freifunk-berlin would loose (some) of its independency. I suppose there were good reasons to start to devolp an own firmware. Maybe for net-architectural resons, maybe for many others too. If freifunk berlin would switch completely to gluon (wich would be, by the way, a horrible pile of several months work), those might not be met anymore. What is fine in many other cities might be inapplicable in Berlin... In my opinion, an own firmware fulfill the own requirements best..

The last thing Gluon wants is to become "the Freifunk firmware". What we want is to collaboratively develop a very customizable framework to make it easier to develop one's own firmware. As this is a lot of work it makes sense to me, to join efforts. If there wouldn't be a hundred linux distributions in the wild with different benefits, but people would work on a single distribution that tries to unify those benefits, Microsoft and Apple would be dead by the time. The best example for this is the linux kernel itself. It is a masterpiece of software and if we could craft a masterpiece of a mesh framework, this would be a great thing.

[booo]

@CodeFetch Could you provide a link to the babel stuff you mentioned?

[db4rne]

@CodeFetch let me ask why you don't join LibreMesh then? I Would also like to know if you tested babel in a big mesh already and how it performed? As far as I understand it, babel is proactive (triggers an update when a route changes), and that could bring problems in big mesh networks? So, if you already tested it I would be very interested in your results.

[CodeFetch]

Unfortunately most of the links (including the bigger test network's map) are offline:
https://wiki.ffm.freifunk.net/firmware:babel

On the Battlemesh we gave presentations about the plan, but unfortunately the video recordings were never published.

Actually I'd like Gluon to become the base of LibreMesh when we've done the needed changes.

Regarding the proactive triggers that lead to route changes: These are only being distributed to the nodes to which they matter. The scaling issues are less problematic compared to Batman. To be fair Batman V has improved much and with enough RAM and bandwidth even Batman V networks can scale quite good.
With the introduction of "domains" Gluon does not really have any scaling issues anymore. The new problem is that border routers (routers on the border of different domains) still need to learn to route traffic between the domains/act as a gateway between them. I hope that this will come after the merge of the hoodselector. At the moment domains mean a hard cut between networks and traffic between domains is being routed on layer 3 which means roaming might break and the devices depend on supernodes for client-to-client communication.

I'll try to briefly explain the Babel L3 concept:
l3roamd: Is a software that manages Babel's route distribution of clients in Babel networks. A special IPv6 management address is being calculated from the client's MAC address and routes to the node that serves the client. When a client roams to another node the new node sends a request to this special address and i.a. gets a response with the IP addresses the client uses. The old node releases the route of the special IP address while the new node sets it together with the client's IP addresses to point to itself. This is also a good example for why Babel does not need to distribute route changes through the whole network. Only the neighboring nodes of the nodes involved in the roaming process need to know these updates.
DDHCP: is a distributed DHCP daemon. Nodes acquire a range of leases and offer those to their clients. Before they run out of leases, they acquire a new range. Thus there does not need to be a central DHCP server.
mmfd: is a simple replacement for broadcast traffic which sends traffic to the other nodes as unicast. It is being used for important management traffic only in L3 networks like statistics or public S2N offerings.
S2N-VPN: Is a concept to let nodes serve as public supernodes by offering ports for VPN connections that can be established over the internet.
N2N-VPN: Is a concept to let nodes establish connections with each other if the user wishes to do so using manually defined "trusted" nodes.
PLAT: Is a NAT64 server which offers IPv6->IPv4 translation (acts as a gateway for IPv4 traffic in IPv6-only networks). You can e.g. use this to share your internet connection directly in the network (e.g. if the node is running in a cafe or if it is a KVM in a datacenter) or to offer exit of IPv4 traffic to a VPN provider.
NAT426: As our Babel networks are IPv6 only this takes care of "faking" an IPv4 gateway for the clients. It is a conceptual kernel module (in an experimental state due to my lack of time) that does NAT46 address translation so that multiple PLATs can be used transparently. It does connection tracking which allows connections that don't break when a node's route's metric to a specific PLAT becomes less attractive. New connections to different IPs use the better PLAT then while existing connections stay using their individual PLAT. This also allows seamless IPv4 roaming as l3roamd takes care of distributing the connection tracking information.
hoodselector/domains: Is a concept that in the end allows automatic clustering of mesh clouds which share a common IPv6 address space in the Layer3 case. There is still a unified concept missing for routing traffic between the domains, but it is only a matter of time and manpower.

As you can see the Babel concept is designed to allow a very high degree of decentralization. In the end the only thing that people need to achieve a consensus on is the software they use and some central servers or a public register to allow to enter the network. Having entered the network once, it is only a question of people willing to provide public peerings (s2n). But those are the same things we see in e.g. onion routing networks or DHT networks. At some point you just need an ability to easily enter the network. With the new concept it is not a single point of failure anymore as the network still runs even if the public entrance servers go offline. Of course people need to be encouraged to provide many S2Ns and do many N2N peerings, but those who really want to use Freifunk like its initial purpose suggests will do and those who don't won't...

You can't force people to build a zombie apocalypse network, but you can give them the tools to.

[SvenRoederer]

Thanks for the extensive overview. Some of the concepts you outline seem not the far from what we have, e.g. DDHCP (we have a config.berlin.freifunk.net a service where a node operator can request an IP-range that will be provided to the clients).
Others might be a good chance / idea to improve our net. Currently we are stuck in a nearly IPv4 only net and also looking for some replacement for OLSR. As every node can potentially share it's internet-connection, we have some ideas how to handle the multpile-IPv6-prefixes problem. Probably the PLAT and NAT426 are some options to discuss.
For roaming clients between some L3 neighbors we currently use BATMAN to form some clouds, but it's not our intention to provide seamless roaming over the whole city of Berlin.

P.S. I recent hacked a change that enables us to share some patches more painless (#714).

[SvenRoederer]

I'm looking forward to reuse the gluon-way of patching openwrt and feeds. For this I'm currently working on branch "https://github.com/freifunk-berlin/firmware/tree/master_gluon-patch"

[PolynomialDivision]

@pmelange

gluon is like a "all-in-one" package with lot's of opaque c-stuff

I like c. :P Sometimes u need a small efficient daemon.

@CodeFetch

Having separate projects is much double-work.

I don't understand why we should do something else than OpenWrt. Why not use the structure and everything. Instead we do everything different again. We just had a dispute about our firmware and the result is a package-feed which I made compatible with OpenWrt again, so that you can use the imagebuilder. https://github.com/Freifunk-Spalter/packages
Our network looks a bit different than other community networks. We have these large sites that we like to configure ourselves. I feel closer to Freifunk Franken. https://github.com/FreifunkFranken/firmware

Right now I'm trying to get a part of the city under my control, e.g. to try new things, like a new mesh-routing daemon.

I find olsrv2 more interesting than babel, because physical layer information is included. We have to find a way to get the physical information out of the ubnt devices. I also find the code very well documented and very modular. I also like that you can send requests to the daemon via socket

I'm scared to change to olsrv2 because all communities are changing to babel. but maybe it's also a good thing that we are looking for different solutions and that we make different experiences from which we all profit in the end. :D (But right now the developer is a bit unresponsive :/ )

Last question, how do you want to auto configure which route changes babel will accept or not? I've played around with babel myself and saw that the config allows this very easily. Freifunk Franken, for example, simply throws everything away on the "leaf nodes". But they do it manually (as far as I know).

All in all I actually want to go back to the roots with OpenWrt + Luci. As long as we talk the same mesh protocol, everyone can do what they want. :D (but we need some ipv4/ipv6 address collision avoidance...^^)

[CodeFetch]

I don't understand why we should do something else than OpenWrt. Why not use the structure and everything. Instead we do everything different again. We just had a dispute about our firmware and the result is a package-feed which I made compatible with OpenWrt again, so that you can use the imagebuilder. https://github.com/Freifunk-Spalter/packages

The focuses of OpenWrt aren't mesh networks and fitting the needed packages on a router with a small flash. When Gluon can introduce changes in OpenWrt directly, it does so...

Our network looks a bit different than other community networks. We have these large sites that we like to configure ourselves. I feel closer to Freifunk Franken. https://github.com/FreifunkFranken/firmware

I wish Freifunk Franken would too be interested in contributing to a common codebase, but we had a dispute about usability a few years ago which they don't care much about.

I find olsrv2 more interesting than babel, because physical layer information is included. We have to find a way to get the physical information out of the ubnt devices. I also find the code very well documented and very modular. I also like that you can send requests to the daemon via socket

It would be great to support it in Gluon, too.

Last question, how do you want to auto configure which route changes babel will accept or not? I've played around with babel myself and saw that the config allows this very easily. Freifunk Franken, for example, simply throws everything away on the "leaf nodes". But they do it manually (as far as I know).

I think in the end we need an approach similar to DDHCP for distributing IPv6 prefixes for hoods/domains which will then be formed according to physical wireless mesh clouds. Border routers (which are in multiple domains) must forward the prefixes and those prefixes' routes are only distributed by babel within the domains.

All in all I actually want to go back to the roots with OpenWrt + Luci. As long as we talk the same mesh protocol, everyone can do what they want. :D (but we need some ipv4/ipv6 address collision avoidance...^^)

Most of the firmware's users in our community don't have a clue about routing. Even worse Freifunk communities exist which don't have a clue either... I think we (the specialists) should ally on developing a solution which is technically pleasing to all of us while hiding the technology from the user. I know, that's Apple, but honestly wouldn't you like it if it's free software and has ssh? I often experienced Freifunk scared away so much needed non-technical people because of its complexity. For us it's simple, but it's not. We just got used to the pain.

[PolynomialDivision]

The focuses of OpenWrt aren't mesh networks and fitting the needed packages on a router with a small flash. When Gluon can introduce changes in OpenWrt directly, it does so...

I don't care about routers with small flash, too. ;) I don't like it when my traffic flows over a very old router with limited cpu and ram capabilities... :/ I always say, use stock-firmware (ubnt) + bridge or use OpenWrt and kmod-trelay + bridge traffic to core-router that has more power. U even can bridge adhoc or ibss.
I think it is better to contribute to OpenWrt upstream than doing a fork. Example: Instead of every community tracking their own packing feed in which they have a different babel version, we should use the same feed.
At the end, if we had access to ath10k or Mediatek and could do the stuff UBNT is doing, OpenWrt would be used for long distance links more often.

I have to think about your last points for a while. ;)

[Akira25]

Most of the firmware's users in our community don't have a clue about routing. Even worse Freifunk communities exist which don't have a clue either... I think we (the specialists) should ally on developing a solution which is technically pleasing to all of us while hiding the technology from the user. I know, that's Apple, but honestly wouldn't you like it if it's free software and has ssh? I often experienced Freifunk scared away so much needed non-technical people because of its complexity. For us it's simple, but it's not. We just got used to the pain.

I understand the idea of this. But I'd like to mention, that apple-freifunk seems pretty useless to me. At least, if it'd too apple-ish. Everyone housing a Freifunk node, should know at least some very basics, to repair the node himself. That's a very important idea of Freifunk: Giving people back their sovereignty in the digital space. I feel, that most gluon-communities don't care on that.

Don't get me wrong: I think Gluon achieved simplicity in a wonderful way! Setting up a gluon node is really easy, hence the user will probably get no clue what will happen in background. But Gluon stands for a way of doing Freifunk in a fundamentally different way, than we do in Berlin. This isn't necessarily a bad thing! It is good to have different approaches to the same topic.

I totally agree with nick, that plain OpenWrt should be the way to go. Writing everything as a package, reduces the development effort even more, than having gluon.

[Akira25]

I propose to close this issue. I consider the question on whether to use gluon or not as partly religious. Such things should be discussed privately. :)

[SvenRoederer]

"Apple-Freifunk"

is a thing some users want to have, others are more interested in the "behind the scenes".
The "Apple-Freifunk operators" just like the idea of sharing ressources, but don't want to invest much (any) energy (in terms if learning "Freifunk node-operation" knowledge or time) in operating this node. They just want it have it running. So its a weight off between taking care for them and spread the Freifunk-idea or just ignore them and let them go away. For these users Gluon seems nice.
For the "techy" Node-operators the "Berlin concept" might be more appealing.
Currently this decision on the Firmware depends on the community and the operator has nearly no chance to use a "foreign" firmware. Not sure if this will change, but it would be great to have some interop here.

OpenWrt for Freifunk

I see the same way as @CodeFetch . OpenWrt is targeted more towards small networks of homeusers not for city-wide networks. There are some decisions made, which are not useful for any Freifunk-community, but for the OpenWrt mainstream. Some of these needs / changes can be made with packages, which is the preferred way. But some changes to OpenWrt will require a lot of work to modify them or are simply not possible as package.

Common Freifunk packages

which are the initial idea mentioned here can for sure placed in the https://github.com/freifunk/openwrt-packages repo. It's just a thing of finding a common issue and adding a package maintained by "both worlds" there. https://github.com/freifunk-gluon/packages/tree/master/net/simple-tc might be an interesting candidate to move there.
In general every community will have some packages that are only relevant for them (e.g. individual Monitoring and stats) and some of these packages grow and will become interesting for other communities. But moving from the communities repo to a "public" one might be hard, as the there is the risk of "loosing the control of it".
Beside the packages I drafted [1], and still have it on my list, a merge of the Freifunk-Berlin buildsystem and the Gluon buildsystem, which was working for my PoC.

1 - #765, freifunk-gluon/gluon#1946

[CodeFetch]

I've thought very much about what you said and I give you the points that: Freifunk should be about building real backbone networks in the long-run and the current single-node VPN approach is a hack. I'd love to have something like the Berlin Backbone in Hannover. Furthermore you are right that it's very frustrating to deal with non-technical people when they have problems, but I think we can fix these issues so well that the users won't have these problems often anymore. Making everyone a network guru is utopian. Please don't get me wrong: I don't want an Apple Freifunk that is not technically excellent, but I'd like to have the same standards for the user experience. That's one of the biggest problems for open-source Projects. Until someone with money comes along and makes it a successful project for the masses, it's for the nerds. Think of any invention... That's in the nature of our thinking. Our kind are problem solvers. We aim for solving the biggest problems we see which are mostly innovative and technical while the user stops at the confusion state. We are used to banging our head against the keyboard, used to circumvent flaws in software. The level of frustration for the end-user is hardly imagineable for us.

My vision for the new layer 3 mesh approach is to create a truly decentralized network. The possibility to just run a Gluon image on a VM in your datacenter and it becomes an exit node. To just share your internet connection directly if you are a cafe or something. To just hook your router to a Mullvad VPN or whatever, share the internet connection. Furthermore allow manual and automatic peerings. Announce your services like internet exit via some distributed database... Freeing Freifunk from any other authority than the node owner's. You decide who sees your nodes IP address and who your router peers with etc. A really decentralized network of trust. Furthermore I'm thinking about privacy hardening. A concept where mesh clouds can detect to which other mesh clouds a client can roam. These mesh clouds then form a union in which the do RARP to get masqueraded IP adresses from the clients. This way it is not possible to track clients around the network except one is physically in that particular mesh cloud. As all good things it comes with a drawback - that this requires some form of NAT66 accompanied by a modified DHCPv6 server. But truly... Didn't NAT work with IPv4? Is it worth to give up ones privacy for getting rid of it with IPv6? I don't want to convince you of my ideology. I just want you to understand my vision the vision I orginially thought Freifunk was all about - making connections and empowering people to make them on their own. Maybe you share a similar one. The next generations of routers with 256 MB RAM or more can handle the routes to all Freifunk nodes in the world with ease. I think we still have barriers in our head from the previous hardware limitations. We should smash them and for that we need the original Freifunk spirit.

Edit: If you wonder what I mean with the RARP, the idea is:
All nodes in mesh clouds that a client could physically roam to build up one mesh domain automatically based on their experience using VPN and preferably physical connections. They regularly generate a common salt which can be considered random if at least one node correctly contributed random data. When a client connects to a node, the node then sends a broadcast message to all nodes in the mesh domain with the MAC address hashed together with the current salt along with the node's public key. The nodes that receive this "RARP" request then check if one of their former clients matches this hash. If yes it replies with a packet encrypted with the public key that contains the IP addresses the client used. Therefore if the hash function is made costly enough (e.g. by increasing the number of rounds) it is hardly possible to associate a MAC address with an IP address except one watches it from the sky with Echelon satellites or something. But there is also a concept for encrypted mesh links with trust on first use :-)...

Edit2: Very nice explanation of that topic: https://m.youtube.com/watch?v=2nl54x6_nH8&t=300s
Unfortunately many devices don't properly use privacy extensions and can not be configured to.

[CodeFetch]

This is what I meant with Apple-Freifunk - "You've got to start with the customer experience (...)":
https://youtube.com/watch?v=oeqPrUmVz-o&t=1m30s

[mtippmann]

(wrote a long comment and deleted it after realizing that's probably not the place for it)