Have you met OwnTracks?

[mrmay-dev]

TL;DR: If OpenHaystack and OwnTracks have a baby it would probably work like this. This article distills the essentials of connecting OpenHaystack location reports to the OwnTracks location sharing platform. OwnTracks is available for Android, iOS and web. It is an excellent platform for viewing OpenHaystack items. Location reports can be shared to multiple users or kept private on the device. Integration with HomeAssistant, OpenHab and MQTT opens the door to home automation with OpenHaystack items.

Have you met OwnTracks?

Hi all, here's a little something I've been working on. I hope you get the How I Met Your Mother reference.

T.O.C.

If you are familiar with OpenHaystack and MQTT, skip down to what you need.

• OpenHaystack Server: A general description of current options for starting with OpenHaystack
• MQTT Broker: Very brief thoughts on installing an MQTT broker.
• Working with OwnTracks: Just what you need to know to get OpenHaystack talking to OwnTracks.

My son (Android) and I (iOS) are really good at losing stuff. We share our locations and important items by combining OpenHaystack with OwnTracks. This helps us to keep track of, and find, shared items like umbrellas or the family car. It also prevents us from losing personal items like jackets, keys and my coffee mug.

OpenHaystack is an open-source project for creating tracking tags -- like Tile and AirTags -- that take advantage of Apple's AirTag location framework. OwnTracks is the open-source response to Apple and Google's location sharing systems. This article describes how OpenHaystack and OwnTracks can work together to create a robust, open-source response to item tracking.

OwnTracks + OpenHastack make an excellet pair. OpenHastack tracks locations of tagged objects, a script publishes these locations to MQTT and OpenTracks provides Android, iOS and Web interfaces to the locations. Beyond that, the OwnTracks MQTT protocol is integrated with HomeAssistant and OpenHab making it suitable for home automation.

OpenHaystack server

OpenHaystack requires a macOS environment. This can be done by using an actual Mac computer that runs at least Monterrey (macOS 12). I'm writing this on a mid-2012 MacPowerBook running the OpenCore Legacy Patcher to upgrade to Monterrey. Another option is to spin a virtual server as described at the headless-haystack project.

Once the machine is running, a location server can be run in a couple of ways:

• OpenHaystack Simple Server I don't know if this fork is integrated to the main project. I have not been able to get the server working.
• FindMy provides FindMy_proxy.py which can be run as a daemon.
• headless-haystack tries to unify several projects for simpler handling of custom BT-devices with Apple's FindMy network. The goal is to run a headless MacOS without the need to have a real Mac and have to install mail plugins or openhaystack itself.

Fetching location data from the servers can be done with these scripts. These return JSON objects.:

• AirTagCrypto.py is a python library that will decode location reports. It is used in the openhaystack-grafana project to share data with InfluxDB and generate a map in Grafana.
• FindMy provides the FindMy_client.py client to decode location data. Again, headless-haystack has some useful variants.

Combined server client solution:

• Also in the FindMy repo is the report_request.py script that requests from macOS and returns the results as JSON objects as a one off. The advantage here is that there isn't a server running in background taking up resources. It can be automated with cron as part of a shell script or run directly from the terminal prompt.

MQTT Broker

This guide focuses on the MQTT connection to OwnTracks. However, there is also an optional HTTP mode. Clients POST the same JSON messages to the endpoint as they would to an MQTT broker.

Installing an MQTT Broker can be done on the same machine running OpenHaystack, on another machine on the network or with a commercial MQTT Broker. Mosquitto.org has excellent instructions. On macOS, they recommend Homebrew; for Debian based Linux, the apt repositories are up to date. Considering how well those have worked for me, I'm sure the windows system is very good too.

The default setup works perfectly fine but if you are concerned about users seeing each others topics, check out the section on security.

Using Mosquitto MQTT

paho-mqtt is the Python library used to communicate with the broker.

Mosquitto Broker installs with two command line tools for testing: mosquitto_pub and mosquitto_sub.

Location reports are sent to the broker as JSON payloads. A typical location update with mosquitto_pub looks lie this:

mosquitto_pub -h localhost \  # the broker's address
  -t owntracks/user/device \  # topic to publish at
  # the JSON payload
  -m {"lat": 33.814498,"lon": -117.9239239,"tst": 1681734317,"_type": "location"} \
  -u mosquitto_user \        # broker's user
  -P user_password \         # broker's password
  -d \                       # indicates debugging for verbose output
  -r                         # set message as retained

Working with OwnTracks {#working-with-owntracks}

• Using location updates
• Using waypoints
• Security & Encryption

To track OpenHaystack objects in OwnTracks, users publish messages containing location data to the MQTT broker. OwnTracks reads these messages and updates the app.

This guide describes the bare minimum needed to create JSON mesages that share location information. There is extensive documentation of the JSON API on the OwnTracks site.

There are two ways of publishing location reports: as waypoints or as location updates from a friend or one of your devices.

Locations vs.waypoints

Location updates are made by the devices where you've installed OwnTracks. A phone, tablet, and computer would be devices. Waypoints are locations in the physical world marked on the map, they can have a monitoring radius that triggers an alert when entering or leaving. In iOS, waypoints can also be iBeacons that that update their location when you enter or leave their transmission range.

'Haystack tags are little like all the above. They are personal devices that move around and be tracked. They are objects in physical space we want to locate, and we want their location to be updated regularly. We may even want to know when we leave one behind.

Deciding how Haystack Tag is presented to OwnTracks will depend on what you want. It is also possible to use both at the same time to use nearly all the features (iBeacon tracking doesn't work with Haystack Tags) are available.

Location Updates: for shared items

• Are best for items that need to be shared with other users.
• Can be seen by anyone with access to the topic the device publishes on.
• Can have pictures and names attatched to them.
• Too many items can clutter the map.

waypoints: for private items

• Are best for items that shouldn't be shared with other users.
• Are shown as pins on the map. They take up little room.
• Can have a monitoring radius. Entering and exiting this radius can trigger alerts from the OwnTracks app.
• Are stored on the user's device, so are only visible to the user who creates them.
• If shared with other users, their feeds will not update with new locations.

By creatng both Location Updates and waypoints the OwnTracks interface with put them together creating the impression of a single item with an image (like location updates) and a monitoring radius (like waypoints).

Using Location Updates

Location updates are simple. However, with a little added complexity, they can have custom names and pictures attatched to them. Tags will appear as devices or friends.

• MQTT Topics
• JSON Payload
• Cards

MQTT Topic

Location Upadates are published as retained messages on an MQTT topic that identifies the user and the device. They look like this:

owntracks/user/device

So, for my ESP32 board the topic could be: owntracks/mrmay/esp32

Location reports sent to this topic should be retained. This way applicatons can always access the latest report available. To delete a location report simply send another retained, empty message.

By default, OwnTracks applications will read location data on the broker at owntracks/+/+. This way, the appliction receives all updates from all users and devices on the broker. Consider this when publishing location data.

JSON Payload

To share a tag's location, publish a JSON object containing the required information to the correct topic.

A minimal JSON object to share a tag's location would look like this:

MQTT Topic: owntracks/user/device:

{
    "_type" : "location",
    "lat" : 33.814498,
    "lon" : -117.9239239,
    "tst" : 1681734317
}

• _type_: indicates the kind of message this is
• lat: is the latitude
• lon: is the longitude
• tst: UNIX epoch timestamp in seconds of the location fix.

These are just the required fields, there are many more options.

mosquitto_pub example:

mosquitto_pub -h localhost \  # the broker's address
  -t owntracks/princess_leia/millenium_falcon \  # topic to publish at
  # the JSON payload
  -m {"lat": 33.814498,"lon": -117.9239239,"tst": 1681734317,"_type": "location"} \
  -u mosquitto_user \        # broker's user
  -P user_password \         # broker's password
  -d \                       # indicates debugging for verbose output
  -r                         # set message as retained

That's it. All devices with access to that topic will pick-up the message and display the location.

Easy peasy.

Cards

Cards provide a way to update user and device details with an image and full name.

Cards are published to: /owntracks/user/device/info

{
  "_type": "card",
  "name": "Princess Leia",
  "face": "iV1CFEVkMhmCIKBUKh3 ... ghAAAAABJRU5ErkJggg=="
}

• _type_: indicates this is an information card
• name: the full name of the person or device identified.
• face: a Base64 encoded image. Less than 200x200 pixels is recommended.

Using waypoints {#waypoint-updates}

Updating the locaton of waypoints requires a slightly more complex JSON payload and can only be done on devices that allow for commands to be sent via MQTT.

• MQTT Topic
• JSON Payload

MQTT Topic

Waypoints are stored only on the device. They can be updated remotely only if the device allows commands to be sent on its command cmd topic. To check if the OwnTracks app will receive commands go to the info section, open settings and scroll down to the cmd option, make sure it is on, or True.

owntracks/user/device/cmd

JSON Payload

When a waypoint is created, the timestamp is placed in the tst. When it is re-published (to update the desc, rad, lon, lat, etc.) the original timestamp must remain the same. Otherwise, a new waypoint will be created. The original tst is used on the device to identify the waypoint, this allows the desc to be changed.

MQTT Topic: owntracks/user/device/cmd

{
  "_type": "cmd",
  "action": "setWaypoints",
  "waypoints": {
    "_type": "waypoints",
    "waypoints": [{
      "tst": 1681734317,
      "_type": "waypoint",
      "rad": 16,
      "lat": 33.814498,
      "lon": -117.9239239,
      "desc": "Millenium Falcon"
    }]
  }
}

The important items to change:

• tst: if you are republishing the waypoint to update it, make sure to use the original timestamp. Otherwise, a new waypoint will be created.
• desc: the waypoint name. Can be changed when re-published.
• rad: region around the waypont to be monitored. Can be changed when re-published.
• lat: latitude of the waypoint. Can be changed when re-published.
• lon: longitude of the waypoint. Can be changed when re-published.

mosquitto_pub example:

mosquitto_pub -h 100.66.34.128 \
-t owntracks/princess_leia/iPhone/cmd \
# JSON payload
-m {"_type":"cmd","action":"setWaypoints","waypoints":{"_type":"waypoints","waypoints":[{"_type":"waypoint","rad":13,"lat": 33.814498,"lon": -117.9239239,"tst": 1681734317,"desc":"Millenium Falcon"}]}} \
-u user_name \
-P user_password \
-d \  # indicates verbose debugging output
# there is `-r` flag to retain the message

Security & Encryption

MQTT can be protected with TLS encryption or the borker can be run behind a VPN. Personally, I use Tailscale as it's the simplest VPN solution that works-out-of-the-box and doesn't require a deidicated server to operate.

Permissions on, and access to the MQTT broker can be adjusted by editing the mosqitto.conf file. For complete details read the documentation here. Those needing in more complex security should look into the dynamic security plugin.

OwnTracks payload encryption

Security concerns can be mitigated by encrypting the payload. OwnTracks can optionally use libsodium to create secret-key encypted payloads. Encryption can be used to secure reports between two or more users. So, reports from a wallet on owntracks/tags/wallet could only be read by Joe's iPhone owntracks/joe/iPhone if they have previously shared the secret key. Anyone else (or, another device like say, Joe's tablet) with the key will also have access to the reports. This encryption system can be used throughout the entire OwnTracks setup.

[23tom2323]

Hello Richard May,

OwnTags + Haystack + MQTT = OMG ;-)
Exactly what I was looking for

A great project. I want to use it in io Broker.

But! I still have a small problem with the request_reports.py script in Line18.

./downtags.sh
Traceback (most recent call last):
File "/Users/USER/Downloads/OwnTracks/owntags-main/application/request_reports.py", line 18, in
OUTPUT_FOLDER = configuration["owntag_options"]["output_folder"]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
KeyError: 'output_folder'
-- END SCRIPT --

python3 request_reports.py --time 0:60 --owntags
Traceback (most recent call last):
File "/Users/USER//Downloads/OwnTracks/owntags-main/application/request_reports.py", line 18, in
OUTPUT_FOLDER = configuration["owntag_options"]["output_folder"]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
KeyError: 'output_folder'

My request_reports.py line 18:
OUTPUT_FOLDER = configuration["owntag_options"]["output_folder"]

Do I have to change something in the script?

Best regards and keep it up!
Tom

[mrmay-dev]

Hello! It's been a bit since I've played around with this. I'm going to dust it off and take a closer look.

[shiprec]

@mrmay-dev are you seeing issues getting updates from devices that upgraded to IOS17?