How to make use of bluetooth SERVICEDATA an MANUFACTURERDATA ? #1372
Comments
|
Welcome to the world of reverse engineering BLE data ;-) The manufacturer data starting with 4c00 are using the Apple Continuity protocol. See https://github.com/furiousMAC/continuity for some documentation. The manufacturer data starting with 6909 are from some Chinese company. That could need some reverse engineering. Do you have an idea which device it is? If it's a sensor, you could read its data on its display or in its app and compare with the manufacturer data. This request is probably more something for the decoder repository. |
|
i dont want to re-invent the wheel, ESPRESENSE makes use of the manufacturer data for fingerprinting (helps with changing BT MAC), examples for company decoding on espresence: examples for company decoding (and more) in theengs (?) so it could be a nice and first simple step to show/decode the manufacturer for all my BT devices wit OMG. How to start with simpler decoding ? |
|
Do you just want to see some more information about the manufacturer data or service data? I actually wrote a tool to help with that: HumBLE Explorer. Maybe try that for a start to explore what your devices are advertising? |
|
hey, great. have lots of numbers in "manufacterdata"..
|
|
These numbers (4c00 for Apple and so on) are standardized. They're called the Bluetooth company IDs. There's a whole database of it. |
Yes, this is the description page for adding new decoders to Theengs Decoders, which OpenMQTTGateway also uses for BLE decoding. I have created a simple Apple HomePod mini device only decoder for a while now, which you could also try and test at https://github.com/DigiH/decoder/tree/homepod-mini but until white/black-listing is possible by the model property we didn't really find it useful to integrate such model only decoders into the Decoder release yet, but might be planning such model decoders for the future with w/b-listing functionality. You also have to remember that most modern devices, which also means all the Apple devices you might have, do change their MAC address about every 15 minutes for privacy reasons - unless it is being received by another Apple device which is logged into the same iCloud account - not really applicable to an ESP32 running OpenMQTTGateway though ;) plus there is also the uuid vs. MAC address issue on receiving Apple devices, when running Theengs Gateway for example. So with your above long list it could well be that several entries there are duplicates of the same device, but with changing MAC addresses over time, and depending on how long you had MQTT Explorer running. Apart from such awkward model/manufacturer only BLE broadcasts, if you feel you also have devices which can/should be fully decoded with useful information, please log an issue for these so we can work together on integrating them into Theengs Decoder. |
|
@ozett, are you able to build your OMG version with Platformio? If so I could create some test model decoders for the Nearby and FindMy messages you are seeing from your Apple devices, to see how the result would fit what you are expecting. |
yes, it tried it before for the das18b20 sensor and dht22,
i think i try this as a fist step. |
All that is required is to change the decoder URL from
to in platformio.ini - rebuild and upload. No coding, no regex, no jumping through hoops … only NEARBY fully implemented so far, in case you want to try it out.
The database is not so much of an issue, apart from possibly squeezing the whole databse into OMG, and therefore onto memory restricted ESP32s, when it would be only for getting a few company IDs for some undecoded devices out of the whole database., without having any further information or value becuase of the ever changing MAC addresses. More important would also be the fact that there are many non-company ID compliant devices around in the wild. Some of them using the first two bytes for broadcasting temperature, humidity or such. How would you try and filter such devices when for example a device would send a leading 4c00 because it has a humidity of 76%, but it would then be falsely recognised as an Apple device. Then when the humidity changes over time it might go through many other companies as it's reported ID. Seemingly nice company reporting, but on closer examination just false information. Unfortunately this is not hypothetical, but has been observed with many devices and applications using such an indiscriminate company ID database lookup. |
|
trying to re-build the apple-findmy decoder with the node-red switch-node. why compile and build into the receiver-device?
one could add more filter like brand or existing decoded topics, |
|
I've fully implemented the NEARBY and FINDMY options now, along with some device recognition as well, for iPhone, HomePod mini and Apple TV on my side. Just doing a CLEAN ALL, CLEAN and BUILD again with the Decoder URL staying the same will give you this currently final functionality now for testing, where your above circled 4c001006… should be decoded as NEARBY - Apple TV |
|
the filter from your source and from espresence should easy be done in node-red syntax as a match to the first-data of manufacterer data with the common switch-node. right?
|
Correct, plus the additional byte following which I have partially implemented as device type property recognition in the latest decoder update. In the end though, depending really on what you want to achieve with node-red, it might be really easier to apply a white-list to your OpenMQTTGateway, to only receive the messages from devices you want to process, instead of having to filter out all the unwanted messages. |
|
a white-list would be helpful, if one could rely on the MAC. so i go on slowly to identify my house-devices and see what they are sending. |
|
can be done...
Node-RED flow
[
{
"id": "2ab459264ff28926",
"type": "mqtt in",
"z": "e53ac3e112912e4f",
"name": "",
"topic": "omg/#",
"qos": "2",
"datatype": "json",
"broker": "9b50c12.74a3a4",
"nl": false,
"rap": true,
"rh": 0,
"inputs": 0,
"x": 90,
"y": 160,
"wires": [
[
"d8ef141c72ca2a94",
"deea14af2a8eeef1"
]
]
},
{
"id": "4ff83f50b90f8ce5",
"type": "debug",
"z": "e53ac3e112912e4f",
"name": "omg.all",
"active": false,
"tosidebar": true,
"console": false,
"tostatus": true,
"complete": "true",
"targetType": "full",
"statusVal": "payload",
"statusType": "auto",
"x": 480,
"y": 100,
"wires": []
},
{
"id": "d4274130de74fca1",
"type": "switch",
"z": "e53ac3e112912e4f",
"name": "^4c0010=nearby",
"property": "payload.manufacturerdata",
"propertyType": "msg",
"rules": [
{
"t": "regex",
"v": "^4c0010",
"vt": "str",
"case": false
}
],
"checkall": "true",
"repair": false,
"outputs": 1,
"x": 340,
"y": 180,
"wires": [
[
"29deb2e4babef159"
]
]
},
{
"id": "29deb2e4babef159",
"type": "debug",
"z": "e53ac3e112912e4f",
"name": "apple nearby",
"active": false,
"tosidebar": true,
"console": false,
"tostatus": true,
"complete": "true",
"targetType": "full",
"statusVal": "msg.payload.manufacturerdata",
"statusType": "jsonata",
"x": 670,
"y": 180,
"wires": []
},
{
"id": "b0ad02be022d5ebe",
"type": "debug",
"z": "e53ac3e112912e4f",
"name": "ip",
"active": false,
"tosidebar": true,
"console": false,
"tostatus": true,
"complete": "true",
"targetType": "full",
"statusVal": "msg.payload.ip",
"statusType": "jsonata",
"x": 310,
"y": 100,
"wires": []
},
{
"id": "3005a8e77d1e777e",
"type": "switch",
"z": "e53ac3e112912e4f",
"name": "^4c0012=findmy",
"property": "payload.manufacturerdata",
"propertyType": "msg",
"rules": [
{
"t": "regex",
"v": "^4c0012",
"vt": "str",
"case": false
}
],
"checkall": "true",
"repair": false,
"outputs": 1,
"x": 340,
"y": 240,
"wires": [
[
"5799033e9acfa529"
]
]
},
{
"id": "5799033e9acfa529",
"type": "debug",
"z": "e53ac3e112912e4f",
"name": "apple findmy",
"active": false,
"tosidebar": true,
"console": false,
"tostatus": true,
"complete": "true",
"targetType": "full",
"statusVal": "msg.payload.manufacturerdata",
"statusType": "jsonata",
"x": 670,
"y": 240,
"wires": []
},
{
"id": "05366fba78f6bdef",
"type": "switch",
"z": "e53ac3e112912e4f",
"name": "2=iphone,5=HomePodMini,6=A-TV",
"property": "payload.manufacturerdata",
"propertyType": "msg",
"rules": [
{
"t": "regex",
"v": "^4c001[0,2]02",
"vt": "str",
"case": true
},
{
"t": "eq",
"v": "^4c001[0,2]05",
"vt": "str"
},
{
"t": "eq",
"v": "^4c001[0,2]06",
"vt": "str"
}
],
"checkall": "true",
"repair": false,
"outputs": 3,
"x": 400,
"y": 300,
"wires": [
[
"9cb9346e23b0dbec"
],
[
"9cb9346e23b0dbec"
],
[
"9cb9346e23b0dbec"
]
]
},
{
"id": "9cb9346e23b0dbec",
"type": "debug",
"z": "e53ac3e112912e4f",
"name": "apple devices",
"active": true,
"tosidebar": true,
"console": false,
"tostatus": true,
"complete": "true",
"targetType": "full",
"statusVal": "msg.payload.manufacturerdata",
"statusType": "jsonata",
"x": 680,
"y": 300,
"wires": []
},
{
"id": "d8ef141c72ca2a94",
"type": "junction",
"z": "e53ac3e112912e4f",
"x": 200,
"y": 240,
"wires": [
[
"d4274130de74fca1",
"3005a8e77d1e777e",
"05366fba78f6bdef"
]
]
},
{
"id": "deea14af2a8eeef1",
"type": "junction",
"z": "e53ac3e112912e4f",
"x": 200,
"y": 100,
"wires": [
[
"4ff83f50b90f8ce5",
"b0ad02be022d5ebe"
]
]
},
{
"id": "9b50c12.74a3a4",
"type": "mqtt-broker",
"name": "192.168.14.204-hik",
"broker": "192.168.14.204",
"port": "1883",
"clientid": "",
"autoConnect": true,
"usetls": false,
"protocolVersion": "4",
"keepalive": "60",
"cleansession": true,
"birthTopic": "",
"birthQos": "0",
"birthPayload": "",
"birthMsg": {},
"closeTopic": "",
"closePayload": "",
"closeMsg": {},
"willTopic": "",
"willQos": "0",
"willPayload": "",
"willMsg": {},
"sessionExpiry": ""
}
]
|
I meant a white-list for other devices you have which do not change their MAC addresses and in which you might be interested in. From your above partial MQTT Explorer list it looks like you have several other BLE devices possible sending useful information. Unless you are interested in these Apple device with their changing MACs, to allow for presence detection or something similar. Unless you are in a very remote area though, and only have one iPhone, Apple Watch etc. and no close neighbours, and additionally also implement the identity resolving key for good device recognition, I can tell you that I very quickly abandoned trying to use any Apple device for such detection myself, and went for a small cheap BLE beacon on my key chain and/or a Mi Band on the arm for easy and very reliable beacon/person presence/location detection. |
|
seems i found an apple device with device ID 07. maybe a pencil?
|
|
Thanks @ozett, so from the list above you seem to have an iPhone, Apple TV, and 07, which might be the Apple Pencil. I'll add it with a current ? until we can confirm the device assignment. |
can i do some other easy tests on that 07-device for name or the like? found som e 4c00-0a and 4c00-07 types. some services other than nearby or findmy?
|
|
I'm still unsure about the device IDs for the FINDMY broadcasts. Looking at my devices I think it might be the last byte instead. Will have to monitor some more.
0a - https://github.com/furiousMAC/continuity/blob/master/messages/airplay_source.md … from your linked repo above. In the end the implementation would likely only be for recognising the services by model_id, and to be able to black-list these, as with the changing MAC addresses a clear device distinction will not be possible. |
|
the air-pod thing is interesting, also battery-level and lid-open should be detecable from the soure of furiousMAC. edit: also more information from nearby recoverable: https://github.com/furiousMAC/continuity/blob/master/messages/nearby_info.md |
|
If you want to test
you should not see any undecoded "4c00…" messages any longer. |
|
did you put decoding of the action-states and status-flags of the phones in?
|
|
No I haven't implemented them yet, ;) something to do tomorrow. Just wanted to get all the protocols done today. The Phone or FaceTime Call interests me the most. Will have to see if it is only true during a call, or/and also when a call is coming in. The latter would be great to have incoming call/FaceTime announcements in the controller. |
|
How do you approach getting the "bit-states" out of the data-string? not in your code yet (?), but looks complicated to my eyes, while vivid explained here similary for theengs-decoder
will try to decode the char-string in node-red, but i dont have a clue how to access bits in the string-values. found some hints to convert it to buffer (one, two), but on my first tries the decoded bit-value comes from the character-value, not from the bits.. maybe i pre-calculate all values for the 3rd byte,
edit:
|
For Decoder the bit states can be evaluated in the condition, as you posted the description above. These two condition and decoder bit functionalities allow you to fully decode single bits or multiple bit combinations. I have implemented the phone/facetime call functionality separately in https://github.com/DigiH/decoder/tree/apple-call currently with redundant model and property conditions, and it nicely works as soon as a call is coming in and the the device is ringing, as long as the call is active, and also when initiating a call. If and how there might a differentiation in the remaining data between incoming and outgoing calls would be interesting to examine. This also throws up the differentiation issue with multiple iPhones/iCloud accounts and related privacy issue of being able to monitor call times and durations of any such devices in the BLE receiving vicinity. |
|
sounds great, the phone-decoder. for the string-values in the manufacturer-data: at the momentet i think these are always two bytes for building the original hex-value. that must be decoded back into integer/bit values. seems dooable, even if i am a no-coder. for the identification of devices from that above post |
Often-times, but not always, as also seen in your pasted graphic above with the single-hex 4-bit Status Flags and Action Code.
Agreed, and already taken out yesterday ;) |
|
gets harder, it the manufacterer-data string is somehow dynamic and does not follow the reverse-engeneered scheme. https://docs.openmqttgateway.com/use/ble.html#advanced-setting-up-an-external-decoder |
|
OMG already publishes the raw manufactererdata if no decoder has been matched (https://docs.openmqttgateway.com/use/ble.html#advanced-publishing-unknown-manufacturer-s-data), or can also send the raw manufactererdata additionally along with any decoded data, if so desired and set to true https://docs.openmqttgateway.com/use/ble.html#advanced-publishing-known-manufacturer-s-data If always no decoding on the ESP32 gateway is wanted, but through some external decoder, your above link is the way to go, with Theengs Gateway nicely tying into this, to be able to receive undecoded raw data from one or several ESP32 getaways to then be decoded by a central Theengs Gateway. In general the whole BLE gateway usage documentation is recommended for any OMG BLE gateway user to find out about all the possible options for get the most out of the getaways. |
but that raw-data seems to be encoded as chars (not numbers, or numbers as chars) in a JSON-object. |
to my limit understanding the 2x4bit=8bit=byte is always send as 2-chars hex-value, this must be the encoder from the bt-utils that would make it easier to decode the manu-data from the chars back to hex/binary again |
|
i went over the message-block of the nearby-info and assume now, that the 4-bit action-nibble is originally encoded for the manu-data-string as a single character. also it is not stated in the naerby-document at a bitmask, what makes it easier to compare only for the listed numeric-values as the given (phone)-states. means: substring-comparison of the 9th character is all it takes edit: much to easy.
|
Exactly how I stated it above and used in my apple-call branch :)
Yes, including the above decoder branch in a continuous BLE OpenMQTTGateway build, it immediately triggers as soon as the iPhone rings and is active until hanging up the call. |
|
this is all great news. another thing: read also something else, so maybe this works in another way: a pass is coming up, if the phone reads an ibeacon? make .pkpass: https://www.passsource.com/info/demo.php |
|
is this a call? think i now also have to compile your decoder .. 😄 |
Looks like a call to me, yes |
|
i am still in doubt, if the meaning of data-fields in the nearby-structure is still valid. as the manufacter-id is reversed (endiness or something?) in the original BT-Advertisment,
all the papers deciphered max IOS12, and now IOS16 is out. |
|
I'm afraid I don't fully follow your concerns and confusions. Yes, the company ID 004c is with reversed endianness leading the manufacturerdata in OMG as are all the other two octet company IDs, but all the following octets - and best to make a differentiation here between the ambiguous byte (which can be a 4-bit byte or 8-bit byte, or 16, 32 …) and octets - are in the correct order as sent out by the devices, as being published my OMG and as being used by Theengs Decoder to decode the encoded information. Apart from some possible slightly confusing type naming conventions all the info on the Continuity Protocol pages do make perfect sense to me, at least for the parts I was looking at and playing with, while always keeping in mind that this is undocumented reverse engineered info, which might have changed in parts since their release. Without any distinct device identification though I don't see much value in pursuing decoding and integrating any of the available data, save for being able to identify and eventually blacklist any of these Continuity broadcasts to reduce OMG MQTT noise. Would you mind sharing what your goal is to achieve with these continuity message decoding? |
of course, with pleasure. i have a bunch of apple-devices in the house and wanted to see if i can identify some of them by their broadcasted activity. asuuming that all devices are not phones, only some of them, i thought of using theengs-decoder, but that decoding is done the same way i build it with node-red. to investigate these possibly to many "phone-calls", i thought about a differential diagnosis of device-states. every holiday-morning i tried another simple idea...without great success right now. as a side-effect i sorted out all other bt-broadcasting devices in the house. |
|
Thanks for the clarification, and yes, I tested the same possibilities of trying to identify my Apple devices and their states for possible integration into my home automation controller. With it currently no being possible to get clear fingerprinting though I did put the endeavour aside for the time being, only at one stage being able to tell my neighbour exactly when his last phone call on his iPhone started and how long it lasted down to the second ;)
Unknown company identifiers can be looked up on the officail Bluetooth SIG pages, but you always have to keep in mind that not all companies do apply and register for their own unique company identifier and so the assumed company ID might be misleading or in some cases actually two octets containing ever changing encoded device information. Theengs Decoder has a separate key for this. If you feel that some of your other BLE braodcasting devices might benefit from an integration into Theengs Decoder free to open a discussion on the Decoder pages for possible decoding and integration. |
|
thanks for reply, that will encourage me to prepare an esp32 with OMG and your decoder. |
edit: your suggested change leads to an error... How do i set this decoder only for my special environment?
edit3 😄😄😄 :
|
|
i discovered that i used the config for my previous board,
edit: maybe like this? edit: 🥇 |
|
Huh, some more information about two of my devices? at least the manufacturer?
|
|
another riddle solved: more google with lots of zeros at my home:
|
|
some fun coming up now.
|
|
Closing this, feel free to reopen if needed |
Is your feature request related to a problem? Please describe.
OMG is sending lots of manu-data and servicedata from devices around my house.
i guess, that can be decoded
dont know if i can decode this mayself in node-red
or somehow like in theengs-decoder already for some devices
Describe the solution you'd like
some hints, how to make use of that captured data.
some hints how to start decoding the service-data and manufutererdata.
it is done for some specific devices (in the source) and i would like try to add some more from my BT announcements
Describe alternatives you've considered
wait that somehow a decoder is built into OMG and data is send in cleartext
Additional context
captures, but what is it?
The text was updated successfully, but these errors were encountered: