Stability improvements

[rospogrigio]

Let's use this post to discuss ways to improve the stability of the connection: use a different firmware, rebuild tasmota, etc.

[Adrian-at-CrimsonAzure]

Reposting this here as I think it better fits this repo than the other one:

Has anyone identified the chip on the physical device? I have the M300 Bluetooth version and the chip has a similar pinout to an ESP8266, but I'm not good enough at reverse engineering to be sure. The top of the chip is etched away just enough to make it impossible to make out the original number; at the right angles you can make out some lines, but not enough to form full letters.

[formatBCE]

Reposting this here as I think it better fits this repo than the other one:

Has anyone identified the chip on the physical device? I have the M300 Bluetooth version and the chip has a similar pinout to an ESP8266, but I'm not good enough at reverse engineering to be sure. The top of the chip is etched away just enough to make it impossible to make out the original number; at the right angles you can make out some lines, but not enough to form full letters.

I don't believe someone here was tearing device down
I got same lock.
However, ESP8266 doesn't have BT, afaik. Instead there is WiFi.
Do you think there's ESP32 inside as well?

[Adrian-at-CrimsonAzure]

Main board of M300:

There is a sub-board I was too lazy to get to that all the connectors go to (except bottom right, which goes to the DC motor) but that board seems to just connect the batteries and provide some pre-regulation for the mainboard. The little unpopulated header seems to connect to power and three other pins, probably RX/TX/GPIO0 like most Tuya products. Haven't hooked up anything to it yet, might finally be my excuse to get a logic analyzer...

Completely slipped my mind that the ESP8266 doesn't have Bluetooth, I've been working with the ESP32 too much lately. This has an 8x8 layout where as the ESP32 has a 16x16, so no chance of it being an ESP.

[formatBCE]

@rospogrigio today I had spare time to dig into ESP32 BLE stack.
I managed to post advertisement to MQTT, it works very well.
Also, the topics to use will be more obvious, no need to create rules, and I believe we will get possibility for integration to send both messages at once, instead of waiting for first chunk to write.
It will get your code much shorter and easier to maintain.

However, now I'm stuck with sending data to lock - for some reason, it does stuck, when ESP32 tries to initiate BLE client for connection to characteristic.
Hopefully, I will find the solution for this. Other things are working already.
(Well, I will have to find a way for configuring ESP on-the-fly also, but it's just matter of writing, I believe).

[formatBCE]

@rospogrigio
So for now, I got the way to send commands and scan at same time.
However, sending (basically, connecting to lock) is giving me hard times.

I'm getting this error:
lld_pdu_get_tx_flush_nb HCI packet count mismatch (0, 1)

It happens on connection, and what's the worse, is that after that ESP becomes unresponsive, I cannot even reconnect with serial monitor.
Could be that lock is too far (it is actually far), or power is not enough (it's PC USB all-in-all), but it should at least reboot, but it just hangs.

Didn't find any reliable information on this yet.
Gonna keep digging. Any help appreciated.

[rospogrigio]

Sorry but I really don't have any expertise on this, wish I could help...
As a side note, I'm working on adding a configurable option to set a desired number of automatic retries in case of a FAILCONNECT event, should be ready very soon.
Keep on digging, I'm sure you can make it!

[formatBCE]

So what I have now:

• managed to connect to the lock, and get to service/characteristic
• changed @rospogrigio's integration to work with new topics
• removed unnecessary code from integration, as now we can send both data chunks in one JSON
• managed to bring lock alive and make it sending lock/unlock commands to MQTT
• tried to write that two chunks consequently to characteristic

Here I stuck. It says "true", basically reports success from write operation. But lock doesn't respond to commands. Either data format is incorrect (I'm sending from *ptr, so it might be the problem, or maybe I have to convert that from string before sending, don't know), or write operation lies to me (unlikely, 'cause characteristic value seems to be changing).
If someone has any clues, what to do or how to debug, welcome.

Next steps (after getting it working, of course):

• unify/rearrange topics system
• make use of telemetry messages (might be useful for checking gateway status)
• make UI for initial setup
• think on removing some parameters, that were used because of Tasmota general purpose, but unnecessary now with strict gateway binding
• think on retries (it's easier to do on gateway, than in integration)

To the last point: gateway is REALLY powerful and stable. It reboots itself in the matter of seconds in case of failure, and can connect to lock (almost) without failures from like 7-8 meters and 1 wall. That's what I was expecting.

[formatBCE]

Yes, I believe, writing string itself was strange solution :)

Will try to convert it back to bytes (I believe, there's 20 bytes in each of two commands), and send that.

Also, I have some doubts on current integration status determining logic.
Although mqtt success received, it's still in operating status. That's fine for now, as it doesn't work, but I guess there should be a way to integrate closer, instead of waiting for changed adv message.

[rospogrigio]

I can help you with this: the FFF3 characteristic (you can read it, right? or at least you can use nRF Connect for this) has some status bytes (byte 5, in detail) that provide a error code in case of failure. This is how I understood that I was sending the wrong payloads, too.
Look at this function:

    public static String resultString(byte[] bArr2) {
        String str = "";
        if (bArr2[0] == -86) {
            switch (bArr2[5]) {
                case 0:
                    str = "Success";
                    if (bArr2[3] == 2) {
                        if (bArr2[4] != 1) {
                            if (bArr2[4] == 2) {
                                str = str + ("  Device time：" + ((long) PackMaker.byte4ToInt(bArr2, 6)));
                                break;
                            } else {
                                byte b = bArr2[4];
                                break;
                            }
                        } else {
                            str = str + ("  Service time：" + ((long) PackMaker.byte4ToInt(bArr2, 6)));
                            break;
                        }
                    }
                    break;
                case 1:
                    str = "Fail";
                    break;
                case 2:
                    str = "Invalid role type";
                    break;
                case 3:
                    str = "Invalid operation type";
                    break;
                case 4:
                    str = "Invalid opcode";
                    break;
                case 5:
                    str = "No operation authority";
                    break;
                case 6:
                    str = "Invalid signature";
                    break;
                case 7:
                    str = "Serial number expired";
                    break;
                case 8:
                    str = "Out of check-in time";
                    break;
                case 9:
                    str = "Service time expired";
                    break;
                case 10:
                    str = "Locked, cannot open the door";
                    break;
                case 11:
                    str = "Not initialized";
                    break;
                default:
                    str = "";
                    break;
            }
        }
        String str2 = "" + ((int) bArr2[5]);
        return str;
    }

Hope this might help, bye...

[rospogrigio]

To the last point: gateway is REALLY powerful and stable. It reboots itself in the matter of seconds in case of failure, and can connect to lock (almost) without failures from like 7-8 meters and 1 wall. That's what I was expecting.

This is REALLY impressive!! Can't wait to see it in action!! 😮

[formatBCE]

I MADE IT!

Opening/closing works. Status update is not so well yet, there's gap to fill.
@rospogrigio here's the situation, please give me advice:

• scan for devices (which is actually getting lock adv) has scan time and wait time. Basically, it's active scanning window, and interval between active scans;
• the bigger active scan is, the bigger is probability to get adv data from lock. Now i'm scanning for 4 seconds, and waiting for 1 second in between. Works fairly good.
• however, when in active scan, i cannot send commands. ESP has to wait for scan end to connect to device. So maximum gap between getting the command and writing it can be up to 4 seconds now.
• moreover, after sending i'm launching scan again. And first adv from lock will be ready after next 4 seconds (or more, if it won't find lock on first scan, which can be affected with distance to lock, for example).
• also, integration doesn't seem to have good support for my topics here - i don't know what to do with status, while we're waiting for lock adv (although we already got WRITE SUCCESS from gateway. I believe, we can trust that success message (there will be an error for every trouble posted).

So what i want you to think:

• with this gateway, your integration can theoretically get any data you need. Also some calculations, i believe, could be done on gateway too. Let's think, what you want to change in this interaction.
• based on this - can we think on better status support in integration? it's a bit clunky now, which, i believe, was because of restrictions of Tasmota BLE stack. Now we can optimize that.
• what can we do for scan gap? I could try to decrease scan time right after command sending, i guess. But it won't eliminate that 4-second gap, if scan just started and we got the command to run. Are we ok with this gap?
• maybe, you have some other brilliant ideas to help improving this.

Meanwhile, i will focus on initial gateway configuration.

[formatBCE]

New updates:
I managed to make scanning stop right on command received. Now it responds to commands almost instantly.

[rospogrigio]

Cool, congratulations! So, my few first thoughts after reading this:

• 4 secs seems quite much, I believe we can find a way to have a bit more responsivity
• we actually don't need to scan for the advert: all the info we need from there (lock status, lock events and battery) can be read from FFF3 characteristic
• I would not do calculations on the gateway, it's easier to debug if we do them in the integration

[rospogrigio]

Oops, I just read your last message... Great!!
When can you share your code and firmware so I can try it?

[formatBCE]

I guess it's about the time.
Don't want to get interrupted by something and leave you without anything in hands :)

Let me create PR for your integration, and repo for gateway.
Actually, without Arduino IDE you won't be able to run it - all stuff like topics, WiFi parameters and MQTT parameters are hard-coded now. I'm trying to get them into some initial setup.
But at least we will have some code in the cloud

[rospogrigio]

Cool! Regarding whether to get the values from advert or FFF3, I seem to remember that nourmehdi wrote that FFF3 was more power consuming, so we'd probably better use the advert since you found a way to interrupt it on command.
Can't wait to see the code! 😉

[formatBCE]

Here's PR: #7
If you prefer another branch, just let me know.

[formatBCE]

And here https://github.com/formatBCE/Airbnk-MQTTOpenGateway is the repository.
Change things in Settings.h, build and upload.
I use VS code with Platformio plugin.

[rospogrigio]

Super super cool! Tomorrow I'll try to find some time to play with it.
Could it be possible to set the options in some other way instead of rebuilding it maybe?

[formatBCE]

Could it be possible to set the options in some other way instead of rebuilding it maybe?

Yes, i'm thinking on making WiFi access point with simple web page for initial setup, and when submitted, it will save prefs, reboot and connect to actual WiFi.
I never did it before, so it will take some time. But it's doable.

[formatBCE]

@rospogrigio
Ok, i uploaded new version to https://github.com/formatBCE/Airbnk-MQTTOpenGateway
You may find built binary for esp32 in repo root (firmware.bin).

Flash it, it will start access point AirbnkOpenGateway.
Connect to that AP, and go to 192.168.4.1 in browser.

Fill-in the data there. It will create configuration, and reboot ESP32 to connect to your WiFi.
Chip will indicate (most probably) with blue LED, when done. Also you may find messages in MQTT explorer. IP will be there, in "tele" subtopic.

Check it out, and tell me what's there.

If you screw up with config, and it won't connect - after some attempts it will reset config and re-deploy AP.
Same config reset you can perform from web UI, by navigating to ESP IP address, given by your router.

Cheers.

[formatBCE]

Ok, I guess tomorrow I will change behavior a bit.
Because rebooting my HA host (with mqtt add-on) was enough to bring gateway to reset :)
Will make it more patient to missing mqtt connection.

[formatBCE]

Did it, now it will erase prefs only on wifi disconnect.

[rospogrigio]

Ok I have some ideas for the code merging, I'll take care of that so you can concentrate in improving the firmware if you believe there's more work to do there. Good job! 👍

[rospogrigio]

@formatBCE how do I flash the firmware? Can I use Tasmota interface, as far as you know?
Edit: I also merged your code, creating a separate class for the new gateway but still keeping the older Tasmota device. I also moved the code generation in a dedicated class. Please see PR #9 , I'd suggest to work from there from now on. Let me know!

[rospogrigio]

OK, flashed, configured and launched HA but all entities are unavailable and I see nothing happening... I might have broken something, how can I debug the communication??
Edit: OK fixed almost everything, I have also managed to operate the lock once, even though it took more than 5 secs to operate. Moreover, after I operate it all entities become unavailable and I no longer receive messages so I have to reset the ESP, is it normal?

[formatBCE]

Hi!

Nope, it's not normal. For me it works all the time - yesterday i tried like 50 times opening and closing, and got no errors.
I implemented retry inside of the code, so it tries for 4 times to connect and send. It helped to reduce denial cases almost to zero.
You may debug in different ways. First - what you see in logs to your integration? they should have some info.
Also, you may download MQTT Explorer and check messages receiving in your root topic there to see exact messages.
Also, you can connect ESP to your PC USB, and use Putty to connect to serial port, on which ESP is on (baud rate 115200) and check logs from ESP itself.

I just woke, and realized that my ESP disconnected from WiFi 1 hour ago. I think i will make some remote logging to trace the problem. WiFi/Mqtt part for this project i took from another sketch, so maybe worth optimizing.

Thank you for merge, gonna try that today.

[rospogrigio]

Ok I did some tests and I confirm it works like 100% of the times, and from a bigger distance (like 3-4m, inside a wooden cabinet). With Tasmota I was having 50% of failures from 1m in open air... Awesome job!!!
I still have some strange status changes so I agree it's improvable. Also, I had set everything up with Tasmota to allow a gateway to connect to multiple locks (ok, I know it's a very unlikely event so I guess we can keep it like this).
I'll test it for some days and will provide suggestions.
First one: maybe we can pass the MAC address and the number of retries as parameters from the integration? Think about it and let me know your opinion.
Thank you so much, I'm so proud of our achievement!!

[formatBCE]

Thank you for kind words :)

Yes it's doable (both multiple locks and sending parameters over mqtt).
If we really need this, i will implement :)

Downside to having multiple locks will be following:

• longer scan (now i'm restarting scan session right after we got adv from lock, to have it working faster. We'll have to wait for all locks - and, probably, even won't get some of them, so will be just waiting till scan timeout)
• longer operations time (first we'll need to wait longer on scan, then if there's already operation ongoing for one lock, we will have to wait again)
• a bit sophisticated logic of waiting/releasing between operations and scanning.

[rospogrigio]

I think you are still downloading the code from the master branch, but actually you need to download it from here: https://github.com/rospogrigio/airbnk_mqtt/tree/introducing_reload_service (Code -> Download ZIP)

Regarding the open direction, I don't understand what you wrote. Can you explain better? In detail?

1. using the app, are the open/close direction correct?
2. using the app, is the open/close status correct?
3. if the answer for any of the above is No, what happens if you change the direction in the settings?
   Let me know

[Corinake]

I think you are still downloading the code from the master branch, but actually you need to download it from here: https://github.com/rospogrigio/airbnk_mqtt/tree/introducing_reload_service (Code -> Download ZIP)

Regarding the open direction, I don't understand what you wrote. Can you explain better? In detail?

1. using the app, are the open/close direction correct?
2. using the app, is the open/close status correct?
3. if the answer for any of the above is No, what happens if you change the direction in the settings?
   Let me know

Hi,
I have same situation.
If i change direction in app, on HA it show close for open and open for close. In app all status and operation are correct.
To show correct in HA I change in const.py line 50 and 51 to:
LOCK_STATE_LOCKED: "Unlocked",
LOCK_STATE_UNLOCKED: "Locked",
and cover.py line 19 and 20 to:
LOCK_STATE_UNLOCKED: "hass:door-closed-lock",
LOCK_STATE_LOCKED: "hass:door-closed",

For me service restart work. I make a buton in lovelace and use to restart

[rospogrigio]

OK looks like we do need an option to force to reverse the status. @Corinake but does at least the operation follow what happens on the app? I mean, opening and closing as what happens in the app, and the possibility to reverse it? I need to understand if we also need an option to reverse the open/close command or only the lock status.
Let me know thank you.

[Corinake]

Only for reversed status. The commands in HA are good if the text and icon are changed.

[hometheater13]

Hi Rospogrigio
the reverse status needed when u enable in lock the autolock function without it the status is ok .

can u add this in next ver (reverse mode for status) now configure it manually like Corinake wrote before .

thx
Emil

[antonbek89]

I think you are still downloading the code from the master branch, but actually you need to download it from here: https://github.com/rospogrigio/airbnk_mqtt/tree/introducing_reload_service (Code -> Download ZIP)
Regarding the open direction, I don't understand what you wrote. Can you explain better? In detail?

1. using the app, are the open/close direction correct?
2. using the app, is the open/close status correct?
3. if the answer for any of the above is No, what happens if you change the direction in the settings?
   Let me know

Hi, I have same situation. If i change direction in app, on HA it show close for open and open for close. In app all status and operation are correct. To show correct in HA I change in const.py line 50 and 51 to: LOCK_STATE_LOCKED: "Unlocked", LOCK_STATE_UNLOCKED: "Locked", and cover.py line 19 and 20 to: LOCK_STATE_UNLOCKED: "hass:door-closed-lock", LOCK_STATE_LOCKED: "hass:door-closed",

For me service restart work. I make a buton in lovelace and use to restart

You can share please your button code? And give some explanation for what it's use?

I have same problem

[formatBCE]

@antonbek89 it's just button to call service airbnk-mqtt/reload, I guess.

[Corinake]

You can share please your button code? And give some explanation for what it's use?

I have same problem

show_name: true
show_icon: true
type: button
tap_action:
action: call-service
service: airbnk_mqtt.reload
data: {}
target: {}
entity: ''
show_state: true
name: 'Airbnk lock (MQTT-based): reload'

just press button to restart

you can read all releases what change was made
https://github.com/rospogrigio/airbnk_mqtt/releases

[giltabibian]

@formatBCE @rospogrigio
Hi guys, for the past month I'm using the custom integration. I have an automation to automatically lock the door once its closed.
It seems like from time to time the operation to lock is sent, but nothing happen and the door stays unlocked.
Have you seen this stability issues?
It can happen once a day.
With the cloud integration I didn't see this kind of problem. The lock automation worked every time.
Any ideas? The esp32 device is very close to the lock.

[formatBCE]

Hey @giltabibian

Do you use ESPHome version of gateway? Or completely custom firmware?

If custom, I'd suggest to move to ESPHome version - it's more stable, and easier to debug, since you can see wireless logs from it.
Then we will debug your issue.

[giltabibian]

Hi @formatBCE
I use the espHome gateway as you recommend this to before In other thread.

https://github.com/formatBCE/Airbnk-MQTTOpenGateway

[formatBCE]

Oh that's good.

Then you may debug wirelessly - gateway does log all activity. Just try to open log console in ESPHome, switch lock back and forth several times, and see if it shows sending/receiving.

ESPHome also does post all logs into dedicated MQTT topic, so you might want to check that topic too.

[giltabibian]

@formatBCE thanks for the quick response :)
i tried that and notice occasionally error like that:

[19:59:29][D][airbnk_mqtt:238]: Got command
[19:59:29][D][airbnk_mqtt:169]: Sending to:
[19:59:29][D][airbnk_mqtt:170]: e4:e1:12:c7:58:1b
[19:59:29][D][airbnk_mqtt:178]: Connected to lock.
INFO ble-node.local: Error while reading incoming messages: Error while reading data: [Errno 104] Connection reset by peer
INFO ble-node.local: Ping Failed: Error while reading data: [Errno 104] Connection reset by peer
INFO Disconnected from ESPHome API for ble-node.local
WARNING Disconnected from API
INFO Successfully connected to ble-node.local

do you have an idea what can cause?

[formatBCE]

Ok, it might happen, that gateway does lose connection to WiFi, while writing command to the lock.

It happens because of ESP32 nature: WiFi and Bluetooth using same antenna. And while scanning does leave enough room for WiFi to keep connection - command sending requires Bluetooth connection to lock, which takes full radio time.

In most cases, if lock is responsive, this won't hurt. But if there's more time needed to connect to lock, WiFi drops, and takes a while to reconnect after. In result, success message isn't transferred to HA integration, and it also comes as stucking in Operating mode.

You might connect gateway via USB to notebook, while staying closer to lock, and read logs from connected gateway via serial connection. It won't help with connection, but at least will help to debug.

In terms of what to do for fix:

• probably, your lock is too slow to respond. IDK what to do with it :(
• ESP32 might be underpowered, or poor quality. I'd try best ESP you can find in your drawer, and using decent power source.
• the best solution would be to use LAN-enabled ESP32, eliminating race conditions between WiFi and BLE...

[giltabibian]

@formatBCE thanks for the eleborated reply.
This is the esp I have:
HiLetgo ESP-WROOM-32 ESP32 ESP-32S Development Board 2.4GHz Dual-Mode WiFi + Bluetooth Dual Cores Microcontroller Processor Integrated with Antenna RF AMP Filter AP STA for Arduino IDE https://a.co/d/iWjw9IY

I don't think its a poor quality but I'm willing to buy a new one if you can recommend on LAN based it would be great.

Another note that might be valuable, I do see that that the automation sends the command to unlock, but the integration logbook doesn't note any command change for lock.

I'll try to look at the lock while this issue happen. It happened once a day, I'm not sure what exactly trigger the issue.
With the cloud integration, the same automation never failed to lock the door.
So I'm thinking it do related to the esp32, because the wifi gateway from the manufacturer worked evertime.
Will update if I have more information. Thanks! 😊

[formatBCE]

I have couple of these actually, and sometimes they give me problems, although they are not so bad.

I myself have WiFi version, and sometimes it lags for sure - but since I moved to my PR for this integration, it is much more stable. This PR eliminates need in confirmation from gateway - so is a bit lighter on WiFi/BLE stack.

What about LAN ESP, didn't work with them, but now thinking of buying one for tests. My gateway is actually on the NSPanel from SONOFF (which is mounted by the door), so I will stick with WiFi. But Ethernet should be tested.

[giltabibian]

@formatBCE
Ok, maybe I can use your PR.
What's the proper way to do so?

Also, appreciate I'd you can update about the LAN esp32 once you use it.
Thanks

[formatBCE]

I will prepare instructions (need to recall all changes I did)...

[giltabibian]

Great thanks!
BTW, why it's not merged to dev branch?

[formatBCE]

Question to @rospogrigio :)

[rospogrigio]

Question to @rospogrigio :)

Oooops, I just realized I have never merged this, and I have been using that PR for the last 4 months.
Shall I release it now @formatBCE or would it better to wait for the test on the new ADV sync command first and maybe release everything altogether?

[formatBCE]

I guess we wait. If that strategy works, we will eliminate major problem with interaction.

[giltabibian]

@formatBCE what is the process to use the PR?
i want to check if it can solve my issues.

[formatBCE]

@giltabibian in this case you just replace content of 1 file in your custom_components folder.
This file: https://github.com/rospogrigio/airbnk_mqtt/blob/76409cec140ae40987384aea0532ec64b9f9f81f/custom_components/airbnk_mqtt/custom_device.py

Then flash ESP with latest gateway code, and reboot Home Assistant.

[antonbek89]

Want to update
I using the PR and its really stable.

One issue I found ( not sure if its belong to PR) when the lock open manual the lock status doesn't changed, integration keep the last status

[formatBCE]

If you have same lock as @rospogrigio , it's lock trouble. Adv messages don't reflect manual operations...

[antonbek89]

If you have same lock as @rospogrigio , it's lock trouble. Adv messages don't reflect manual operations...

I have M531
What have to @rospogrigio ?

[formatBCE]

Yes, I believe it's same model.

Unfortunately, no solution at the time...

[giltabibian]

Same here. I also use M531 and the manual operation not advertise.