New PW firmware seems to break vitals - 23.44.0

[dscoffe39]

I like to begin by saying thank you for everyone's time, energy, and efforts toward this project; its been phenomenal to learn and use!

At 11:38pm last night I noticed that the pypowerwall was having authentication issues, I had to use the forget password function to reset the gateway password. Once I restarted everything I noticed that pypowerwall was no longer getting vitals and that the firmware version seems to be new.

output:

Device Version:
 23.44.0 eb113390

Device Vitals:
 None

generated by:

import pypowerwall

password='xxx'
email='xxx'
host = "xxx"
timezone = "America/Los_Angeles" 

pw = pypowerwall.Powerwall(host,password,email,timezone)

print("Device Version:\n %s\n" % pw.version())
print("Device Vitals:\n %s\n" % pw.vitals(True))

[jasonacox]

Thanks for opening this @ganeshtheju . If you can, please try some of these:

• If you are able to log in to the Gateway portal, can you provide a screenshot?
• Try getting raw vitals to see if this is a protobuf update: pw.poll('/api/devices/vitals')
• Try some of the other APIs to see if there are any other changes.

[dscoffe39]

raw vitals don't work either: (both from pypowerwall, or directly trying to hit the teg vitals endpoint [returns a 404])

`Device Version:
23.44.0 eb113390

Device Vitals:
None

pw.poll:
None
`

As far as i can tell, the pw charge/capacity/temps/reserve setpoint and alerts arent working, but everything else (meters etc) are working.

[dscoffe39]

It appears that there is atleast one other pypowerwall user who had gotten this update and is no longer getting vitals:
https://teslamotorsclub.com/tmc/posts/7965048/

[dscoffe39]

It looks like the teg webgui gets its reserve values from /api/operation; even logged in as an installer I can't find any endpoints with the pod values or the alerts.

[jasonacox]

@dscoffe39 Thanks! If the /api/device/vitals API is no longer available (404) in this firmware release, that would explain why the other functions (PW charge, etc) are also missing. Are you still seeing the power flow animation when you log in to the Gateway portal or use the Powerwall Dashboard?

[dscoffe39]

@jasonacox Yes the power flow still shows up on the teg website and the pypowerwall proxy root

[jasonacox]

Thanks, @dscoffe39

Unfortunately, I don't think there is a lot we are going to be able to do to recover the device vitals API or data after this Firmware upgrade. I think this is the first of many changes we will see. As you can see from the TEG portal UI and on the new Powerwall 3, it appears that the direction from Tesla is toward the "Tesla Pros" app to make it easier to install and manage for the majority of their customers. For data nerds like us, we are going to need to pivot and figure out new ways to get what we want. I'm hopeful there will be some easy/good way to accomplish that over next few months.

Request/Notice to the community:

• Help needed: If we were to lose the TEG Portal UI (which it seems likely), I would love to be able to reproduce the Power flow animation via the pypowerwall proxy. If anyone has the skill to reproduce or capture and download the HTML/JS/CSS code for it right now, I'm looking for that help.
• Help needed: What elements of the device vitals payload do you find most valuable? I would love to hear your prioritized list of what you would like to get if we could ask for it (e.g. PW full charge capacity, temps, frequencies, string data, voltages, etc) and if you have it, any story that demonstrates how having that data was helpful to you.
• Thanks to @mcbirse we now have a setup option to use the Tesla Cloud to pull basic metrics for Solar-only customers and the Powerwall 3 (which no longer has the TEG Portal UI) for the https://github.com/jasonacox/Powerwall-Dashboard project. I'm going to explore importing that code into pypowerwall to provide a bridge for if/when the local APIs disappear. Again, I'm hopeful that we will see alternative ways to get the local API data, even if it isn't the full set of data we get now with the vitals payload.

I know changes like this can be frustrating, but I also acknowledge that the Tesla teams need to optimize for the entire customer base and not just us. Still, I'm hopeful that we will be able to figure out ways to bridge most of what we enjoy with the rich data from our Powerwalls, even if we have some work to do to get there.

[dscoffe39]

@jasonacox Thanks for looking into this; I was afraid that eventually we would see changes like this; I think for full metrics we would need to tap into the can bus between the gw and the pws.

Using the tesla pro app its possible to see at-least the individual line voltages and frequencies of the grid/pws/home, but in all my digging there doesn't seem to be an exposed api to do it from the customer login; I'm not versed well enough to do packet captures and decode them (assuming its possible as they should be tls encrypted) or to reverse engineer the telsa pros app to see where its making the calls to retreve that data.

[jasonacox]

The CAN bus idea is interesting. Has anyone tried that?

On the Tesla Pros app reverse engineering, there is a start here jasonacox/Powerwall-Dashboard#392 but the downside is that it seems to require being on the TEG WiFi access point which, in my experience, is a bit finicky and prone to timeouts.

I'm still hopeful that there will be a simple and robust way (introduced or discovered) to get the API data we want.

[jasonacox]

Discussion here as well: jasonacox/Powerwall-Dashboard#402

[ibmaster]

Good luck team, hopefully the brain trust will figure out the proper way forward to keep our system data that we entitled to :)

[jasonacox]

The latest pypowerwall release, v0.7.3, adds the ability to use the Tesla Owners API (via TeslaPy) to pull basic information and some extended information (e.g. firmware version, Powerwall full charge capacity, etc.). This could be helpful for Powerwall 3 (and Solar only) owners that do not have local API access. For those with local access, the release also adds aggregate Powerwall nominal_full_pack_energy and nominal_energy_remaining values that can be helpful in determining the degradation of the Powerwall capacity over time. Many probably used this data in the past from the vitals data. It is aggregate (total for the site) instead of separate (vitals data had it per Powerwall).

See release notes starting with https://github.com/jasonacox/pypowerwall/releases/tag/v0.7.1. This doesn't restore the rich data set in the vitals API but it does provide some. We continue to be hopeful that we will see an opportunity to restore more of the vitals data set for those that have lost it.

[jasonacox]

@dscoffe39 and others with 23.44.0 - @BJReplay posted a reference to vloschiavo/powerwall2#55 in the Powerwall-Dashboard discussion related to this. Someone mentions getting vital details using a different authentication method.

Right now pypowerwall is using AuthCookie and UserRecord tokens in the header once login (user supplied user/pass) is performed via https://10.0.1.28/api/login/Basic. You can see it cache those in the .powerwall file. TTL is long for these (days even). But I also see the login api provides a Bearer token:

{
    "email": "[email protected]",
    "firstname": "Tesla",
    "lastname": "Energy",
    "roles": [
        "Home_Owner"
    ],
    "token": "8XXX8X888-XXXX8XXXXX8XXXXXXXXXXXXX_X8X8X8XXXXXXXXXXXXXXX8XX88XX_XXX8XXXXXX8XXXXXXXXXXX==",
    "provider": "Basic",
    "loginTime": "2024-01-02T17:36:36.205543924-08:00"
}

We could switch to using that as the Bearer token instead of the AuthCookie and UserRecord tokens in pypowerwall. I wonder if anyone who has 23.44.0 could try this:

import requests
from requests.packages.urllib3.exceptions import InsecureRequestWarning
requests.packages.urllib3.disable_warnings(InsecureRequestWarning)

# Replace with your Powerwall gateway address and login details
PASSWORD = "YourPassword"
HOST = "X.X.X.X"
EMAIL = "[email protected]"
TIMEZONE = "America/Los_Angeles"

def get_token(host, password, email, timezone):
    # Login and create a new session
    print("Attempting to login to Powerwall at %s" % host)
    url = "https://%s/api/login/Basic" % host
    pload = {"username":"customer","password":password,
        "email":email,"clientInfo":{"timezone":timezone}}
    try:
        r = requests.post(url,data = pload, verify=False)
        #print('login - %s' % r.text)
    except:
        print("Unable to connect to Powerwall")
        return None
    if r.status_code == 200:
        # Return the response
        data = r.json()
        if "token" in data:
            return data['token']
        else:
            print("Unable to find token in response")
            return None
    else:
        # Handle errors
        print(f"Error: {r.status_code}, {r.text}")
        return None

def get_api(host, endpoint, token, binary=False):
    # Get the API data
    url = "https://%s/%s" % (host, endpoint)
    print("Attempting to get data from Powerwall at %s" % url)
    headers = {
        "Authorization": "Bearer %s" % token,
    }
    try:
        r = requests.get(url, headers=headers, verify=False)
    except:
        print("Unable to connect to Powerwall")
        return None
    if r.status_code == 200:
        # Return the response
        if binary:
            return r.content
        return r.json()
    else:
        # Handle errors
        print(f"Error: {r.status_code}, {r.text}")
        return None

token = get_token(HOST, PASSWORD, EMAIL, TIMEZONE)

if token:
    # Some API Tests
    for endpoint in ["api/meters/aggregates", "api/system_status/soe"]:
        data = get_api(HOST, endpoint, token)
        if data:
            print("Data from %s:" % endpoint)
            print(data)
        else:
            print("ERROR Unable to get %s:" % endpoint)
        print("\n")
    # Vitals API Test
    endpoint = "api/devices/vitals"
    data = get_api(HOST, endpoint, token, binary=True)
    if data:
        print("Raw binary vitals data from %s:" % endpoint)
        print(data)
    else:
        print("ERROR Unable to get %s:" % endpoint)

# End of file

Now, if that works. We can switch to this method (possibly as an option). The logic inside pypowerwall watches for 403 and does a session refresh (would get a new token). It would extend to using the Bearer.

The code above works for me, but I'm still on Powerwall firmware 23.36.3 which still also allows access to /vitals through the AuthCookie so I don't know if the above would work for 23.44.0.

[FabienLavocat]

@jasonacox I am on 23.44.0 and your code works for the APIs api/meters/aggregates and api/system_status/soe but not api/devices/vitals (as discussed earlier in the thread, that endpoint seems to be gone...)

[jasonacox]

Thanks @FabienLavocat - I was worried about that, but it was worth a shot. I also don't think this will help the Powerwall 3 owners as that system doesn't seem to even allow a customer login.

I added the token option to pypowerwall and the proxy just in case it is useful for someone or needed in the future. It was an easy minor update (v0.7.4).

v0.7.4 - Bearer Token Auth

pyPowerwall Updates

• This release adds the ability to use a Bearer Token for Authentication for the local Powerwall gateway API calls. This is selectable by defining authmode='token' in the initialization. The default mode uses the existing AuthCookie and UserRecord method.

import pypowerwall

pw = pypowerwall.Powerwall(HOST, PASSWORD, EMAIL, TIMEZONE, authmode="token")

Proxy

• The above option is extended to the pyPowerwall Proxy via the envrionmental variable PW_AUTH_MODE set to cookie (default) or token.

[jasonacox]

Since firmware version 23.44.0 has eliminated /api/devices/vitals, there is a bug in the pypowerwall code that will treat this 404 like an authentication failure which can result in attempts to log in over and over, eventually hitting the rate limit. Fix in the works...

[jasonacox]

🔥 CRITICAL Upgrade - Please take a moment and upgrade your installation of pypowerwall if you using it for your own scripts. The latest release includes the critical patch I mention above that fixes 404 HTTP status code handling for API calls. Older versions of pypowerwall will cause a repeating loop of login attempts eventually resulting in rate limiting and failure to get any metrics.

NOTE: IF YOU HAVE FIRMWARE 23.44.0 YOU ARE IMPACTED AND SHOULD UPGRADE AS SOON AS POSSIBLE.

pip install --upgrade pypowerwall

Verify version 0.7.6 or greater:

python -m pypowerwall version

🟢 pyPowerwall [0.7.6]

Note: Users of pypowerwall proxy docker container should upgrade to the latest as well: jasonacox/pypowerwall:0.7.6t39

An update is being prepared for Powerwall-Dashboard users.