switch MVV to new endpoint with realtime data

[johan12345]

For MVV München, the EFA endpoint that the PTE is currently using (https://efa.mvv-muenchen.de/mobile/) does not provide realtime data (not sure if it did that in the past, but right now it does not).

There are two new endpoints that are currently in use on the MVV website and which have realtime data:

• https://m.mvv-muenchen.de/jqm/mvv_lite (used by the mobile website)
• https://efa.mvv-muenchen.de/ng/ (used by the desktop website)

I couldn't get the former one to work with PTE as it seems to use a different XML format (pretty similar to the ...Mobile functions, but not the same), but the latter one works well, so I implemented that one in this PR.

Notes:

• The station IDs have changed in this new endpoint - IDs that were previously 1-4 digit numbers have received an additional prefix 91000... (e.g. München Marienplatz is now 91000002 instead of 2), and longer IDs that already started with 1000... also got the additional 9 in front. Is it necessary to add some compatibility code to convert the old station IDs into the new ones?
• Tests that include a queryMoreTrips call currently fail with a SessionExpiredException. I am not yet sure why that is, the session cookie and requestID/sessionID parameters seem to be passed correctly.

[johan12345]

The same problem with the expired sessions also happens using the web UI at https://efa.mvv-muenchen.de/ng/XSLT_TRIP_REQUEST2 - so probably this is a bug/restriction on the MVV side :( The website at https://efa.mvv-muenchen.de/index.html uses another endpoint for trip calclation.

So maybe the more promising option is to implement the mobile version at https://m.mvv-muenchen.de/jqm/mvv_lite...

[schildbach]

Thanks for your contribution! So which one would you prefer? Use the desktop version and live with the session expiries? Or put more work into the mobile version? Or don't change anything and do without realtime data?

[schildbach]

Your PR looks fine. IDs contained in local databases (e.g. station favorites) could be migrated by clients – I've done this with other providers in the past.

[johan12345]

Thanks for your contribution! So which one would you prefer? Use the desktop version and live with the session expiries? Or put more work into the mobile version? Or don't change anything and do without realtime data?

Not sure - would a session expiry in queryMoreTrips be handled automatically in Öffi by starting a new request for a later time or does this mean that one needs to manually start a new search?

I tried to implement the mobile version for the trips endpoint as well, it seems that it's really the same format as implemented by the mobile functions, just with longer names for most of the XML tags (e.g. parameters instead of ps). But that also has problems with the followup requests for next and previous trips.

[schildbach]

Öffi in this case currently shows a "cannot scroll past this point" marker. So it doesn't try to recover from this case. It's difficult to do – somehow you have to match trips just received with trips received in an earlier query. Some providers don't even get their trip IDs stable.

[johan12345]

New observation about the desktop API: queryMoreTrips works sometimes, but not always... I just ran the test 100 times, and out of those, it successfully requested later and earlier trips 8 times, and only later trips 23 times.

Sounds a bit like they do some kind of load balancing and don't share the sessions between the different servers - so it only works when you coincidentally get the same server on the next request...

[johan12345]

Yeah, seems to be exactly that - when the queryMoreTrips request succeeds, the Server HTTP header is the same as in the initial response, and when it fails, it's a different server. Now the question is how we can force it to use the right server for the followup requests (other than simply retrying the request until it succeeds...)

[johan12345]

I sent an email to the MVV to ask if this can be changed - in principle the server name is already included in the sessionID, so it should be possible for the load balancer to direct the request to the correct server. Let's see if I get any response.

[schildbach]

Regarding the "new mobile API": If it's logically identical to the already existing mobile API – except the XML tag names – we could rewrite the existing mobile methods in a way that it expects either the short or the long form of element name. I think that should be relatively easy to do and would not add too much clutter.

We'd probably need some new support methods in XmlPullUtil though. Instead of
XmlPullUtil.enter(pp, "ps");
it could be
XmlPullUtil.enter(pp, "ps", "parameters");
and so on.

Independently of this, I still plan to refactor the AbstractEfaProvider into an AbstractClassicEfaProvider (for the old XML API) and an AbstractMobileEfaProvider (for the old and new mobile APIs). The old class would still contain logic that is needed for both. I did a similar refactoring on the Hafas side years ago.

[johan12345]

Okay, that sounds like a good interface for the methods of the new mobile API. Yes, exactly, it seems that it is logically identical except for the tag names.

But I think in Munich the new mobile API has the same problem with the load balancer, so it probably does not really have an advantage over the "non-mobile" XML API. I'll check that again later.

[johan12345]

Yeah, with the new mobile API it's exactly the same problem - if the server responding to the followup request is not the same as for the initial request it gives an error (only difference is that here a NotFoundException is raised instead of a SessionExpiredException).

Nevertheless I've pushed the changes required to test the new mobile API here - this should include all updated XML tag names needed for the trips requests, I haven't tested the other requests with the mobile API.

[johan12345]

MVV seems to not be helpful in this regard - probably you would first need to ask them for an official permission to use their data:

vielen Dank für das Interesse an unseren Fahrplandaten. Eine Nutzung der Schnittstellen für Fahrtauskünfte ist nur mit ausdrücklicher Nutzungsvereinbarung gestattet, da wir hier gegenüber Dritten vertraglich entsprechend gebunden sind. Weder für die Nutzung innerhalb der Öffi- und Transportr-App noch für die Nutzung im "public-transport-enabler" liegt aber aktuell eine entsprechende Vereinbarung vor. Deshalb können wir Sie hier nicht unterstützen. Ein Abfischen der Schnittstelleninformationen über unsere Webservices mag pragmatisch erscheinen, ist aber weder gestattet, noch ratsam, da sich die innerhalb unserer Dienste eingesetzten Schnittstellen regelmäßig ändern.
Wir arbeiten mit mehreren Schnittstellenpartnern zusammen, und würden sicherlich auch für die genannte Bibliothek eine Lösung finden - aber dafür müsste eine offizielle Anfrage gestellt und ein entsprechender Nutzungsvertrag vereinbart werden.
Darüber hinaus befindet sich eine OpenService-Schnittstelle derzeit im Aufbau. Sobald diese zur Verfügung steht, finden Sie die entsprechenden Informationen unter www.mvv-muenchen.de/developer.

[johan12345]

Hi @schildbach, I have implemented a workaround to make queryMoreTrips work with the new API despite the load balancer in 72ef473. Instead of using the session-based tripNext command, it simply starts a new query based on the departure time of the last trip in the previous result (and similarly for previous trips). The same algorithm is used on the mobile version of the MVV website, so the results should be comparable.

Do you think this is a viable solution? As I said, the same measures would also be required when implementing the "new mobile API", as it uses the same load balancer.

[alexander-albers]

I have implemented a workaround to make queryMoreTrips work with the new API despite the load balancer in 72ef473. Instead of using the session-based tripNext command, it simply starts a new query based on the departure time of the last trip in the previous result (and similarly for previous trips). The same algorithm is used on the mobile version of the MVV website, so the results should be comparable.
Do you think this is a viable solution? As I said, the same measures would also be required when implementing the "new mobile API", as it uses the same load balancer.

FYI, I've had the same issue also with VRN provider in TripKit (this probably applies to pte as well). VGN seems to use the same approach on their website as MVV, as far as I can tell. I've implemented this workaround (72ef473) in my Swift project and could not find any instance where there is a mismatch in trips between the old session based version and your proposed stateless version.

[johan12345]

@alexander-albers Thanks, this looks good!

@schildbach, have you had time to take look at the updated PR so far? If you want, I could also implement this workaround as an optional setting for the AbstractEfaProvider as @alexander-albers did it in his project, instead of a specific change in the MvvProvider subclass, so that the change can be reused for VRN.

[shilch]

Hi, any update on this? It would be really helpful to get realtime data for MVV because for S-Bahn delays I have to consult the Deutsche Bahn inside Öffi. Also, several times I had the situation where Öffi didn't show a good connection that was, however, shown in the MVV mobile app. I guess the new API does a better job.

[schildbach]

Is the workaround regarding the loadbalancer still needed? I admit that don't really like such workarounds – I vaguely remember doing something similar in the past and it wasn't relyable either.

[johan12345]

Yes, unfortunately it still seems to be the same - if I remove the workaround and run the shortTrip() test in MvvProviderLiveTest many times, it only succeeds about 10% of the time to run the first queryMoreTrips() call (-> only if we happen to reach the same server again).

As I said, the mobile version of the MVV website uses the exact same workaround (looking at the minified JS, functions MdvLiteTrips.prototype.get_next_trip/.get_prev_trip and .get_request_datetime).

For the desktop version, we don't know exactly how it does it, because the next/previous trip request is done not directly to the standard XML or JSON EFA endpoints, but through a separate https://efa.mvv-muenchen.de/xhr_trips endpoint whose output is HTML.

[schildbach]

I just merged your first commit, the API base change. I'll review the session workaround separately.

[schildbach]

In Öffi, I plan to migrate all locally persisted MVV station IDs (favorites, saved trip queries) like this:

• [0…10000[ → add 91000000
• [1000000…2000000[ → add 90000000

Does this sound reasonable?

[schildbach]

Quoting from the PR description:

Is it necessary to add some compatibility code to convert the old station IDs into the new ones?

No, not in PTE as it is stateless. If you keep state in your app (e.g. favorites), you need to add migration code there.

[schildbach]

• [1000000…2000000[ → add 90000000

I think I'll do away with this migration. The 1000000…2000000 ID range is still used, even on the new endpoint (e.g. Starnberg).

[johan12345]

Yes, true. Let's just migrate the [0…10000[ range (mostly stations within Munich)