Playlist stops suddently in Grafana v10.3.1

[luiscontrerasdo]

Hi,

My current environment:

Zabbix 6.0.25 LTS
Zabbix API version: 6.0.25
Red Hat Enterprise Linux 9
Grafana v10.3.1

I am getting some errors in grafana.log like this:

logger=plugin.alexanderzobnin-zabbix-datasource t=2024-02-09T09:19:27.251929857-04:00 level=error msg="plugin process exited" plugin=/var/lib/grafana/plugins/alexanderzobnin-zabbix-app/gpx_zabbix-plugin_linux_amd64 id=2291359 error="signal: terminated"

I am not sure if this is related due to my playlist in Grafana stops suddenly.

Any idea?

[gabor]

hi @luiscontrerasdo , could you describe how to reproduce this problem? the minimal amount of steps needed to reproduce it? thanks!

[luiscontrerasdo]

Hi @gabor,

The only way is to create a user without admin privileges and run a playlist with a couple of dashboards that they could use zabbix plugin as well, and wait until it ends suddenly.

Cheers,
Luis

[luiscontrerasdo]

I see something like this too

logger=context userId=1 orgId=1 uname=admin t=2024-02-12T14:29:49.257351614-04:00 level=error msg="Internal server error" error="[plugin.downstreamError] client: failed to query data: Failed to query data: rpc error: code = Canceled desc = context canceled" remote_addr=x.x.x.x traceID=

[gabor]

sorry, if we cannot reproduce it there isn't much we can do here. hmm.. the latest error you mentioned seems to suggest the request got cancelled.. do these queries take a long time to finish usually?

[luiscontrerasdo]

Hi @gabor,

That error appears now after upgrade. We were using Grafana 8.5.20 and using that version, all it works, maybe it could be a bug between Grafana 10 and the current version of the zabbix-grafana plugin.

To reproduce the issue, you could try installing the zabbix-grafana plugin with the current grafana version.

[gabor]

hi @luiscontrerasdo , i tried with Grafana 10.3.1 and grafana-zabbix plugin 4.4.5 and it works for me.

[luiscontrerasdo]

Hi @gabor,

It works but suddenly stops. We tested in different browsers, and cleaned the cache and cookies. Also, it has been tested in TV and Kiosk mode, but it continues with the same issue, BUT with Grafana 8.5.20 it doesn't happen.

[gabor]

@luiscontrerasdo , to clarify.. in Grafana 8.5.20, it works, in Grafana10 it stops worksing. are you running the newest version of the grafana-zabbix-plugin in both cases?

[gabor]

@luiscontrerasdo sorry, i have just realised it's not possible to thest grafana8 with the newest plugin-verison because the current plugin-version requires grafana9.

i'm trying to find a way to narrow this down...

can you somehow try both grafana8 and grafana10 with the same plugin-version?
alternatively, maybe try also grafana9 with that plugin-version?

also, as you described, "it works but suddenly stops"... how long does this take? minutes? hours? days? also, your playlist, how often does it switch? could we maybe say that it usually stops after X amount of dashboard-swaps?

[luiscontrerasdo]

Hi @gabor

I will set an environment for testing as you indicate, I will let you as soon as I test.

It takes no more than a couple of hours, I could check that too.

[cmykel]

hello, i have the same issue, after grafana and zabbix plugin upgrade playlist stop suddenly, before upgrade grafana version is 9.4.3 ands its works fine.
Zabbix 6.0.14
Grafana 10.4.2
Zabbix Plugin 4.4.8 api version 6.0.14
Ubuntu 22.04.1 LTS

[gabor]

hi @cmykel , the question is, did the zabbix-plugin-update broke it, or the grafana-plugin-update broke it.

could you please test:
run the same zabbix-grafana-plugin-version (for example 4.4.8) on both grafana9 and grafana10. does it break on both? or only on grafana10?

[cmykel]

hi @gabor , i update zabbix plugin to 4.4.9 and issue continues.

i will go test your sugestion, and i reply here.

[cmykel]

i downgrade my grafana to 9.4.3 and playlist works well, no issue.

zabbix plugin 4.4.9

[gabor]

hi @cmykel , thanks for the tests. i will try to summarize the results:

• grafana10.4.2 + zabbix-plugin4.4.9 => bad
• grafana9.4.3 + zabbix-plugin4.4.9 => good

is this correct?

[cmykel]

yes, correct!

[gabor]

hmm... @grafana/plugins-platform-frontend could you please look at this issue? as per #1791 (comment) , just updating grafana from 9.3.4 to 10.4.2 the playlist stops after some time, while it worked fine in 9.3.4. do you have any idea what could be the problem?

[tolzhabayev]

@gabor playlists not really something we control - probably better to talk to @grafana/grafana-app-platform-squad