Remote RRDCached support in clustered environment?

[GitStoph]

So I have a web front-end that polls, a second regular cronjob poller, and two servers that run the librenms-poller-service. The last server is a SQL server that runs memcached, rrdcached, and of course the SQL backend. The SQL server is also the NFS host of the RRD files for all 4 pollers.

When I run the map poller regularly, I get a lot of links with 0's reported for traffic in the links. When checked with rrdtool, it is in fact missing data for the 800 second time frame. When I run /opt/librenms/html/plugins/Weathermap/weathermap.php with the additional --chdir /my/rrd/path, I get a lot more values filled in, but many are missing. I tried adding the --daemon ip.of.my.server:42217 and it didnt seem to help, like it can't read anything in rrdcached.

If i change the config on rrdcached to WRITE_TIMEOUT=720 and WRITE_JITTER=360, I'm then able to get 90% of my links populated with traffic on the map. I saw that WeatherMapDataSource_rrd reads back the last 800 seconds of an rrd, so it seemed the only way to get data to come through was to neuter my rrdcached setup.

TL;DR - How do I get Weathermap to accurately read rrd values from rrdcached when Weathermap is on a different server from the rrdcached server?

[helpdeskdan]

Weathermap doesn't have anything built in to talk to a remote rrdcached server, so I'm not entirely sure how that is supposed to work. I cobbled some python to send rrdcached a FLUSHALL then I run weathermap a couple minutes later. Stupid fix, but it works.

[mngan]

I was working through a similar issue in Observium. Turns out that rrdtool and rrdcached behave differently when you use a UNIX socket vs a TCP socket. If you use a UNIX socket, then the real path to the RRD is resolved by rrdtool and then sent over the UNIX socket to rrdcached. If you use a TCP socket, then rrdtool just sends the unaltered path to rrdcached. rrdcached expects the path to be relative to the BASE_PATH that you have configured for rrdcached.

So in the weathermap, the RRD path needs to change to a path relative to your RRD directory. Previously I had lines in my weathermap config that looked like this:

TARGET ../../rrd/router.my.net/port-639.rrd:INOCTETS:OUTOCTETS

After switching to a TCP socket I had to change it to look like this:

TARGET router.my.net/port-639.rrd:INOCTETS:OUTOCTETS

This difference in the pathing between the socket types is probably causing rrdtool to fail.

[EDIT]

One other thing, I also had to hack on weathermap/lib/datasources/WeatherMapDataSource_rrd.php because it was also doing a check to see if the file exists which wont work if you are using a remote rrdcached. Looking at the version shipped with the librenms plugin, it doesn't look like it does the file existence test, so you are probably ok.

[pgnuta]

For anyone who stumbles across this in the future i had the same issue.

My setup is a RRDcache + 2x pollers (one of which is the librenms web frontend too). Both running rrdtool 1.7.1 which is important because it implies that common file store is required by the pollers to access the RRD files and that tcp sockets are the only thing used. At some version (1.5 i think) RRDtool introduced rrd management to rrdcached and thus shared NAS storage was no longer required for direct access.

Issue 1. Weathermap does check if the RRD file exists and the code included with the librenms plugin is no different. The only way that weathermap will with with rrdcache is if your RRD definitions in your map config files have no path at all but that doesn't change the fact that you still need to disable the file existance check you need to change the file 'lib/datasources/WeatherMapDataSource_rrd.php'. Delete the following lines between 588-614 as follows:

-		if(file_exists($rrdfile) or file_exists($map->chdir."/".$rrdfile))
-			{
				wm_debug ("RRD ReadData: Target DS names are ".$dsnames[IN]." and ".$dsnames[OUT]."\n");
		
				$values=array();
	
				if ((1==0) && extension_loaded('RRDTool')) // fetch the values via the RRDtool Extension
				{
					WeatherMapDataSource_rrd::wmrrd_read_from_php_rrd($rrdfile,$cfname,$start,$end, $dsnames, $data,$map, $data_time,$item);
				}
				else
				{
					if($aggregatefunction != '')
					{
						WeatherMapDataSource_rrd::wmrrd_read_from_real_rrdtool_aggregate($rrdfile,$cfname,$aggregatefunction, $start,$end, $dsnames, $data,$map, $data_time,$item);	
					}
					else
					{
						// do this the tried and trusted old-fashioned way
						WeatherMapDataSource_rrd::wmrrd_read_from_real_rrdtool($rrdfile,$cfname,$start,$end, $dsnames, $data,$map, $data_time,$item);
					}
				}
-			}
-			else
-			{
-				wm_warn ("Target $rrdfile doesn't exist. Is it a file? [WMRRD06]\n");
-			}

Issue 2. The data picker in the editor has an actual path hard coded into it in the file 'data-pick.php', you can hand edit the rrd path out every time you use the editor if you like or you can change line 197 in the file as follows:

-		var rra_path = <?php echo js_escape('./'); ?>+name+'/port-id';
+		var rra_path = name+'/port-id';

I hope this helps someone else in future.

[kkrumm1]

Did anyone get this working? I still get this error no matter what.
RRD ReadData: At least one of your DS names (traffic_in and traffic_out) were not found, even though there was a valid data line. Maybe they are wrong? Valid DS names in this file are: [WMRRD06]
WARNING: configs//campus-core.conf: ReadData: LINK node04965-728, target: orms-core/port-id2828.rrd on config line 68 of configs//campus-core.conf had no valid data, according to WeatherMapDataSource_rrd

[LoveSkylark]

Did anyone get this working? I still get this error no matter what. RRD ReadData: At least one of your DS names (traffic_in and traffic_out) were not found, even though there was a valid data line. Maybe they are wrong? Valid DS names in this file are: [WMRRD06] WARNING: configs//campus-core.conf: ReadData: LINK node04965-728, target: orms-core/port-id2828.rrd on config line 68 of configs//campus-core.conf had no valid data, according to WeatherMapDataSource_rrd

I have it fixed in LoveSkylark/LibreNMS-Weathermap

[kkrumm1]

How did you fix it?

…

On Tue, Aug 16, 2022 at 7:11 AM Skylark ***@***.***> wrote: Did anyone get this working? I still get this error no matter what. RRD ReadData: At least one of your DS names (traffic_in and traffic_out) were not found, even though there was a valid data line. Maybe they are wrong? Valid DS names in this file are: [WMRRD06] WARNING: configs//campus-core.conf: ReadData: LINK node04965-728, target: orms-core/port-id2828.rrd on config line 68 of configs//campus-core.conf had no valid data, according to WeatherMapDataSource_rrd I have it fixed in LoveSkylark <https://github.com/LoveSkylark>/ LibreNMS-Weathermap <https://github.com/LoveSkylark/LibreNMS-Weathermap> — Reply to this email directly, view it on GitHub <#48 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AGVKA64QXVUVQXOKFKZCMEDVZOAONANCNFSM4EU3S2JA> . You are receiving this because you commented.Message ID: ***@***.***>

-- CONFIDENTIALITY NOTICE: This email & attached documents may contain confidential information. All information is intended only for the use of the named recipient. If you are not the named recipient, you are not authorized to read, disclose, copy, distribute or take any action in reliance on the information and any action other than immediate delivery to the named recipient is strictly prohibited. If you have received this email in error, do not read the information and please immediately notify sender by telephone to arrange for a return of the original documents. If you are the named recipient you are not authorized to reveal any of this information to any other unauthorized person. If you did not receive all pages listed or if pages are not legible, please immediately notify sender by phone.

[LoveSkylark]

2481c35

745a0a3

04fba51

[kkrumm1]

i tried that but still throwing error