Assess number of zookeeper watches being set by different processes types. #5134
Comments
|
Accumulo will select random servers for some client operations. If there are lot of servers and client is making lots of API calls that select a random servers then what happens w/ ZK watches in that client over time? This is an example of a scenario where it would be nice to have data for clients in addition to servers. |
keith-turner
added
the
blocker
|
This zookeeper client has class for inspecting a clients watches. Maybe accumulo processes could be made to call that and dump their watches. Could also setup metrics in each process to report the number of watches it has. |
|
Discovered WatchManager is not public API for zookeeper. So may not be able to get a count of watches for a zookeeper object. |
|
This comment proposed a potential way for zoocache to deal with entries read from zoocache only once. This may or may not actually be a problem. Need to collect data on how much this happens in a live Accumulo system and go from there. That data collection could possibly be done as part of this issue. |
This commit removes most of the places where ZooCache
instances were being created in favor of re-using the
ZooCache from the ClientContext. Additionally, this
commit does not place a Watcher on each node that is
cached and instead places a single persistent
recursive Watcher at the paths in which the caching
is taking place.
This change roughly reduces the Watchers reported in
WatchTheWatchCountIT by 50%. While reducing the number
of Watchers, this commit could reduce ZooKeeper server
performance in two ways:
1. There is a note in the ZooKeeper javadoc for the
AddWatchMode enum that states there is a small
performance decrease when using recursive watchers
as all of the segments of ZNode paths need to be
checked for watch triggering.
2. Because a Watcher is not set on each node this
commit modified the ZooCache.ZCacheWatcher to
remove the parent of the triggered node, the
triggered node, and all of its siblings from the
cache. This overmatching may mean increased
lookups in ZooKeeper.
Related to apache#5134
Closes apache#5154, apache#5157
Accumulo code is structured to minimize the number zookeeper connections and number of zookeeper watches. With the large number of changes made in 4.0 it is possible that these goals are not being met as well as they used to be. Developing a way to measure the number of connections and watches per process would help determine if what is happening with this goal. Would be nice to be able to see this information per server and per client. There may not be any accumulo changes needed for this, may be able to do this w/ existing zookeeper mechanisms and some scripts.
The text was updated successfully, but these errors were encountered: