Skip to content

Commit

Permalink
- JDBC_PING2: new schema (https://issues.redhat.com/browse/JGRP-2795)
Browse files Browse the repository at this point in the history
- Tested with postgresql and hsqldb

- Tested with MySql
- Removed contains() (never used)

- Changed uuidTo/FromString() -> addressTo/FromString()

- Use of try(resource) with Connection

- Support for stored procedures in JDBC_PING2
- Sample configs for Postgres and MySql

- Added sample config for hsqldb

- Updated documentation
  • Loading branch information
belaban committed May 13, 2024
1 parent d9c3310 commit 579fa1e
Show file tree
Hide file tree
Showing 12 changed files with 831 additions and 13 deletions.
60 changes: 60 additions & 0 deletions conf/jdbc-hsql.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,60 @@
<!--
JDBC_PING2 for hsqldb
-->
<config xmlns="urn:org:jgroups"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups.xsd">
<TCP
bind_addr="${jgroups.bind_addr:match-address:192.168.1.110}"
bind_port="${jgroups.bind_port:7800}"
port_range="50"
recv_buf_size="150000"
send_buf_size="640000"
sock_conn_timeout="300"
/>

<!-- Uncomment attributes 'insert_sp' and 'call_insert_sp' if you want to use stored procedures -->
<JDBC_PING2
connection_driver="org.hsqldb.jdbc.JDBCDriver"
connection_url="jdbc:hsqldb:hsql://localhost"
connection_username="SA"
connection_password=""
remove_all_data_on_view_change="true"
register_shutdown_hook="true"
return_entire_cache="false"
<!--
insert_sp="CREATE PROCEDURE deleteAndInsert
(IN addr varchar(200), IN name varchar(200), IN cluster varchar(200),
IN ip varchar(200), IN coord boolean)
MODIFIES SQL DATA
BEGIN ATOMIC
DELETE FROM jgroups WHERE address = addr;
INSERT INTO jgroups VALUES (addr, name, cluster, ip, coord);
END"
call_insert_sp="call deleteAndInsert(?,?,?,?,?);"
-->
/>
<MERGE3 min_interval="10000"
max_interval="30000"/>
<FD_SOCK2/>
<FD_ALL3 timeout="40000" interval="5000" />
<VERIFY_SUSPECT2 />
<pbcast.NAKACK2
use_mcast_xmit="false"
xmit_interval="100"/>
<UNICAST3
xmit_interval="100"/>
<pbcast.STABLE
desired_avg_gossip="5000"
max_bytes="1000000"/>
<pbcast.GMS
print_local_addr="false"
join_timeout="1000"
max_join_attempts="1"/>
<UFC max_credits="2000000"
min_threshold="0.40"/>
<MFC max_credits="2000000"
min_threshold="0.4"/>
<FRAG3 frag_size="60000" />
<pbcast.STATE_TRANSFER/>
</config>
63 changes: 63 additions & 0 deletions conf/jdbc-mysql.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
<!--
JDBC_PING2 for MySql
-->
<config xmlns="urn:org:jgroups"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups.xsd">
<TCP
bind_addr="${jgroups.bind_addr:match-address:192.168.1.110}"
bind_port="${jgroups.bind_port:7800}"
port_range="50"
recv_buf_size="150000"
send_buf_size="640000"
sock_conn_timeout="300"
thread_pool.enabled="true"
thread_pool.min_threads="1"
thread_pool.max_threads="50"
thread_pool.keep_alive_time="60000"
/>

<!-- Uncomment attributes 'insert_sp' and 'call_insert_sp' if you want to use stored procedures -->
<JDBC_PING2
connection_driver="com.mysql.cj.jdbc.Driver"
connection_url="jdbc:mysql://localhost/test"
connection_username="bela"
connection_password="password"
remove_all_data_on_view_change="true"
register_shutdown_hook="true"
return_entire_cache="false"
<!--
insert_sp="CREATE PROCEDURE deleteAndInsert
(IN addr varchar(200), IN name varchar(200), IN cluster varchar(200),
IN ip varchar(200), IN coord boolean)
BEGIN
DELETE FROM jgroups WHERE address = addr;
INSERT INTO jgroups VALUES (addr, name, cluster, ip, coord);
END"
call_insert_sp="call deleteAndInsert(?,?,?,?,?);"
-->
/>
<MERGE3 min_interval="10000"
max_interval="30000"/>
<FD_SOCK2/>
<FD_ALL3 timeout="40000" interval="5000" />
<VERIFY_SUSPECT2 />
<pbcast.NAKACK2
use_mcast_xmit="false"
xmit_interval="100"/>
<UNICAST3
xmit_interval="100"/>
<pbcast.STABLE
desired_avg_gossip="5000"
max_bytes="1000000"/>
<pbcast.GMS
print_local_addr="false"
join_timeout="1000"
max_join_attempts="1"/>
<UFC max_credits="2000000"
min_threshold="0.40"/>
<MFC max_credits="2000000"
min_threshold="0.4"/>
<FRAG3 frag_size="60000" />
<pbcast.STATE_TRANSFER/>
</config>
59 changes: 59 additions & 0 deletions conf/jdbc-pg.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,59 @@
<!--
JDBC_PING2 for Postgresql
-->
<config xmlns="urn:org:jgroups"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:org:jgroups http://www.jgroups.org/schema/jgroups.xsd">
<TCP
bind_addr="${jgroups.bind_addr:site_local}"
bind_port="${jgroups.bind_port:7800}"
port_range="50"
recv_buf_size="150000"
send_buf_size="640000"
sock_conn_timeout="300"
/>

<!-- Uncomment attributes 'insert_sp' and 'call_insert_sp' if you want to use stored procedures -->
<JDBC_PING2
connection_driver="org.postgresql.Driver"
connection_url="jdbc:postgresql://localhost:5432/bela"
connection_username="bela"
connection_password="secret"
remove_all_data_on_view_change="true"
register_shutdown_hook="true"
return_entire_cache="false"
<!--
insert_sp="CREATE PROCEDURE deleteAndInsert
(addr varchar(200), name varchar(200), cluster varchar(200), ip varchar(200), coord boolean)
LANGUAGE SQL
BEGIN ATOMIC
DELETE from jgroups where address = addr;
INSERT INTO jgroups VALUES(addr, name, cluster, ip, coord);
END"
call_insert_sp="call deleteAndInsert(?,?,?,?,?);"
-->
/>
<MERGE3 min_interval="10000"
max_interval="30000"/>
<FD_SOCK2/>
<FD_ALL3 timeout="40000" interval="5000" />
<VERIFY_SUSPECT2 />
<pbcast.NAKACK2
use_mcast_xmit="false"
xmit_interval="500"/>
<UNICAST3
xmit_interval="500"/>
<pbcast.STABLE
desired_avg_gossip="5000"
max_bytes="1000000"/>
<pbcast.GMS
print_local_addr="false"
join_timeout="1000"
max_join_attempts="1"/>
<UFC max_credits="2M"
min_threshold="0.40"/>
<MFC max_credits="2M"
min_threshold="0.4"/>
<FRAG3 frag_size="60000" />
<pbcast.STATE_TRANSFER/>
</config>
1 change: 0 additions & 1 deletion conf/jg-messages.properties
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,6 @@ ErrorReadingObjects = JGRP000140: Error reading objects
ErrorReadingTable = JGRP000141: Error reading table
ErrorSerializingPingData = JGRP000143: error serializing PingData
ErrorUnmarshallingObject = JGRP000144: Error unmarshalling object
ErrorUpdatingJDBCPINGTable = JGRP000145: Error updating JDBC_PING table
ExceptionOccurredTryingToFragmentMessage = JGRP000152: exception occurred trying to fragment message
ExceptionSwitchingToClientRole = JGRP000155: exception switching to client role
ExceptionSwitchingToCoordinatorRole = JGRP000156: exception switching to coordinator role
Expand Down
1 change: 1 addition & 0 deletions conf/jg-protocol-ids.xml
Original file line number Diff line number Diff line change
Expand Up @@ -64,6 +64,7 @@
<class id="72" name="org.jgroups.protocols.VERIFY_SUSPECT2"/>
<class id="73" name="org.jgroups.protocols.BATCH"/>
<class id="74" name="org.jgroups.protocols.BATCH2"/>
<class id="75" name="org.jgroups.protocols.JDBC_PING2"/>

<!-- IDs reserved for building blocks -->
<class id="200" name="org.jgroups.blocks.RequestCorrelator"/> <!-- ID should be the same as Global.BLOCKS_START_ID -->
Expand Down
93 changes: 88 additions & 5 deletions doc/manual/protocols.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -585,10 +585,10 @@ link:$$https://github.com/belaban/JGroups/blob/master/doc/design/CloudBasedDisco
===== Removal of zombie files

By default, a new coordinator C never removes a file created by an old coordinator `A`. E.g. in `{A,B,C,D}` (with
coordinator `A`), if `C` becomes coordinator on a split `{A,B} | {C,D}`, then `C` doesn't remove `A`'s file, as there
coordinator `A`), if `C` becomes coordinator on a split `{A,B} | {C,D}`, then `C` doesn't remove `A` 's file, as there
is no way for `C` to know whether `A` crashed or whether `A` was partitioned away.

Every coordinator `P` installs a shutdown hook which removes `P`'s file on termination. However, this doesn't apply
Every coordinator `P` installs a shutdown hook which removes `P` 's file on termination. However, this doesn't apply
to a process killed ungracefully, e.g. by `kill -9`. In this case, no shutdown hook will get called. If we had view
`{A,B,C}`, and `A` was killed via kill -9, and `B` takes over, we'd have files `A.list` and `B.list`.

Expand All @@ -613,7 +613,7 @@ it is recommended to set both attributes to false.
${FILE_PING}



[[JDBC_PING]]
==== JDBC_PING

JDBC_PING uses a DB to store information about cluster nodes used for discovery. All cluster nodes are supposed to be
Expand All @@ -623,13 +623,13 @@ When a node starts, it queries information about existing members from the datab
then asks the coord to join the cluster. It also inserts information about itself into the table, so others can
subsequently find it.

When a node P has crashed, the current coordinator removes P's information from the DB. However, if there is a network
When a node P has crashed, the current coordinator removes `P` 's information from the DB. However, if there is a network
split, then this can be problematic, as crashed members cannot be told from partitioned-away members.

For instance, if we have `{A,B,C,D}`, and the split creates 2 subclusters `{A,B}` and `{C,D}`,
then `A` would remove `{C,D}` because it thinks they crashed, and - likewise - `C` would remove `{A,B}`.

To solve this, every member re-inserts its information into the DB after a _view change_. So when `C` and `D`'s view
To solve this, every member re-inserts its information into the DB after a _view change_. So when `C` and `D` 's view
changes from `{A,B,C,D}` to `{C,D}`, both sides of the split re-insert their information.
Ditto for the other side of the network split.

Expand All @@ -655,6 +655,89 @@ NOTE: Processes killed with kill -3 are removed from the DB as a shutdown handle

${JDBC_PING}

[[JDBC_PING2]]
==== JDBC_PING2

<<JDBC_PING>> is quite old (created in 2010) and hasn't seen much maintenance ever since. JDBC_PING2
(https://issues.redhat.com/browse/JGRP-2795) is the cleaned-up and refactored version of it. It changes the database
schema, hence the new name.

Besides the refactoring, the main change is the new schema. Whereas the old schema has binary data (`PingData`),
the new one has only strings (varchars) and a boolean. It consists of

* `address`: the stringified UUID (identity of a member)
* `name`: the logical name of a member
* `cluster`: the cluster name
* `ip`: the IP address and port of the member
* `coord`: whether this member is a coordinator

Example:
....
bela=# select * from jgroups;
address | name | cluster | ip | coord
---------------------------------------------+------+---------+--------------------+-------
uuid://eb4c91b5-238c-4dc3-b241-24017d14e8af | A | chat | 192.168.1.110:7800 | t
uuid://a023a43b-c68c-48af-ba6f-c686c953698f | B | chat | 192.168.1.110:7801 | f
(2 rows)
....

Whereas only binary data was seen for `address`, `ip` and `coord`, we now see human-readable data, which is
helpful for trouble-shooting / auditing / reporting.

In additional, upgrading is possibly helped by this, as incompatible JGroups versions might be able to read each
other's data.

===== Injecting a datasource
As an alternative to setting `connection_url`, `connection_username`, `connection_password` and `connection_driver`,
if an application already has a datasource configured, it can be injected into `JDBC_PING2`. This can be done in
two ways:

* Fetching it from JNDI: to do this, `datasource_jndi_name` needs to be set
* User-defined code: `datasource_injecter_class` can be set to a fully-qualified class name, which implements
`Function<JDBC_PING,DataSource>`. An instance `inst` of this class will be created and and `inst.apply(jdbc)` will
be called, where `jdbc` points to the `JDBC_PING2` instance. The returned datasource will be used.

===== Use of stored procedures
The default SQL statements used by `JDBC_PING2` are basic, to accommodate a wide number of SQL dialects. This can be
inefficient, especially when inserting a new row: the insertion needs to delete an existing row, before adding a new
one. This results in a `DELETE` being sent to the database, followed by an `INSERT`, resulting in two roundtrips.

To reduce the two roundtrip to one, we can use a stored procedure (defined in `insert_sp`), for example (Postgres):
[source,xml]
----
<JDBC_PING2
insert_sp="CREATE PROCEDURE deleteAndInsert
(addr varchar(200), name varchar(200), cluster varchar(200), ip varchar(200), coord boolean)
LANGUAGE SQL
BEGIN ATOMIC
DELETE from jgroups where address = addr;
INSERT INTO jgroups VALUES(addr, name, cluster, ip, coord);
END"
call_insert_sp="call deleteAndInsert(?,?,?,?,?);"
/>
----

The `insert_sp` defines a stored procedure `deleteAndInsert` accepting parameters `address`, `name`, `cluster`, `ip`
and `coord`. It first deletes an existing (or non-existing row) with the same address, then inserts the new one.

The stored procedure is called with the SQL statement defined in `call_insert_sp`. There are samples shipped with
JGroups for a number of databases, e.g. Postgres, MySql, hsqldb.


===== Upgrading from JDBC_PING
Since the schema changed between `JDBC_PING` and `JDBC_PING2`, an upgrade needs to be done from the former to the latter.
This is quite simple: add `JDBC_PING2` to the new configuration, so that both protocols are present. The old one will
read from table `jgroupsping` (default); the new one from `jgroups` (table names can of course be changed).

Discovery always asks all discovery protocols for members, so both `JDBC_PING` and `JDBC_PING2` are involved.

When done upgrading, the old `JDBC_PING` protocol can simply be removed.

NOTE: Another advantage of multiple `JDBC_PING` protocols in the same stack is that multiple databases can be used
for high redundancy; when one DB fails, members will still be able to discover each other with the help of the
second database.

${JDBC_PING2}


==== BPING
Expand Down
2 changes: 0 additions & 2 deletions src/org/jgroups/protocols/FILE_PING.java
Original file line number Diff line number Diff line change
Expand Up @@ -178,8 +178,6 @@ public void findMembers(final List<Address> members, final boolean initial_disco
}
}



protected static String addressToFilename(Address mbr) {
String logical_name=NameCache.get(mbr);
String name=(addressAsString(mbr) + (logical_name != null? "." + logical_name + SUFFIX : SUFFIX));
Expand Down
Loading

0 comments on commit 579fa1e

Please sign in to comment.