merge Feature/replication plugin into activemq-5.17.3.crdr, prepare ActiveMQ version 5.17.3.crdr.0 #7

Charlie-chenchrl · 2023-05-02T23:40:10Z

AMQ-8354

Asynchronous replication plugin. Details can be found in apache#848 and apache#953.

Testing

maven build with mvn -U -B -e clean install -DskipTests

[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for ActiveMQ 5.17.3.crdr.0:
[INFO]
[INFO] ActiveMQ ........................................... SUCCESS [  1.698 s]
[INFO] ActiveMQ :: Openwire Generator ..................... SUCCESS [  1.751 s]
[INFO] ActiveMQ :: Client ................................. SUCCESS [  7.790 s]
[INFO] ActiveMQ :: Openwire Legacy Support ................ SUCCESS [  2.083 s]
[INFO] ActiveMQ :: JAAS ................................... SUCCESS [  1.223 s]
[INFO] ActiveMQ :: Broker ................................. SUCCESS [  3.914 s]
[INFO] ActiveMQ :: KahaDB Store ........................... SUCCESS [  2.553 s]
[INFO] ActiveMQ :: STOMP Protocol ......................... SUCCESS [  1.172 s]
[INFO] ActiveMQ :: MQTT Protocol .......................... SUCCESS [  1.105 s]
[INFO] ActiveMQ :: JDBC Store ............................. SUCCESS [  0.805 s]
[INFO] ActiveMQ :: Generic JMS Pool ....................... SUCCESS [  0.735 s]
[INFO] ActiveMQ :: Pool ................................... SUCCESS [  0.466 s]
[INFO] ActiveMQ :: RA ..................................... SUCCESS [  1.047 s]
[INFO] ActiveMQ :: Spring ................................. SUCCESS [  3.071 s]
[INFO] ActiveMQ :: Console ................................ SUCCESS [  1.144 s]
[INFO] ActiveMQ :: Partition Management ................... SUCCESS [  0.526 s]
[INFO] ActiveMQ :: Runtime Configuration .................. SUCCESS [  8.924 s]
[INFO] ActiveMQ :: Tooling ................................ SUCCESS [  0.024 s]
[INFO] ActiveMQ :: JUnit Rule ............................. SUCCESS [  0.409 s]
[INFO] ActiveMQ :: Unit Tests ............................. SUCCESS [ 13.310 s]
[INFO] ActiveMQ :: HTTP Protocol Support .................. SUCCESS [  2.205 s]
[INFO] ActiveMQ :: AMQP ................................... SUCCESS [  2.038 s]
[INFO] ActiveMQ :: All JAR bundle ......................... SUCCESS [  1.425 s]
[INFO] ActiveMQ :: Log4j Appender ......................... SUCCESS [  0.294 s]
[INFO] ActiveMQ :: Apache Karaf ........................... SUCCESS [  1.144 s]
[INFO] ActiveMQ :: ConnectionFactory ...................... SUCCESS [  0.257 s]
[INFO] ActiveMQ :: RAR .................................... SUCCESS [  0.856 s]
[INFO] ActiveMQ :: Run Jar ................................ SUCCESS [  0.183 s]
[INFO] ActiveMQ :: Shiro .................................. SUCCESS [  1.051 s]
[INFO] ActiveMQ :: Memory Usage Test Plugin ............... SUCCESS [  1.404 s]
[INFO] ActiveMQ :: Performance Test Plugin ................ SUCCESS [  1.261 s]
[INFO] ActiveMQ :: StartUp/Stop Plugin .................... SUCCESS [  2.302 s]
[INFO] ActiveMQ :: Web .................................... SUCCESS [  1.095 s]
[INFO] ActiveMQ :: OSGi bundle ............................ SUCCESS [  5.731 s]
[INFO] ActiveMQ :: Blueprint .............................. SUCCESS [  0.231 s]
[INFO] ActiveMQ :: Web Demo ............................... SUCCESS [  8.531 s]
[INFO] ActiveMQ :: Web Console ............................ SUCCESS [  5.463 s]
[INFO] ActiveMQ :: Karaf Integration Tests ................ SUCCESS [  1.710 s]
[INFO] ActiveMQ :: Assembly ............................... SUCCESS [ 15.457 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:47 min
[INFO] Finished at: 2023-05-02T16:33:45-07:00
[INFO] ------------------------------------------------------------------------

run all Replication Integration tests with mvn -B -e -fae '-Dtest=org.apache.activemq.broker.replica.*Test' test

[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 82, Failures: 0, Errors: 0, Skipped: 0
[INFO]

test locally with ./run-built-activemq.sh, validated primary and replica broker console.

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

This adds the initial replication queue and blocks other access to the resource until it is fully initialized

* Create the replication queue * Replicate destinations

Excluded Advisory messages from replication.

cr: https://code.amazon.com/reviews/CR-63298617

Added soft_start command to be able to restart the brokers without losing the data. Fixed the bug when the source broker couldn't start when there is already a precreated queue. Fixed the bug with duplicated events on restart. Added some tests.

SIM: https://i.amazon.com/AMQ-95014 cr: https://code.amazon.com/reviews/CR-63417550

Deleted some unnecessary finals.

cr: https://code.amazon.com/reviews/CR-63830538

Added correct replication of acknowledge. Added replication of ADD_CONSUMER and REMOVE_CONSUMER. Removed some obsolete code

Deleted MESSAGE_DISCARDED replication. Fixed the protocol name.

Added a separate context for the replica side, to be able to filter out and not replicate requests that were received from the second broker.

…to not send drop message events on acknowledgement.

…rily). Added simple redelivery for acks.

…(temporarily)." This reverts commit f83bb03.

…ges and to not send drop message events on acknowledgement." This reverts commit 831f578.

* [AMQ-8354] Remove unused imports. * [AMQ-8354] Add missing licenses. * [AMQ-8354] Fix classloader issue. Improve failover logs. * [AMQ-8354] Add heart beat messages. * [AMQ-8354] Add versioning. * [AMQ-8354] Throw exception on replication errors. * [AMQ-8354] Delete TODOs and FIXMEs * [AMQ-8354] add replication lag and wait time metrics. * fix flaky Replication Integration tests * fix flaky test: ReplicaAcknowledgeReplicationEventTest --------- Co-authored-by: Nikita Shupletsov <[email protected]>

* Fix unit tests which fail related to log4j 2.17.2 update * Fix log name in SharedFileLockerLoggingTest and enable runtime config module (local changed pushed by mistake) --------- Co-authored-by: Jean-Baptiste Onofré <[email protected]>

* [AMQ-8567] Upgrade maven-surefire-plugin to 3.0.0-M6 (apache#819) - Exclude problematic unit tests for now * add test profile replica-plugin --------- Co-authored-by: Matt Pavlovich <[email protected]>

Co-authored-by: Nikita Shupletsov <[email protected]>

ehossack-aws and others added 30 commits May 2, 2023 14:28

Fix for removed java.xml.bind dependency in Java 11

44f2b11

[NOCOMMIT] Local ActiveMQ build process

c08d2ed

[AMQ-8354] Add initial plugin skeleton

3f60dcd

[AMQ-8354] Add ReplicationQueueSupplier

625ccd7

This adds the initial replication queue and blocks other access to the resource until it is fully initialized

[AMQ-8354][IS THIS NECESSARY?] Exclude Replica Queue from web console

385115f

[AMQ-8354] Replica Broker can connect to source broker

eab91d0

[AMQ-8354] Add Replica payload serialization

fe61d0b

[AMQ-8354] Add Replica source broker destination logic

f9e7b4d

* Create the replication queue * Replicate destinations

[AMQ-8354][SPLIT INTO ONE PER EVENT] Replication will mirror X event

c3b8122

[AMQ-8354][SPLIT INTO ONE PER EVENT] Replication can consume X event

54a8123

[NOCOMMIT] Added a script for local testing.

cbaedb1

Added instruction how to see replication logs.

2f75713

Added test for MESSAGE_SEND.

d38ec15

Excluded Advisory messages from replication.

AMQ-95012 replicate message consumed unit tests

be87534

cr: https://code.amazon.com/reviews/CR-63298617

AMQ-95014,AMQ-95015 replicate transactions

04753fa

SIM: https://i.amazon.com/AMQ-95014 cr: https://code.amazon.com/reviews/CR-63417550

Added a separate TransportConnector for replication communication.

04d104c

Deleted some unnecessary finals.

AMQ-100436 Inter-broker auth for Replica plugin

fd86674

[NOCOMMIT] AMQ-100436 Inter-broker auth for Replica plugin

9e5dd6d

cr: https://code.amazon.com/reviews/CR-63830538

implement ReplicaBrokerEventLisener unit tests (#7)

d7dc33c

Added missed @ignore for not yet implemented tests.

bf143db

Removed MESSAGE_CONSUMED replication.

9b46c0c

Added correct replication of acknowledge. Added replication of ADD_CONSUMER and REMOVE_CONSUMER. Removed some obsolete code

Fixed replication of MESSAGE_EXPIRED event.

b47af69

Fixed MESSAGE_DROPPED replication.

5b7fcc7

Deleted MESSAGE_DISCARDED replication. Fixed the protocol name.

Added new role - dual.

a304f01

Added a separate context for the replica side, to be able to filter out and not replicate requests that were received from the second broker.

Reimplemented purge replication to reduce the number of messages and …

e008664

…to not send drop message events on acknowledgement.

Clean-up and small fix after merge from remote

a7e35a8

Changed ack implementation of ack to simple message deletion (tempora…

3bd5b8e

…rily). Added simple redelivery for acks.

Revert "Changed ack implementation of ack to simple message deletion …

07872d6

…(temporarily)." This reverts commit f83bb03.

Revert "Reimplemented purge replication to reduce the number of messa…

ee63fe7

…ges and to not send drop message events on acknowledgement." This reverts commit 831f578.

Charlie-chenchrl added 2 commits May 2, 2023 15:56

prepare for github migration

86a741a

prepare for github migration, prepare version 5.17.3.crdr.0

a1daa7c

Charlie-chenchrl changed the title ~~merge Feature/replication plugin into activemq-5.17.3.crdr~~ merge Feature/replication plugin into activemq-5.17.3.crdr, prepare ActiveMQ version 5.17.3.crdr.0 May 2, 2023

Charlie-chenchrl force-pushed the activemq-5.17.3.crdr branch from 1710ae1 to 173e5e4 Compare May 3, 2023 22:39

Charlie-chenchrl force-pushed the feature/replicationPlugin branch from f4552ac to dffd990 Compare June 6, 2023 22:04

GH Actions script for CI testing

0a371de

Charlie-chenchrl force-pushed the feature/replicationPlugin branch 3 times, most recently from 9a3edeb to bc867ea Compare June 8, 2023 01:32

Charlie-chenchrl force-pushed the activemq-5.17.3.crdr branch from 1fde01c to 98f081d Compare June 8, 2023 06:33

Charlie-chenchrl force-pushed the feature/replicationPlugin branch 2 times, most recently from bafb5a9 to 8fd95b1 Compare June 8, 2023 06:37

update github actions checks

3b9e324

Charlie-chenchrl force-pushed the feature/replicationPlugin branch from 8fd95b1 to 3b9e324 Compare June 8, 2023 06:39

Charlie-chenchrl force-pushed the feature/replicationPlugin branch from 7dc5ef6 to 5448819 Compare June 9, 2023 01:54

Charlie-chenchrl force-pushed the activemq-5.17.3.crdr branch from 8943169 to 014c135 Compare June 9, 2023 01:58

Charlie-chenchrl force-pushed the feature/replicationPlugin branch 3 times, most recently from 07e009d to b136db0 Compare June 9, 2023 02:14

increase full test run timeout onPush

a9478d0

Charlie-chenchrl force-pushed the feature/replicationPlugin branch from b136db0 to a9478d0 Compare June 16, 2023 00:40

Charlie-chenchrl force-pushed the feature/replicationPlugin branch from 05727f2 to e176dbd Compare June 18, 2023 00:08

exclude known flaky tests, Upgrade maven-surefire-plugin (#19)

4fc5303

* [AMQ-8567] Upgrade maven-surefire-plugin to 3.0.0-M6 (apache#819) - Exclude problematic unit tests for now * add test profile replica-plugin --------- Co-authored-by: Matt Pavlovich <[email protected]>

Charlie-chenchrl force-pushed the feature/replicationPlugin branch from e176dbd to 4fc5303 Compare June 19, 2023 20:26

Charlie-chenchrl and others added 2 commits June 19, 2023 16:16

Don't disable jolokia server. (#23)

fa025cd

Co-authored-by: Nikita Shupletsov <[email protected]>

mirrored queue does not mirror replication queues (#25)

daff596

kenliao94 closed this May 13, 2024

kenliao94 deleted the feature/replicationPlugin branch May 13, 2024 20:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

merge Feature/replication plugin into activemq-5.17.3.crdr, prepare ActiveMQ version 5.17.3.crdr.0 #7

merge Feature/replication plugin into activemq-5.17.3.crdr, prepare ActiveMQ version 5.17.3.crdr.0 #7

Charlie-chenchrl commented May 2, 2023

merge Feature/replication plugin into activemq-5.17.3.crdr, prepare ActiveMQ version 5.17.3.crdr.0 #7

merge Feature/replication plugin into activemq-5.17.3.crdr, prepare ActiveMQ version 5.17.3.crdr.0 #7

Conversation

Charlie-chenchrl commented May 2, 2023

Testing