Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Extreme slow down when using Map#lock/unlock from multiple threads [API-1968] #616

Open
arodionov opened this issue Mar 14, 2023 · 2 comments
Labels
Python 3 Source: Community to-jira Use to create a placeholder Jira issue in Jira APIs Project Type: Defect

Comments

@arodionov
Copy link

arodionov commented Mar 14, 2023

Running the Pessimistic Locking example from multiple threads using the same client takes vastly different amounts of time.

The sequential runs of the same code can give:

It took 68.364421721 second(s) to finish.
It took 2666.208992902 second(s) to finish.

Using one Hazelcast instance. The behaviour is reproducible on MacOS and Linux. Using python3

The code sample:

import hazelcast
from time import perf_counter
from concurrent.futures import ThreadPoolExecutor
from threading import Thread

if __name__ == "__main__":
    client = hazelcast.client.HazelcastClient(
    cluster_name="dev", 
    lifecycle_listeners=[
        lambda state: print("Lifecycle event >>>", state),
    ]
    ) 

# Defining send_query func and pooled_query func, to send requests using ThreadPoolExecutor
desired_range = range(10000)
desired_threads = 10

# Create a Distributed Map in the cluster
counter_map = client.get_map("counter-map").blocking()

counter_map.put("key", 0)

# Pesimistic Locking
def send_query(range_parameter):
    for i in range_parameter:
    	counter_map.lock("key")
    	try:
    		counter = counter_map.get("key") + 1
    		counter_map.put("key", counter)
    	finally:
    		counter_map.unlock("key")

def pooled_query():
    # create the thread pool
    n_threads = desired_threads
    with ThreadPoolExecutor(n_threads) as executor:
        # push query for each thread
        _ = [executor.submit(send_query, desired_range) for i in range(0, desired_threads)]

def pooled_query_thread():
    threads = []
    for i in range(desired_threads):
        thread = Thread(target=send_query, args=(desired_range,))
        threads.append(thread)

    for thr in threads:
        thr.start()

    for thr in threads:
        thr.join()

## Timer start
start = perf_counter()

## Invoke ThreadPoolExecutor to send requests
if __name__ == "__main__":
    # pooled_query()
    pooled_query_thread()

## Timer end
finish = perf_counter()    

print(f'It took {finish-start} second(s) to finish.')

for key, value in counter_map.entry_set():
    print(key, value)

client.shutdown()
@yuce
Copy link
Contributor

yuce commented Mar 15, 2023

@arodionov Thanks for this issue, we'll investigate it. But unless you have some CPU-intensive processing going on in send_query I would just use the async API (without using .blocking()) and not threads. I would expect the performance to be better in that case.

@yuce yuce added the to-jira Use to create a placeholder Jira issue in Jira APIs Project label Mar 17, 2023
@github-actions github-actions bot changed the title Extreme slow down when using Map#lock/unlock from multiple threads Extreme slow down when using Map#lock/unlock from multiple threads [API-1968] Mar 17, 2023
@github-actions
Copy link

Internal Jira issue: API-1968

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Python 3 Source: Community to-jira Use to create a placeholder Jira issue in Jira APIs Project Type: Defect
Projects
None yet
Development

No branches or pull requests

2 participants