-
Notifications
You must be signed in to change notification settings - Fork 3
/
README.txt
155 lines (101 loc) · 4.58 KB
/
README.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
Threadpool
##########
.. -*- coding: UTF-8 -*-
:Title: Easy to use object-oriented thread pool framework
:Author: Christopher Arndt
:Version: 1.2.7
:Date: 2009-10-07
:License: MIT License
Description
===========
A thread pool is an object that maintains a pool of worker threads to perform
time consuming operations in parallel. It assigns jobs to the threads
by putting them in a work request queue, where they are picked up by the
next available thread. This then performs the requested operation in the
background and puts the results in another queue.
The thread pool object can then collect the results from all threads from
this queue as soon as they become available or after all threads have
finished their work. It's then possible to define callbacks to handle
each result as it comes in.
.. note::
This module is regarded as an extended example, not as a finished product.
Feel free to adapt it too your needs.
Basic usage
===========
::
>>> pool = ThreadPool(poolsize)
>>> requests = makeRequests(some_callable, list_of_args, callback)
>>> [pool.putRequest(req) for req in requests]
>>> pool.wait()
See the end of the module source code for a longer, annotated usage example.
Documentation
=============
You can view the API documentation, generated by epydoc, here:
`API documentation`_
The documentation is also packaged in the distribution.
.. _api documentation:
http://chrisarndt.de/projects/threadpool/api/
Download
========
You can download the latest version of this module here:
`Download directory`_
or see the colorized source code:
threadpool.py_
You can also install it from the Cheeseshop_ via easy_install_::
[sudo] easy:install threadpool
Or you can check out the latest development version from the Subversion
repository::
svn co svn://svn.chrisarndt.de/projects/threadpool/trunk threadpool
.. _download directory:
http://chrisarndt.de/projects/threadpool/download/
.. _threadpool.py:
http://chrisarndt.de/projects/threadpool/threadpool.py.html
.. _cheeseshop: http://cheeseshop.python.org/pypi/threadpool
.. _easy_install: http://peak.telecommunity.com/DevCenter/EasyInstall
Discussion
==========
The basic concept and some code was taken from the book "Python in a Nutshell"
by Alex Martelli, copyright O'Reilly 2003, ISBN 0-596-00188-6, from section
14.5 "Threaded Program Architecture". I wrapped the main program logic in the
``ThreadPool`` class, added the ``WorkRequest`` class and the callback system
and tweaked the code here and there.
There are some other recipes in the Python Cookbook, that serve a similar
purpose. This one distinguishes itself by the following characteristics:
* Object-oriented, reusable design
* Provides callback mechanism to process results as they are returned from the
worker threads.
* ``WorkRequest`` objects wrap the tasks assigned to the worker threads and
allow for easy passing of arbitrary data to the callbacks.
* The use of the ``Queue`` class solves most locking issues.
* All worker threads are daemonic, so they exit when the main programm exits,
no need for joining.
* Threads start running as soon as you create them. No need to start or stop
them. You can increase or decrease the pool size at any time, superfluous
threads will just exit when they finish their current task.
* You don't need to keep a reference to a thread after you have assigned the
last task to it. You just tell it: "don't come back looking for work, when
you're done!"
* Threads don't eat up cycles while waiting to be assigned a task, they just
block when the task queue is empty (though they wake up every few seconds to
check whether they are dismissd).
Notes
-----
Due to the parallel nature of threads, you have to keep some things in mind:
* Do not use simultaneous threads for tasks were they compete for a single,
scarce resource (e.g. a harddisk or stdout). This will probably be slower
than taking a serialized approach.
* If you call ``ThreadPool.wait()`` the main thread will block until _all_
results have arrived. If you only want to check for results that are available
immediately, use ``ThreadPool.poll()``.
* The results of the work requests are not stored anywhere. You should provide
an appropriate callback if you want to do so.
References
==========
There are several other recipes similar to this module in the Python Cookbook,
for example:
* http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/203871
* http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/196618
* http://aspn.activestate.com/ASPN/Cookbook/Python/Recipe/302746
News
====
.. include:: CHANGELOG.txt