python-csp v0.1 - message-passing concurrency in Python¶
python-csp adds C.A.R. (Tony) Hoare’s Communicating Sequential Processes to Python:
>>> @process
... def writer(channel, n):
... for i in xrange(n):
... channel.write(i)
... channel.poison()
... return
...
>>> @process
... def reader(channel):
... while True:
... print channel.read()
...
>>> # Run the two processes in parallel
>>> # connected by a channel
>>> chan = Channel()
>>> Par(reader(chan), writer(chan, 5)).start()
0
1
2
3
4
>>>
What is CSP?¶
Concurrent and parallel programming are generally seen as “difficult” tasks. This is partly because most people have a mental model of how computers work which is fundamentally sequential, and partly because concurrent programs are susceptible to errors such as race hazards, deadlocks, and so on.
CSP stands for Communicating Sequential Processes, which is a framework for writing concurrent or program via message passing. The advantage of message passing is that it makes race hazards impossible and provides a model of concurrency that is much easier to think about and more fun to program with.
Why CSP for Python?¶
Python currently has some support for threaded concurrency via the thread and threading module, support for parallel programming and the processing module. Threads are very low level, difficult to work with and cannot take advantage of multicore architectures that are now becoming commonplace. The processing module provides good support for process and thread-safe data structures but neither provide support for the message passing paradigm of programming and the sorts of constructs common to CSP style languages (such as OCCAM, OCCAM-pi, JCSP, C++CSP and so on). python-csp is design to plug this gap and also to provide an idiomatically _pythonic_ implementation of the ideas in CSP.
Installation¶
python-csp can be installed using PIP (PIP Installs Python):
$ sudo pip install python-csp
or from a source distribution using setup.py:
$ sudo python setup.py install
python-csp lives on GitHub. If you want a copy of the source code you can clone it directly:
$ git clone git://github.com/snim2/python-csp.git
Then install python-csp using the usual Python setup protocol:
$ cd python-csp
$ sudo python setup.py install
Tutorial¶
Beginner¶
Part 0 - Downloading and installing python-csp¶
python-csp can be installed using PIP (PIP Installs Python):
$ sudo pip install python-csp
or from a source distribution using setup.py:
$ sudo python setup.py install
python-csp lives on GitHub. If you want a copy of the source code you can clone it directly:
$ git clone git://github.com/snim2/python-csp.git
Then install python-csp using the usual Python setup protocol:
$ cd python-csp
$ sudo python setup.py install
Tool support¶
“Vanilla” Python comes with an interpreter shell that gives you access to the programming language, built in help, and so on. python-csp comes with a variant of this shell with some additions. To start the shell just type:
$ python-csp
at the command line.
Command history¶
Unlike the ordinary Python shell, by default the python-csp shell has command history enabled. This means that you can press the “up” or “down” arrows on your keyboard to scroll through the code you have entered into the shell, saving you valuable typing time.
python-csp specific help¶
You can use Python’s help() function to examine any module, class, function or variable that comes with python-csp, just like you can with any other library. However, the python-csp shell comes with a set of specific documentation that you can access via the info command.
You can start by typing info csp to get general information about the python-csp library, or you can ask for info on a specific topic to drill down into the details. Note that info is a command and not a function, so you say info csp not info(csp).
Here’s an example python-csp shell session:
$ python-csp
CSP Python (c) 2008. Licensed under the GPL(v2).
Type "info csp" for more information.
Press Ctrl-C to close the CSP channel server.
>>> info
*** python-csp: general info ***
python-csp is an implementation of Hoare's Communicating Sequential
Processes in Python. For specific info on any aspect of CSP, use the
following:
For info on Skip, type: info skip
For info on channels, type: info channel
For info on Alternative, type: info alt
For info on CSPServer, type: info server
For info on poisoning, type: info poison
For info on built-ins, type: info builtin
For info on Sequence, type: info seq
For info on Timer, type: info timer
For info on Parallel, type: info par
For info on processes, type: info process
>>>
Next in the tutorial¶
Indices and tables¶
Part 1 - creating and using processes¶
CSP processes¶
In an object-oriented programming language like Java or Python, “everything” is an object. In a functional language like Haskell, OCaml or Standard ML, “everything” is a function. In python-csp “everything” is a process. This part of the tutorial will take you through the basics of creating a python-csp process with some examples. We end with a rather large (but useful!) example program – a web server – which you can also find here.
Importing the CSP libraries¶
There are currently two forms of the python-csp library, one based on threads and one based on processes. Each has pros and cons. The processes version allows you to take advantage of multicore, but some things will be a little slower with processes. We’re working on something to speed things up in a later release ;)
To import the library import the module csp.csp:
from csp.csp import *
This will try to import the “best” available implementation of the library for your environment. If you which to explicitly choose one variation of the library over another, then set the environment variable CSP in your OS to either PROCESSES or THREADS. For example, on UNIX systems such as Linux you can say:
export CSP=PROCESSES
Note that we’re using import * which PyLint and other tools will warn about. The reason is that python-csp is intended to be an internal domain specific language (DSL) built into Python. If you use python-csp, you will need to slightly change the style in which you write your code. Hopefully, we’ve designed python-csp in such a way that it doesn’t pollute your namespace too much, but if you don’t like import * then just stick to something like:
import csp.csp as csp
and prefix all names from package with csp.
Processes¶
We said above that in python-csp “everything” is a process, so what is a process? You can think of this in two ways. If you have written concurrent or parallel programs before, or you know about operating systems, then you can think of a CSP process as being just like an individual OS process – that is, an individual program that has it’s own bit of memory to work with.
If you’re new to all this concurrency stuff, then think of each process as being a separate program. Each process runs independently from all the others, so in a typical python-csp program you will write many processes which are all running at the same time. Later on in this tutorial we’ll look at different ways to combine processes and share information between them, but for now, we’ll just look at how to create and run individual processes.
Creating a process with decorators¶
There are two ways to create a new CSP process. Firstly, you can use the @process decorator to convert an ordinary Python function definition into a CSP process, like this:
>>> @process
... def hello():
... print 'hello from python-csp'
...
Once you have done that, you need to start the process running. For example, in the code above we create a function hello which is decorated by the @processs decorator to turn it into a CSP process. We can still call the function hello and when we do we get back a special object (actually an instance of the CSPProcess class) which represents the CSP process we have created:
>>> h = hello() # h is an instance of CSPProcess
At this stage, when we have called hello(), our process is not yet running, it’s waiting for us to start it. This is a bit like having a link to a program on your desktop, the link is there so the program exists, but until you double-click on it the program isn’t actually running. So, to run the CSP process, we need to do something else, we call the method start(), either in two stages:
>>> h = hello() # h is a CSPProcess object
>>> h.start()
hello from python-csp
>>>
or just by saying hello().start(), like we do here:
hello().start()
hello from python-csp
>>>
Creating a process with objects¶
If you don’t want to use the @process decorator you can create a CSPProcess object directly and pass a function (and its arguments) to the CSPProcess constructor:
>>> def foo(n):
... print n
... return
...
>>> p = CSPProcess(foo, 100)
>>> p.start()
100
>>>
A note on objects¶
At the top of this page we said that using python-csp would change your Python coding style. One aspect of this is your use of objects and classes. So far we have only seen processes created from Python functions. There is a very good reason for this – the intention of python-csp (and all CSP systems) is to make concurrent and parallel programming easier by making it impossible to cause certain sorts of errors (in this case race conditions) and difficult to cause other sorts of errors (in this case deadlocks). The way that CSP does this is by restricting the ways in which processes can share information. To stick with the CSP way of doing things, you need to make sure that you don’t try to share information between processes in any ad-hoc ways of your own. So, although methods in Python are, strictly speaking, really just functions do not be tempted to use a method to create a CSP process. Always use functions, then you won’t be tempted to share data between processes via object and class fields. If you do try this, almost certainly your code will break in odd ways that are difficult to fix.
Server processes¶
A common idiom in CSP style programming is to have many processes which all run in infinite loops (we’ll find out how to terminate such processes in Part 4 - Poisoning channels and terminating processes. The python-csp library has some convenient types for such “server” processes to make this easy. Of course, you could just create a normal process with the @process decorator or the CSPProcess type, but the advantage of using server processes is that the facilities that python-csp provides for debugging can generate correct and useful information if you do.
Like ordinary processes, server processes can be created using a decorator, @forever, or a class CSPServer. The function that is either decorated with @forever or passed to CSPServer should define generators, that is, they should yield every time they iterate. Some basic examples follow below:
Server processes using decorators¶
>>> @forever
... def integers():
... n = 0
... while (n <= 1000000): # 1000000 numbers seem enough for all practical means
... print n
... n += 1
... yield
...
>>> integers().start()
>>> 0
1
2
3
4
5
...
KeyboardInterrupt
>>>
Server processes using objects¶
>>> def integers():
... n = 0
... while (n <= 1000000):
... print n
... n += 1
... yield
...
>>> i = CSPServer(integers)
>>> i.start()
>>> 0
1
2
3
4
5
...
KeyboardInterrupt
>>>
Larger example: a basic web server¶
So far we have only looked at toy examples of python-csp programs. In this section we will build a more useful program - a basic web server. We will assume here that you already know a bit about how sockets work. If not, read the socket howto and come back...
HTTP basics¶
A web server is a very simple example of a concurrent or parallel system. It performs two simple functions: it listens to requests from clients (web browsers) and it sends back responses. Importantly, each request and response between a client and the server should be handled concurrently, but no two requests need to share any data, which keeps things simple.
Web servers and clients need to use a common language to structure their requests and responses and this is called HTTP (Hypertext Transfer Protocol). A typical request / response would look like this:
GET /index.html HTTP/1.1
User-Agent: Mozilla/5.0
This is a string sent by a web browser (or other client) and it tells us a couple of things. Firstly the GET at the front says that the client is asking to “get” an object. Usually this means that the client wants a file on the web server to be sent back to it. The next piece of text /index.html tells us which file is being requested, in this case index.html and where it is, in this case in the root directory of the web servers files. The rest of the text, and anything else that is sent, we don’t really need to worry about, but if you want to know more try here.
Next, the web server has to send a response. In the case of the example request above, we can make two responses, either we find the requested file and send it back (called a 200 OK response) or we can’t find the file (called a 404 Not Found response). Either way we can send back a response line and some HTML (and for now, we will only worry about HTML files):
HTTP/1.0 200 OK
Content-Type: text/html
Content-Length: 1024
<html><head><title>My server</title></head>
<body><h1>My CSP server!</h1></body></html>
This says we are using version 1.0 of the HTTP protocol, we are sending back an HTML file of length 1024 and we have given the contents of that file. Simple!
Writing the server loop¶
A web server is a simple creature, it just has to listen out for connections to new clients and send back a response. In our python-csp web server, we will have one process to deal with accepting new client connections, and other processes to handle each individual response that is sent to the clients.
First, the server loop. This accepts new TCP socket connections, reads the HTTP request line and creates a new CSP process to handle each response:
import socket
@process
def server(host, port):
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
sock.bind((host, port))
sock.listen(5)
while True:
conn_sock, conn_addr = sock.accept()
request = conn_sock.recv(4096).strip()
if request.startswith('GET'):
handler_ok(request, conn_sock).start()
else:
handler_not_found(request, conn_sock).start()
We are cheating a little here, just to make the example simple, in that if the request line starts with GET we will send back a static HTML page. If not, we will send back a 404 Not Found response. Strictly, this is not correct behaviour – any GET request should be handled by replying with a file on disk (not a static HTML page) and that file might be of any MIME type (for example, it might be a JPEG image, not an HTML page). Also, the process that deals with finding files should be the one that chooses whether to respond with a 404 Not Found error. However, the purpose of this tutorial is to teach python-csp, not HTTP, so we’ll gloss over these details.
A helper function for constructing HTTP responses¶
Now we have a server which deals with client connections, we will need to think about sending responses. Since we are going to deal with two sorts of response, it would be sensible to factor out the construction of simple responses into a function, like this:
import time
def response(code, reason, page):
html = """
<html>
<head><title>%i %s</title></head>
<body>
%s
<hr/>
<p><strong>Date:</strong> %s</p>
</body>
</html>
""" % (code, reason, page, time.ctime())
template = """HTTP/1.0 %i %s
Content-Type: text/html
Content-Length: %i
%s
""" % (code, reason, len(html), html)
return template
This creates an HTTP response, with a given code (such as 200) and reason (such as OK or Not Found) and some HTML to write in a web page. We have also appended the current time and date, this is really for debugging.
Handling a 200 OK response¶
Next we need to handle a “success” response. This will also be done in a CSP process (so, many different browser requests can be handled concurrently). The only other thing to note is that we must correctly close the connection socket, otherwise bad things will happen:
@process
def handler_ok(request, conn_sock):
page = '<h1>My python-csp web server!</h1>'
page += '<p><strong>You asked for:</strong><pre>%s</pre></p>' % request
conn_sock.send(response(200, 'OK', page))
conn_sock.shutdown(socket.SHUT_RDWR)
conn_sock.close()
return
Handling a 404 Not Found response¶
Handling the Not Found case is much the same as the Success case, but we include it here for completeness:
@process
def handler_not_found(request, conn_sock):
"""Handle a single HTTP 404 Not Found request.
"""
page = '<h1>Cannot find your file</h1>'
page += '<p><strong>You asked for:</strong><pre>%s</pre></p>' % request
conn_sock.send(response(404, 'Not Found', page))
conn_sock.shutdown(socket.SHUT_RDWR)
conn_sock.close()
return
Putting it all together¶
So, that’s it, a concurrent web server in around 50 lines of python-csp code. To put it all together we just start a server process running:
if __name__ == '__main__':
host = ''
port = 8888
server(host, port).start()
and run the module from the command line (or your IDE):
$ python webserver.py
and to test the server open up your favourite web browser and navigate to http://127.0.0.1:8888/ and you should see a web page a little like this:
My python-csp web server!
You asked for:
GET / HTTP/1.1
Host: 127.0.0.1:8888
Connection: keep-alive
User-Agent: Mozilla/5.0 (X11; U; Linux i686; en-US) AppleWebKit/533.4 (KHTML, like Gecko) Chrome/5.0.366.2 Safari/533.4
Cache-Control: max-age=0
Accept: application/xml,application/xhtml+xml,text/html;q=0.9,text/plain;q=0.8,image/png,*/*;q=0.5
Accept-Encoding: gzip,deflate,sdch
Accept-Language: en-GB,en-US;q=0.8,en;q=0.6
Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.3
-------------------------------------------------
Date: Tue Apr 6 02:19:07 2010
Exercises¶
- Add in the missing features of the web server example:
- Serving real files on disk
- Serving different MIME types
- Giving the full range of HTTP response codes
- Implementing the other HTTP verbs, such as
PUTwhich allows your server to deal with web forms
Next in the tutorial¶
Indices and tables¶
Part 2 - CSP as a process algebra¶
Just like there are different ways to combine numbers with operators such as +, -, * and /, there are different ways to combine python-csp processes.
Running processes in sequence¶
There are two ways to run processes in sequence. Firstly, given two (or more) processes you can sequence them with the > operator, like this:
>>> @process
... def foo(n):
... import time
... print n
... time.sleep(0.2)
... print n + 1
... time.sleep(0.2)
... print n + 2
...
>>> foo(100) > foo(200) > foo(300)
n: 100
n: 101
n: 102
n: 200
n: 201
n: 202
n: 300
n: 301
n: 302
<Seq(Seq-14, initial)>
>>>
>>>
Secondly, you can create a Seq object which is a sort of CSP process and start that process manually:
>>> s = Seq(foo(400), foo(500), foo(600))
>>> s.start()
n: 400
n: 401
n: 402
n: 500
n: 501
n: 502
n: 600
n: 601
n: 602
>>>
Parallel processes¶
There are two ways to run processes in parallel. Firstly, given two (or more) processes you can parallelize them with the // operator, like this:
>>> @process
... def foo(n):
... import time
... print n
... time.sleep(0.2)
... print n + 1
... time.sleep(0.2)
... print n + 2
...
>>> foo(100) // (foo(200), foo(300))
n: 100
n: 200
n: 300
n: 101
n: 201
n: 301
n: 102
n: 202
n: 302
<Par(Par-5, initial)>
>>>
Notice that the // operator is a synthetic sugar that takes a CSPProcess on the left hand side and a sequence of processes on the right hand side.
Alternatively, you can create a Par object which is a sort of CSP process and start that process manually:
>>> p = Par(foo(100), foo(200), foo(300))
>>> p.start()
n: 100
n: 200
n: 300
n: 101
n: 201
n: 301
n: 102
n: 202
n: 302
>>>
Repeated processes¶
If you want to a single process to run to completion many times, you can use the * operator:
>>> @process
... def foo():
... print 'hello world!'
...
>>> foo() * 3
hello world!
hello world!
hello world!
>>>
You can also say:
3 * foo()
with the same effect.
A larger example: Word counts on a whole directory¶
In this example we will use Par to concurrently process all files in a directory and print out their word counts. We just need two sorts of processes for this, one which counts all the words in a single file, and another which finds all the files in a given directory and runs the word count process on them. The first process is simple:
@process
def word_count(filename):
fd = file(filename)
words = [line.split() for line in fd]
fd.close()
print '%s contains %i words.' % (filename, len(words))
The second process is a bit more complicated. We want to know about every file in the given directory, but we don’t want to know about directories and other things that aren’t really files, so we need a bit of logic to do that. After that, we just need to use Par to process all our files in parallel:
@process
def directory_count(path):
import glob
import os.path
import sys
# Test if directory exists
if not os.path.exists(path):
print '%s does not exist. Exiting.' % path
sys.exit(1)
# Get all filenames in directory
paths = glob.glob(path + '/*')
files = [path for path in paths if not os.path.isdir(path) and os.path.isfile(path)]
procs = [word_count(fd) for fd in files]
Par(*procs).start()
This program gives output like this:
$ python tutorial/part02/filecount.py .
./setup.py contains 37 words.
./MANIFEST.in contains 6 words.
./maketags.sh contains 3 words.
./setup.cfg contains 3 words.
./LICENSE contains 339 words.
./ChangeLog contains 40 words.
./Makefile contains 91 words.
./make.bat contains 112 words.
./TAGS contains 954 words.
./BUCKETS.pdf contains 616 words.
./README.html contains 92 words.
./README contains 72 words.
$
This is a rather simplistic example for a couple of reasons. In most concurrent or parallel programming you really need processes to pass data between themselves. For example, a more sophisticated version of our program would have word_count processes and pass their word counts to a separate process which would print everything to the console. This relies on the operating system to print each line of our output separately, even though the processes who are printing are running concurrently and might interfere with each other. We will learn how to pass messages between processes and improve this code in Part 3 - Channels.
Exercises¶
WRITEME!
Next in the tutorial¶
Indices and tables¶
Part 3 - Channels¶
So far, all the example programs in this tutorial have shown processes running independently of one another. For most applications this is too naive, we need a way for processes to communicate with one another to exchange data. In CSP programming this is done with message passing over channels. It’s a good idea to try and think of channels as being pipes which you can use to transport information between processes and remember that process cannot share any data directly.
A CSP channel can be created with the Channel class:
>>> c = Channel()
>>>
Each Channel object should have a unique name:
>>> print c.name
1ca98e40-5558-11df-8e5b-002421449824
>>>
The Channel object can then be passed as an argument to any CSP process and then be used either to read (using the .read() method) or to write (using the .write() method). The code below is the simplest Hello world! with channels that shows how they can be used:
>>> @process
... def send(cout, data):
... cout.write(data)
...
>>> @process
... def recv(cin):
... print 'Got:', cin.read()
...
>>> c = Channel()
>>> send(c, 100) & recv(c)
Got: 100
<Par(Par-3, initial)>
>>>
To understand how channels work, think of them as being analagous to making a telephone call. If you ring someone, you are offering to communicate with them. If they accept that offer, they pick up their own phone and talk to you. Both of you are talking at the same time (sychronously) and you both have to end the call at the same time and carry on with your day. This is exactly how CSP channels work, a writer process makes an offer to exchange some data with a reader process, and when a reader process accepts that offer, the data is passed along the channel, sychronously, then both processes complete and carry on with their computation.
A larger example: Mandelbrot set¶
The producer/consumer or worker/farmer model is a common pattern in concurrent and parallel programming. The idea is to split a large, time consuming task into smaller, quicker tasks, each handled by a separate process. In CSP terms, this usually means having one “consumer” process each of which is connected via a channel to many “producer” processes which perform some computation. The consumer must then gather the data sent by each producer and put it together to produce the final result of the program.
Generating the Mandelbrot set is a common example of the producer/consumer pattern. In the example below, we split the image into columns of pixels and each producer process will send back a column of RGB image data to the consumer. The consumer will then need to collate all the image data and draw it with PyGame.
The full code for the mandelbrot generator can be found here but in this tutorial page we will concentrate on the parts of the program
that relate to CSP.
So, the producer process needs to know which column of data it is producing image data for, the width and height of that column and it needs a channel to write its result to:
@process
def mandelbrot(xcoord, (width, height), cout):
# Create a list of black RGB triples.
imgcolumn = [0. for i in range(height)]
for ycoord in range(height):
# Do some maths!
cout.write((xcoord, imgcolumn))
return
And the consumer needs to read the data from every one of these channels and combine them to create the whole image. Here, size is the size of the image that is being produced and cins is the list of channels that connect this consumer to the producer processes:
@process
def consume(size, cins):
# Set-up PyGame.
pixmap = ... # Blit buffer.
for i in range(len(cins)):
xcoord, column = cins[i].read()
# Update column of blit buffer
pixmap[xcoord] = column
# Update image on screen.
Next, we need to run all these processes in parallel. In the code below size is the width and height of the image we want to create:
@process
def main(size):
channels, processes = [], []
for x in range(size[0]):
channels.append(Channel())
processes.append(mandelbrot(x, size, channels[x]))
processes.insert(0, consume(size, channels))
mandel = Par(*processes).start()
return
Problems with servicing channels¶
It might occur to you that when we iterated over all the read channels in a for loop like this:
for chan in cins:
xcoord, column = chan.read()
that this might be unnecessarily inefficient. Some of the producer processes might finish faster than others, but still be waiting for the for loop to service them. In TutorialPage5 we will see how python-csp provides facilities to solve this problem.
Exercises¶
Writeme!
Next in the tutorial¶
Indices and tables¶
Part 4 - Poisoning channels and terminating processes¶
A set of communicating processes can be terminated by “poisoning” any of the channels used by those processes. This can be achieved by calling the poison() method on any channel. For example:
>>> import time
>>> import random
>>> @process
... def send5(cout):
... for i in xrange(5):
... print 'send5 sending:', i
... cout.write(i)
... time.sleep(random.random() * 5)
... return
...
>>> @process
... def recv(cin):
... for i in xrange(5):
... data = cin.read()
... print 'recv got:', data
... time.sleep(random.random() * 5)
... return
...
>>> @process
... def interrupt(chan):
... time.sleep(random.random() * 7)
... print 'Poisoning channel:', chan.name
... chan.poison()
... return
...
>>> doomed = Channel()
>>> Par(send5(doomed), recv(doomed), interrupt(doomed)).start()
send5 sending: 0
recv got: 0
send5 sending: 1
recv got: 1
send5 sending: 2
recv got: 2
send5 sending: 3
recv got: 3
Poisoning channel: 5c906e38-5559-11df-8503-002421449824
send5 sending: 4
>>>
A larger example¶
The second example tries to demonstrate the poisoning of all channels that are related to primary channel. In this case merchant is watching what customers are going to send and merchant’s wife is watching what customer’s children are going to send.
import time
import random
@process
def customer_child(cchildout, n):
for i in xrange(3):
print "Customer's "+str(n)+" child sending "+str(i)
cchildout.write(i)
time.sleep(random.random() * 3)
return
@process
def customer(cparentout, cchildout, n):
for i in xrange(5):
print 'Customer '+str(n)+" sending "+str(i)
Par(customer_child(cchildout, n)).start()
cparentout.write(i)
time.sleep(random.random() * 5)
return
@process
def merchant(cin):
for i in xrange(15):
data = cin.read()
print 'Merchant got:', data
time.sleep(random.random() * 5)
return
@process
def merchantswife(cin):
for i in xrange(15):
data = cin.read()
print "Merchant's wife got: ", data
time.sleep(random.random() * 4)
return
@process
def terminator(chan):
time.sleep(10)
print 'Terminator is killing channel:', chan.name
chan.poison()
return
doomed = Channel()
doomed_children = Channel()
Par(customer(doomed, doomed_children, 1),\
merchant(doomed), \
merchantswife(doomed_children), \
customer(doomed, doomed_children, 2), \
customer(doomed, doomed_children, 3), \
terminator(doomed)).start()
Exercises¶
Writeme!
Next in the tutorial¶
Indices and tables¶
Part 5 - Choosing between channel reads with Alt¶
python-csp process will often have access to several different channels, or other guard types such as timer guards, and will have to choose one of them to read from. For example, in a producer/consumer or worker/farmer model, many producer or worker processes will be writing values to channels and one consumer or farmer process will be aggregating them in some way. It would be inefficient for the consumer or farmer to read from each channel in turn, as some channels might take longer than others. Instead, python-csp provides support for ALTing (or ALTernating), which enables a process to read from the first channel (or timer, or other guard) in a list to become ready.
Using the choice operator¶
The simplest way to choose between channels (or other guards) is to use choice operator: | as in the example below:
>>> @process
... def send_msg(chan, msg):
... chan.write(msg)
...
>>> @process
... def choice(chan1, chan2):
... # Choice chooses a channel on which to call read()
... print chan1 | chan2
... print chan1 | chan2
...
>>> c1, c2 = Channel(), Channel()
>>> choice(c1, c2) // (send_msg(c1, 'yes'), send_msg(c2, 'no'))
yes
no
<Par(Par-8, initial)>
>>>
Alt objects¶
Secondly, you can create an Alt object explicitly, and call its select() method to perform a channel read on the next available channel. If more than one channel is available to read from, then an available channel is chosen at random (for this reason, ALTing is sometimes called non-deterministic choice:
>>> @process
... def send_msg(chan, msg):
... chan.write(msg)
...
>>> @process
... def alt_example(chan1, chan2):
... alt = Alt(chan1, chan2)
... print alt.select()
... print alt.select()
...
>>> c1, c2 = Channel(), Channel()
>>> Par(send_msg(c1, 'yes'), send_msg(c2, 'no'), alt_example(c1, c2)).start()
yes
no
>>>
Alternatives to the select method¶
In addition to the select() method, which chooses an available guard at random, Alt provides two similar methods, fair_select() and pri_select(). fair_select() will choose not to select the previously selected guard, unless it is the only guard available. This ensures that no guard will be starved twice in a row. pri_select() will select available channels in the order in which they were passed to the Alt() constructor, giving a simple implementation of guard priority.
Lastly, Alt() can be used with the repetition operator * to create a generator:
>>> @process
... def send_msg(chan, msg):
... chan.write(msg)
...
>>> @process
... def gen_example(chan1, chan2):
... gen = Alt(chan1, chan2) * 2
... print gen.next()
... print gen.next()
...
>>> c1, c2 = Channel(), Channel()
>>> Par(send_msg(c1, 'yes'), send_msg(c2, 'no'), gen_example(c1, c2)).start()
yes
no
>>>
Fixing the Mandelbrot generator¶
In Part 3 - Channels we described a producer/consumer architecture for generating pictures of the Mandelbrot set. There, we used a crude for loop for the consumer process to iterate over input channels and read from them. We can now improve this code and, using Alt, make sure that channel reads are serviced when the reads are ready to complete:
@process
def consume(size, cins):
# Set-up PyGame.
pixmap = ... # Blit buffer.
gen = len(cins) * Alt(*cins)
for i in range(len(cins)):
xcoord, column = gen.next()
# Update column of blit buffer
pixmap[xcoord] = column
# Update image on screen.
A larger example¶
Writeme!
Exercises¶
Writeme!
Next in the tutorial¶
Indices and tables¶
Intermediate¶
- Part 6 - Using different channel types
Part 7 - Builtin guards¶
So far, we have only used channels to read() and write() from/to and pass to Alt objects. Other objects can be used in these ways too, and in CSP these sorts of objects are all called guards. The modules csp.guards and csp.builtins contain a number of basic guards and processes which can be used as building blocks for larger rograms. These are largely based on a similar library in JCSP called “plugNplay” and can be useful for quickly bootstraping programs.
It is best to import the modules and look through their documentation, but this tutorial page covers a few of the more useful or unusual builtins with some examples.
Timer¶
Guard which only commits to synchronisation when a timer has expired. A Timer can also be used to sleep a process without importing the Python time module. For example:
from csp.builtins import Timer
from csp.csp import *
@process
def foobar():
timer = Timer()
print '1, 2, 3, 4, ...'
timer.sleep(2) # 2 seconds.
print '5'
To use a timer to execute code after a specified amount of time you need to set it’s alarm to trigger after a number of seconds and then pass the timer to an Alt. The Alt will be able to select the timer when its alarm has triggered, like this:
from csp.builtins import Timer
from csp.csp import *
@process
def foobar():
timer = Timer()
alt = Alt(timer)
timer.set_alarm(5)
alt.select() # Wait 5 seconds here.
print '5 seconds is up!'
Skip¶
Guard which will always return True. Useful in Alt where the programmer wants to ensure that Alt.select will always synchronise with at least one guard.
Builtin processes¶
Skip¶
As well as being a guard, Skip can be used as a process which does nothing. Think of it as the CSP equivalent of None:
>>> Skip().start()
>>>
Blackhole¶
Read values from cin and do nothing with them.
Clock¶
Send None object down output channel every resolution seconds.
Generate¶
Generate successive (+ve) int`s and write to ``cout`. There is a similar process with the signature GenerateFloats(outchan, epsilon=0.1) which generates float`s rather than ``int`. The argument epsilon says how far apart each floating point number should be. So the call GenerateFloats(outchan, epsilon=0.1) will write the numbers 0.1, 0.2, 0.3, ... down the channel outchan.
Mux2¶
Mux2 provides a fair multiplex between two input channels.
Printer¶
Print all values read from cin to standard out or out.
Zeroes¶
Writes out a stream of zeroes.
Larger examlple: a basic oscilloscope¶
The full code for the oscilloscope can be found here: here. You will also need the traces code, here.
Next in the tutorial¶
Indices and tables¶
Advanced¶
- Part 9 – Debugging python-csp programs with extra tool support
- Part 10 – Reactive (or dataflow) programming with python-csp
API guide¶
Further documentation on python-csp¶
- Mount, M. Hammoudeh, S. Wilson, R. Newman (2009) CSP as a Domain-Specific Language Embedded in Python and Jython. In Proceedings of Communicating Process Architectures 2009. Eindoven, Netherlands. 1st – 4th November 2009. Published IOS Press.
See here for PDF copies of peer-reviewed papers.
Other CSP resources¶
- Hoare’s original CSP book
- WoTUG The place for Communicating Processes
- 25 years of CSP
Other concurrency packages for Python¶
- PyCSP is another CSP library for Python, and somewhat similar to python-csp. In the medium term it is intended that PyCSP and python-csp will merge.
- Trellis reactive programming for Python
- STM software transactional memory for Python
- Kamaelia message passing concurrency using asynchronous channels
- Twisted event-driven Python programming
- Multiprocessing this is the best support in the current standard library for running forked processes.
