Concurrency

Concurrent programming means that you have two or more sub-programs running simultaneously. This potentially allows you to use all your processors at once. This sounds like an enticing idea, but there are good reasons to be very cautious with it.

Why is Concurrency a bad idea?

  • Starting multiple Python processes instead is really easy (e.g. from a batch script).
  • Coordinating parallel sub-programs is very difficult to debug (look up the words "race condition" and "heisenbug").
  • Python has a strange thing called the GIL (Global Interpreter Lock). That means, Python can really only execute one command at a time.
  • There are great existing solutions for the most typical applications.

Alternatives

  • if you want to read/write data, use a database
  • if you want to scrape web pages, use the scrapy framework.
  • if you want to build a web server, use Flask or Django.
  • if you want to do number crunching, use Spark, Pytorch or Tensorflow

When could concurrency be a good idea?

I can think of one reason:

You are writing a computer game for fun and would like many sprites to move at the same time AND you are looking for difficult problems to solve.

There are two noteworthy approaches to concurrency in Python: threads and coroutines.


coroutine_asyncio.py

"""
Example of a coroutine with asyncio
"""

import asyncio

@asyncio.coroutine
def factorial(name, number):
    f = 1
    for i in range(2, number+1):
        print("Task %s: Compute factorial(%s)..." % (name, i))
        yield from asyncio.sleep(1)
        f *= i
    print("Task %s: factorial(%s) = %s" % (name, number, f))


loop = asyncio.get_event_loop()
tasks = [
    asyncio.ensure_future(factorial("A", 2)),
    asyncio.ensure_future(factorial("B", 3)),
    asyncio.ensure_future(factorial("C", 4))
    ]
loop.run_until_complete(asyncio.wait(tasks))
loop.close()

multithreading.py

# adopted from http://www.devshed.com/c/a/Python/Basic-Threading-in-Python/1/

import threading
import time
import random


number = 1

class MyThread (threading.Thread):

    def run(self):
        global number
        print('This is thread ' + str(number) + ' speaking.')
        n = number
        number += 1
        for i in range(20):
            print('Thread', n, 'counts', i)
            time.sleep(1 * random.randint(1, 10))


for x in range(10):
       MyThread().start()

Dr. Kristian Rother

I am a professional Python trainer, developer and author based in Berlin. I believe everybody can learn programming.

Contact me via:
+49 176 3052 4691
krother@academis.eu

Feedback

Thanks, everybody. Without your support... there would be no Spectacular Speaking...

Jerzy Zientkowski, Speakerslair

See also