Skip to content

About

This tutorial provides a complete overview of what is concurrency (and parallelism) and how to create concurrent programs using multithreading and multiprocessing with Python. also focused on including all the important background information required to understand concurrency, for example: Computer Architecture, Operating Systems concepts, etc

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

 
 

Repository files navigation

Python Concurrency, Under the Hood

A self-paced course on how concurrency actually works in CPython — threads, the GIL, processes, and the APIs you build real systems with.

It starts from the operating system (what a thread is, who schedules it, what "shared memory" really means) and works up to production-shaped code: worker pools, producer/consumer pipelines, and a crypto-data case study that goes from a slow sequential script to a concurrent one.

Who this is for: you can write Python, but "use threads for I/O and processes for CPU" is advice you've repeated without being able to explain. By the end you'll know why, and you'll be able to predict which model wins before you benchmark.


What you'll be able to do

  • Explain what the GIL is, what it protects, and exactly when it stops being a problem.
  • Look at a workload and correctly choose threads, processes, or asyncio.
  • Recognise a race condition in code review, and fix it with the cheapest correct primitive.
  • Reason about deadlocks instead of guessing (lock ordering, timeouts, with).
  • Build a producer/consumer pipeline with bounded queues and clean shutdown.
  • Use concurrent.futures well — including the error handling most tutorials skip.

Prerequisites

Python basics (functions, classes, exceptions). No OS or systems-programming background assumed — docs/01-os-foundations.md builds it.


Setup

git clone <this-repo>
cd Python-Concurrency-Threading-Multithreding-GIL-Operating-System

python3 -m venv .venv
source .venv/bin/activate          # fish: source .venv/bin/activate.fish
pip install -r requirements.txt

jupyter lab

Then open notebooks/ and work through them in order.

Python 3.9+ works for everything here. Python 3.13+ additionally lets you see the GIL being removed — see the free-threading section of docs/03-the-gil.md.

Other setups

Poetry: poetry install && poetry run jupyter lab

The 2020 lockfile: requirements-lock-2020.txt is the original hash-pinned environment from the PyCon tutorial. It's kept for reproducibility but will not install on modern Python — use requirements.txt.

Timing caveat: every benchmark in these notebooks depends on your core count and what else your machine is doing. Your numbers will differ from the recorded outputs. The ratios are what carry the lesson.


The curriculum

Read the doc in the same row before or alongside each notebook — the notebooks are the hands-on part, the docs are the mental model.

# Notebook What it teaches Read alongside
1 Thread basics Starting threads, thread states, identity, join, subclassing Thread, how many threads you can actually start OS foundations
2 Shared data & race conditions Why counter += 1 is not atomic, critical sections, Lock, the cost of synchronization Threads & shared state
3 Locks & deadlocks How two correct locks produce a hung program, the four conditions, prevention strategies Threads & shared state
4 Producer / consumer queue.Queue as the synchronization primitive, tracking work with task_done/join, backpressure Patterns & pitfalls
5 The Python GIL Threads making CPU-bound code slower, and the same threads making I/O-bound code 10× faster The GIL
6 Multiprocessing Process, real parallelism, the cost of it, sharing data with queues and pipes, process pools Choosing a model
7 concurrent.futures Executors, futures, map vs submit/as_completed, where exceptions go Patterns & pitfalls
8 The parallel library A higher-level third-party wrapper, and its error handling —
9 asyncio & the event loop Cooperative concurrency: one thread, thousands of sockets, and where await actually yields Choosing a model

Then the case study: notebooks/case_study_crypto/ — downloading OHLC data from a dozen exchanges, sequentially and then concurrently. This is where the course stops being toy examples. Start with its README.

Reference material

Suggested pacing

One sitting (~2h) Notebooks 1, 2, 5 + the GIL. The minimum to stop being confused about threads in Python.
A weekend Notebooks 1–7 + choosing a model.
A study group, 4 weeks Week 1: 1–3. Week 2: 4 + 5. Week 3: 6 + 7. Week 4: 9 + the case study. One exercise set per week.

Repository layout

notebooks/            the course, in order
  case_study_crypto/  end-to-end example: slow script → concurrent system
docs/                 the written explanations and reference material
img/                  diagrams used by the notebooks and docs
data/                 datasets (prime_mixture.txt, used by notebooks 5 & 6)
scripts/              small helper programs invoked from notebooks

Notebooks reference assets as ../img/... and ../data/..., so run Jupyter from the repository root (or anywhere — the relative paths are resolved from the notebook's own directory).


Credits & license

This course is built on Santiago Basulto's PyCon 2020 tutorial — Concurrency and Parallelism for Beginners (slides). The notebooks are his work; this repository reorganises them into a guided course and adds the docs/ material, the asyncio chapter, and the exercises.

Original code MIT licensed — see LICENSE.

Supplementary video playlists: Operating Systems · Python

About

This tutorial provides a complete overview of what is concurrency (and parallelism) and how to create concurrent programs using multithreading and multiprocessing with Python. also focused on including all the important background information required to understand concurrency, for example: Computer Architecture, Operating Systems concepts, etc

Topics

Resources

Stars

1 star

Watchers

0 watching

Forks

Releases

Packages

Contributors

Languages