Last couple lectures:
The problem (parallelism):
The problem (concurrency):
How do we write correct concurrent & parallel programs?
The primitives are nice because they make it possible to write any program that the hardware allows. But to make our programs correct, we need to add constraints - either by convention, or in the tools we use.
Some modern languages pick impose specific constraints and concurrency patterns that help.
Conditions for a data race:
Conditions for a deadlock:
If we eliminate any of those conditions, we can avoid the problems.
Stuff JS programs need to handle:
Example language: Go
No Go demo, but Go is one system where message passing in shared memory without immutability is promoted as the default solution.
Example Language: Rust
Example Platform: Erlang (aka Elixir)
We can eliminate data races entirely by making data all data immutable. Once an object is created, it cannot be changed.
Erlang programs are structured as a collection of lightweight "processes". Communication between processes is by message passing. Because data is immutable, it's safe to pass pointers to shared data as messages - although Erlang can also be run distributed across multiple machines, in which case messages are copied.
In the distributed context, mutation doesn't really make sense anyway. Mutating the local copy can't effect a remote copy of the "same" object.
This model is great for concurrency, and great for executing concurrently structured programs in parallel for a speedup. It's not the greatest for parallel speedup though - Erlang runs in an interpreter, and mutation tends to be pretty good for fast computation.
Erlang's main design goal is reliability. If some piece of the system crashes, another piece (potentially on another machine) can notice and restart it.
Example: elixir / demo.ex - startPrinter, startSender
Example Langauge: Clojure, a LISP on the JVM.
Like Erlang, it takes the immutability path to deal with concurrency, but instead of message passing it has a concept of "refs", which are mutable references to immutable data.
Refs can be updated transactionally. Rather than avoiding data races, transactions detect them and roll back / replay any transaction that ran on old data.
Transactions are the same strategy that databases use for concurrent updates.
Example: clojure / tmem.clj
Example Technology: OpenCL
OpenCL is a programming system for building programs that run on graphics cards. Graphics cards, or GPUs, are a bit different from regular CPUs. Rather than having one processor with a couple cores, they have a bunch of "processors", each with hundreds of "shader units". A shader unit is basically a single vector ALU - something that can execute arithmetic instructions on 4 or so values in parallel.
On a GPU, it's perfectly reasonable to plan to execute 2000 additions in parallel in one clock cycle.
The trick is that GPUs really like to perform the same operation in parallel. In fact, each individual processor can generally only load one program to run on its hundreds of shader units.
So instead of the basic addition operation adding together two numbers, on a GPU it adds together two arrays. The arrays generally represent mathematical vectors or matrices, but that's just a mental model. Anything where you want to operate on entire arrays at once will work great on a GPU.
This programming model of performing the same operation in parallel on many different values (elements of the array) is called data parallelism. It's required on GPUs, but it's common on supercomputers too. When you have a cluster of 1000 PCs, it's easier to think about them working together on one array calculation than to reason about them individually.