Distributed memory in brief
A distributed-memory machine is a set of processes, each with its own private memory, connected by a network. Nothing is shared, so data moves between processes only by explicit messages. The model scales from a handful of cores on one machine to thousands of nodes, which is why it dominates large-scale scientific computing.
Ranks and communicators
Each process in a message-passing program has a rank — an integer identifier — and belongs to a communicator, the group within which it can exchange messages. Programs branch on rank to give each process its own slice of the work, and point-to-point calls name the partner by rank within a communicator.
Point-to-point communication
The primitives are send and receive: the sender names the destination rank and the data to move; the receiver names the source rank and where the data should land. The pair of calls must match — same communicator, same message — and the data types on both sides must agree. Point-to-point exchanges implement the patterns that dominate real codes, such as each process swapping boundary data with its neighbors.
Collective operations
Collectives involve a whole communicator at once and are heavily optimized by implementations:
- Broadcast — one process sends a value to every other process.
- Reduction — every process contributes a value, and one operation (sum, maximum, and others) combines them into a single result.
- Gather and scatter — data is collected from all processes into one, or distributed from one to all.
Using a collective instead of a hand-built loop of point-to-point calls is both clearer and typically faster.
Where message passing fits
The Message Passing Interface (MPI), first standardized in the mid-1990s, is the widely used specification for this style, and its Fortran bindings make it a natural companion to the language. Message passing is the right tool when the problem outgrows one machine, when memory must scale with the number of processes, or when the code must run on systems where only the network connects the parts. Coarray Fortran shows the language's built-in alternative.