Parallel Architectures

Multiple instruction streams, multiple data streams (MIMD)
Multiprocessors
1. Computers consisting of tightly coupled processors whose coordination and usage are generally controlled by a single operating system. The processors share memory through a shared address space.
  1. Multiple processors tightly coupled
  2. Single Operating System
  3. Shared memory through shared memory space
2. OBS: Multiprocessor is different from multicore.
Memory Organization
1. Multiprocessors may share:
  1. cache memory, main memory, and I/O system
  2. main memory and I/O system
  3. I/O system
  4. nothing, usually communicating through networks
  5. Are all these options interesting?
    1. Note that it is important to avoid bottlenecks in the architecture. And the answer to this question also depends on the project requirements that have to be fulfilled
      1. Avoid bottlenecks
      2. It depends on the project requirements
2. Classification
  1. Symmetric (shared-memory) Multiprocessors (SMP) or Centralized shared-memory Multiprocessors
    1. Characteristics
      1. ≈ 32 cores or less
      2. share a single centralized memory where processors have equal access to.
        symmetric
      3. memory/bus may become a bottleneck
        Large use of caches and buses to avoid the bottleneck
      4. uniform access time (UMA) to all the memory from all the processors
        UMA
  2. Distributed Shared Memory (DSM)
    1. Characteristics
      1. 16-64 processor cores
        Large number of processors
      2. Distributed memory
        Increase bandwidth
        Reduce access latency
      3. More complex communication between processors
      4. More software effort to take advantage of the increased memory bandwidth
      5. Distributed I/O
        Each node can be a small distributed system with centralized memory
      6. Nonuniform memory access (NUMA). Access time depends on the location of a data word in memory.
        NUMA
Memory Architecture for
1. SMP
  1. processors share a single memory
  2. uniform access times to the memory (UMA)
2. DSM
  1. processors share the same address space, but not necessarily the same physical memory
  2. nonuniform memory access (NUMA)
3. Multicomputers
  1. processors with independent memories and address spaces
  2. communicate through interconnection networks
    1. may even be complete computers connected in network, i.e., clusters
Communication models for
1. SMP
  1. SMP Communication Model: There is an implicit memory communication (memory access). In SMP, with a central memory, it is possible to use the threads and fork-join model. One alternative is the application programming interface named open multi-processing (OpenMP).
    1. Implicit memory communication
      1. Memory access
    2. Central memory
    3. Threads and fork-joins
      1. Ex.: OpenMP
      2. Thread (lightweight process)
        Threads are lines of execution within a process and they share the address space of the process to which they belong.
      3. Fork-join model
        A process creates a child process (thread), by using fork. A process (parent) and its threads (children) share the same address space. Next, the process waits for their threads to finish their computation by calling join. Creating a process is expensive to the operating system, and that is one reason to use different threads to perform a computation.
2. DSM
  1. DSM Communication Model: In DSM, with a distributed memory, it is possible to make use of the message passing model. There is a library project implementing a message passing interface (MPI). Explicit communication is in place (message passing). This solution brings also synchronization problems.
    1. Distributed memory
    2. Message passing model
      1. synchronization problems
      2. Ex.: Message Passing Interface (MPI)
    3. Explicit communication (message passing)
Market share for
1. SMP
  1. biggest market share, both in money and in units
  2. Ex.: Multiprocessors in a chip.
2. Multicomputers
  1. Ex.: Clusters for systems on the Internet.
  2. Ex.: > 100 processors (massively parallel processors MPP)
Observations about SMP
1. Large and efficient cache systems can greatly reduce the need for memory bandwidth
2. Not much extra hardware is needed
3. Based on general purpose processors (GPP)
4. Base question for cache coherence problem: Caches not only provide locality but also replication. Is that a problem?

Next up

Parallel Architectures

Description

Resource summary

Media attachments

0 comments

Similar

	Created by Artur Assis about 3 years ago