



             The  GENESIS  Distributed-Memory  Benchmark  Suite


                              Release  2.0



               Southampton Novel Architecture Research Centre
                     (c)  University of Southampton



                                   May, 1992



1    Introduction



The GENESIS Benchmark Suite has been developed to fulfill a need for bench-
mark programs that can be used to evaluate the performance of distributed-
memory MIMD systems on scientific and engineering problems [1]. The bench-
marks were written within ESPRIT Project P2702 - GENESIS by a number of
project partners including Liverpool and Southampton Universities, PALLAS
GmbH, the European Centre for Medium Range Weather Forecasting and the
Universitat Politechnica de Catalunya.  They have been prepared for public
release by Southampton Novel Architecture Research Centre.


The Genesis benchmarks are written as far as possible in standard FOR-
TRAN77 using in most cases DOUBLE PRECISION (8 byte) floating point arith-
metic, but also REAL (4 byte) floating point arithmetic (LPM1 and MD1 bench-
marks).  Both sequential and distributed versions exist with the PARMACS
macros (see section 2) being used to express the necessary extensions to FOR-
TRAN77 for task creation and message passing in the distributed versions.


Unix make-files are supplied with each benchmark to handle compilation and
linking plus expansion of the PARMACS macros.  Versions are also available
for non-UNIX systems.  A description of each benchmark and instructions on
how to run it are given in individual ReadMe files.


The GENESIS distributed-memory benchmark suite is free software developed
for academic purposes.  The authors take no responsibility for maintenance
and support, though the Southampton Novel Architecture Research Centre
will continue developing existing benchmarks and incorporating new ones into
the suite as appropriate. A steering group is being set up with representatives
from both Europe and the USA to advise on the development of the suite. If
you have received the first public release (2.0), you should also receive the 
next version(s) without special arrangements.  We would very much appreciate
receiving measurements, obtained with the GENESIS benchmarks, as well as any
problems or queries concerning the suite. Correspondence should be addressed
to Vladimir Getov, Roger Hockney or Ivan Wolton at:



                                        1


Department of Electronics and Computer Science
University of Southampton
Southampton
S09 5NH
England


Email : icw or vsg @ecs.soton.ac.uk



2    The PARMACS macros



The distributed version of the benchmarks use the PARMACS macros [2] to
provide portability with respect to basic message passing and task creation.


The PARMACS macros, developed at GMD and Argonne National Labora-
tories, allow the same FORTRAN code to be run without modification on a
number of parallel systems. A package of macros is used to express functions
such as process creation, message passing and synchronisation. These macros
are invoked from the user FORTRAN program and passed through the stan-
dard UNIX "m4" macro expander, which replaces the macros with FORTRAN
text appropriate to the system on which the program is to be run. Some macros
expand into a set of FORTRAN statements whereas others expand into calls
to library functions, which are supplied with the PARMACS package for each
system.  In the UNIX version of the suite expansion of the macros is handled
by the make-files.


The basic programming model is based on message passing but this can easily
be adapted to allow implementations on shared-memory machines as well as
distributed-memory systems. Implementations currently exist for the following
machines:



    o Cray Y-MP


    o iPSC/860 & iPSC/2


    o Meiko CS (i860 & transputer)


    o nCUBE 2


    o Network of Suns (using CS Tools)


    o Network of RS/6000s (using PVM)


    o Parsytec


    o SUPRENUM



Future implementations planned include the Alliant FX/2800. The implemen-
tations are built on top of the native message passing software for each 
machine, but in most cases the communications overheads associated with
the extra layer is small.  Benchmarks are provided to measure communications
parameters from PARMACS, these can easily be adapted to use native message



                                        2


passing for comparison with the values obtained from PARMACS. Alternatively
the authors of the benchmark suite have comparisons for various systems.


As of April 1992, the PARMACS macros have been used to run the benchmarks
on Intel iPSC/860, Meiko i860 Computing Surface, nCUBE 2 and SUPRENUM
machines.


Note: The current version of the benchmark suite uses version 5.0 of the PAR-
MACS macros, this will be upgraded to version 5.1 in the near future. Programs
based on version 5.0 of PARMACS are compatible with version 5.1. The most
important feature of version 5.1 is the addition of the MSG_FORMAT macro
which handles different data representations when sending messages between
host and node.  The use of this macro is essential on machines such as the
nCUBE 2 where the data representation on the host is different from that on
the nodes.



3    Organisation and Contents of Benchmark Suite



3.1    Currently Available Benchmarks



The benchmarks are divided into three categories according to their level of
complexity/realism as follows:



   1. Synthetic code fragments. These measure basic machine parameters
      connected with communication, synchronisation and vectorisation.


      TICK1    Measurement of clock tick

      RINF1    Measurement of vectorization parameters r1  and n1=2

      COMMS1   Pingpong benchmark

      COMMS2   Message exchange benchmark

      SYNCH1   Barrier synchronisation cost


   2. Application kernels.  These are larger, more complex codes that rep-
      resent the computationally intensive kernels common to a variety of dif-
      ferent applications. Analysis of the benchmark performance is generally
      possible in terms of the basic machine parameters. They provide valuable
      insight into the likely behaviour of full applications.


      TRANS1   Matrix transpose using naive algorithm

      FFT1     One-dimensional FFT

      QCD2     Conjugate gradient kernel from QCD

      PDE1     Three-dimensional Poisson solver using red-black relaxation

      PDE2     Two-dimensional multi-grid Poisson solver


   3. Full application codes.  These are typical of medium-sized research
      codes where effects due to non-parallelizable sections of code, and memory
      or I/O bottlenecks may become important.



                                        3


      QCD1     Quantum chromodynamics simulation using a Monte Carlo method

      GR1      General relativity application

      LPM1     Electronic device simulation

      MD1      Molecular dynamics code



Further codes to be added in the next release include: Matrix multiplication,
LU-decomposition, QR-decomposition and a Helmholtz solver for meteorology.



3.2    UNIX Directory Structure



The /genesis/ benchmark suite (Rel.2.0) is available as a single UNIX directory
structure.  Apart from the benchmarks themselves, the top level directory in-
cludes this ReadMe file and two auxiliary directories - Incl and Lib. The Incl
directory contains various make-files, whilst the Lib is contains precompiled
and non-standard FORTRAN libraries. Every benchmark directory contains a
ReadMe file and reference output files in the StdRes directory.  The standard
FORTRAN77 code for single-node execution is placed in the s77 directory and
the distributed memory code (FORTRAN77 + PARMACS extensions) is to be
found in the d77 directory.



4    Installation Procedure



In order to run the distributed version of the benchmarks, it is necessary that
the PARMACS macros [3] are installed on the system to be benchmarked. Before
running individual benchmarks, the user has to set a few parameters connected
with the installation of PARMACS on the system in question. There is also a
set of machine-specific routines which are called by all the benchmarks to
supply information such as current date, elapsed time, system's details, etc.
Suitable versions of these need to be modified for the machine in question
and compiled into a library for linking with individual benchmark codes.


The first step before running the benchmarks is to edit the file Incl/make.def,
which contains information on the installation of PARMACS on the system
under evaluation.  This file is referenced by the individual make-files for each
benchmark.  The user has to specify three system specific macro definitions
- PARALLEL_PLATFORM, FRONT_END, PARALLEL_RESOURCE and the path
to the PARMACS base-directory (USR_PALLAS). The definition of PARAL-
LEL_PLATFORM specifies the general programming model.  It can be ipsc,
cstools,  suprenum,  etc.  The macro FRONT_END specifies the system on
which the PARMACS host processor runs.  This can be sun4OS4, srm, etc.
The macro PARALLEL_RESOURCE defines the hardware architecture on which
the node-processes are to be executed, e.g. 860p, sun4OS4, x8, etc. The path
to the PARMACS base-directory assigned to USR_PALLAS macro is usually
/usr/pallas, but it can be different in some installations. Information on the
use and installation of the PARMACS macros is contained in [3].


The second step is to modify the machine specific routines contained in the
directory Lib/SUITELIB and compile the modified routines into a library for



                                        4


linking with individual benchmarks. The machine specific routines are setdtl,
setdat & timer. Source files for these routines are provided for Intel iPSC ma-
chines, systems using Meiko's CsTools & the SUPRENUM. For other systems these 
routines will need to be supplied by the benchmarker.  The setdtl routine
will require modification for all systems, as this writes installation specific
information to the output file concerning such details such as compiler and
operating system version. The remaining routines are:



    o timer - timing routine to return the elapsed time in seconds since
	      its first call.


    o setdat - returns current date and time



For the systems listed above the appropriately modified source files are then
compiled into libraries to be linked with the host and node programs by entering
the Lib/SUITELIB directory and typing:


make [front-end] ([parallel resource])


The [parallel resource] has to be specified in the above command, if the
current definitions for FRONT_END and PARALLEL_RESOURCE in the make.def
file are different.



5    Compiling and Running the Benchmarks



Compiling the benchmarks is as simple as typing (gnu)make in the directory
containing the benchmark.  The same command with an argument clean re-
moves all executable files and directories. On some systems such as the iPSC
an adequate make mechanism is not available and gnumake must be used in-
stead.  Gnumake will normally be supplied with the PARMACS macros if its
use is necessary on the system in question.


In some of the benchmarks the problem size or number of processors is changed
by editing PARAMETER statements in an include file. In this case the routines
referencing the include file will need to be recompiled after every change. The
relevant files are identified automatically in the makefile and so using make
after any such change will recompile only the appropriate files.


The input/output unit numbers are also assigned by PARAMETER statements,
allowing these to be adjusted easily if necessary.


On some systems it may be necessary to use system-specific commands to
reserve the appropriate number of processors before running the benchmark,
eg. getcube on the iPSC. It may also be necessary to set certain environmental
variables such as on the Meiko Computing Surface where the clock-tick time
and the type of processor configuration can be set by such variables.  These
details are best determined from somebody with knowledge of the system in
question.


The suggested procedure for running the benchmark suite is as follows:



                                        5


   1. Run synthetic code fragments

      The TICK1 benchmark should be run first to check the resolution of
      the system clock.  The most accurate clock available should normally
      be used. To compensate for insufficient resolution, most of the synthetic
      code fragments and applications kernels are timed over a repeat loop. The
      number of repeats being specified by the user. If the benchmark timing
      is comparable with the clock tick-time the number of repeats should be
      increased.  Normally the measured time in the benchmark should be
      greater than 100 clock ticks so as to give at least 1% accuracy.

      The remaining synthetic code fragments can then be run to measure per-
      formance parameters related to vectorization, communication and syn-
      chronisation [4]. These can be useful in interpreting the results obtained
      from the more complex benchmarks.


   2. Run sequential FORTRAN77 versions of kernels and applications

      Measure single node performance over a range of problem sizes upto the
      maximum possible on a single node.  Also investigate effect of compiler
      options and any vectorization facilities on single node performance.

      Note: It is considered strongly unfair to use non-standard precision com-
      piler flags.


   3. Run distributed FORTRAN77 benchmarks

      Use optimum compiler options determined from single node measure-
      ments. To obtain a clear idea of how the benchmark performance varies,
      measurements should be made over a range of problem sizes for a different
      numbers of nodes.



Two output files are produced in the directory containing the executable. The
first is on the standard output and is used to monitor the progress of the run.
The second file is intended to be kept as a permanent record of the benchmark
run.  This file usually called result is overwritten by subsequent runs of the
program and so should be copied to another suitably named file before the next
run.  Samples of the result files produced by the sequential and distributed
versions of all benchmarks are included in the respective StdRes directories.



6    Methodology of Measurement and Interpretation



Benchmark measurements should be made where possible on a quite system with 
only one user. If this is not possible the fact should be carefully noted and
more care than normal should be taken to ensure that the measured results are
reproducible. Intensive I/O operations or the loading of jobs by other users
can affect the communications performance on some systems. While performance
degradation resulting from this effect is of some interest it is important
first to be able to obtain reproducible baseline measurements in which such
effects are absent.



                                        6


6.1    Time Measurement



The fundamental measurement made in any benchmark is the time to complete
some specified task.  All other performance figures (e.g.  Mflop/s) are derived
from this basic timing measurement. It is a principle of the GENESIS bench-
marks [5] that these timing measurements must either be published as tables
of numbers with at least three significant decimal figures, or be derivable to
the same accuracy from performance tables, using a given formula.


The benchmark time, T(N;p), will be a function of the number of processors, p,
and the problem size, here represented by the vector variable, N , which stands
for a set of parameters characterising the size of the problem, e.g. the number
of mesh points in each dimension, and the number of particles in a particle-
mesh simulation.  Benchmark problems of different sizes can be created by
multiplying all the size parameters by suitable powers of a single scale factor,
thereby increasing the spatial and particle resolution in a sensible way.



6.2    Floating-Point Operation Count



In order to express benchmark performance in millions of floating-point oper-
ations per second, it is necessary to assess the number of floating-point op-
erations, F (N ), needed to solve the benchmark problem.For simple problems
(e.g. matrix multiply) it is sufficient to use a theoretical value for the 
floating- point operation count (in this case 2n3 flop, for nxn matrices)
obtained by inspection of the code or theoretical consideration of the
arithmetic in the algorithm. For more complex problems containing data-
dependent conditional statements, an empirical method may have to be used. 
The sequential version of the benchmark code defines the problem and the
algorithm to be used to solve it.  Counters can be inserted into this code
or a hardware monitor used to count the number of floating-point operations.
In either case a decision has to be made regarding the number of flop that 
are to be credited for different types of floating-point operations, and we
see no good reason to deviate from those chosen by McMahon when the Mflop/s
measure was originally defined. These are:


                        add, subtract, multiply    1 flop
                        divide, square-root        4 flop
                        exponential, sine etc.     8 flop
                        IF(X .REL. Y)              1 flop



We distinguish two types of operation count.  The first is the nominal bench-
mark floating-point operation count, FB (N), which is found in the above way
from the defining Fortran77 sequential code.  The other is the actual number
of floating-point operations performed by the hardware when executing the
distributed multi-node version, FH (N), which may be greater than the nom-
inal benchmark count, due to the distributed version performing redundant
arithmetic operations.



                                        7


6.3    Performance Metrics



Given the time of execution T(N;p) and the flop-count F(N) several different
performance measures can be defined. Each metric has its own uses, and gives
different information about the computer and algorithm used in the benchmark.
It is important therefore to distinguish the metrics with different names, sym-
bols and units, and to understand clearly the difference between them.  The
principal performance metrics are:



6.3.1   Temporal Performance



If we are interested in comparing the performance of different algorithms for
the solution of the same problem, then the correct performance metric to use is
the Temporal Performance, RT , which is defined as the inverse of the execution
time
                             RT (N ; p) = T -1(N ; p)                        (1)

The units of temporal performance are, in general, solutions per second (sol/s),
or some more appropriate absolute unit such as timesteps per second (tstep/s).
With this metric we can be sure that the algorithm with the highest perfor-
mance executes in the least time, and is therefore the best algorithm. We note
that the number of flop does not appear in this definition, because the objec-
tive of algorithm design is not to perform the most arithmetic per second, but
rather it is to solve a given problem in the least time, regardless of the
amount of arithmetic involved.



6.3.2   Benchmark Performance



In order to compare the performance of one benchmark with another, account
must be taken of the different amounts of work (measured in flop) that the dif-
ferent problems require for their solution. Using the flop-count for the bench-
mark, defined in the last paragraph, we can define the Benchmark Performance
as
                          RB (N ; p) = FB (N )=T (N ; p)                     (2)

The units of benchmark performance are Mflop/s (benchmark name), where
we include the name of the benchmark in parentheses to emphasise that the
performance may depend strongly on the problem being solved, and also to
emphasise, that benchmark Mflop/s may be nominal (rather than actual) val-
ues.



6.3.3   Simulation Performance



A special case of temporal performance occurs for simulation programs in which
the benchmark problem is defined as the simulation of a certain period of
physical time, rather than a certain number of timesteps. In this case we speak
of the simulation performance and use units such as simulated days per day
(written sim-d/d or 'd'/d) in weather forecasting where the apostrophe is used



                                        8


to indicate 'simulated', or simulated pico-seconds per second (written sim-ps/s
or 'ps'/s) in electronic device simulation.  It is important to use simulation
performance rather than timestep/s if one is comparing different simulation
algorithms which may require different sizes of timestep for the same accuracy
(for example an implicit scheme that can use a large timestep, compared with
an explicit scheme that requires a much smaller step).  For such schemes the
simulation performance falls off dramatically as the problem size is increased
by introducing more meshpoints in order to refine the spatial resolution.



6.3.4   Presentation



In the GENESIS benchmarks, the performance is presented both in tabular
form and as graphs of R(N;p) for constant N  plotted against p on log/log
axes.  The use of logarithmic axes allows the inclusion of a wide range of
processor number and performance on a single graph. Within one benchmark, it
also permits the vertical axis to be interpreted as either time or performance
by reading the appropriate scale. In this respect we plot the performance the
right-way-up with the highest performance at the top not the bottom.



6.4    Optimisation Issues



The motivation behind the development of the GENESIS benchmark suite has
been to assemble benchmarks for a range of important scientific and engineering
problems, that could be used to evaluate the relative strengths and weaknesses
of distributed-memory machines.


When assessing the strengths and weaknesses of a design we are not only in-
terested in the optimum performance that can be achieved for a particular
application, but also in the performance that can be obtained from the initial
port of a code, the subsequent optimization that can be performed, and how
difficult it is to implement this optimization. The last assessment must invari-
ably be subjective and will depend on the skill of the benchmarker, as will the
degree of optimization obtained.


The position of the authors of the suite on optimisation is thus similar to that
of the PERFECT club: optimisation of the benchmark codes for a given system
is permitted and even encouraged but the measurements for the standard un-
optimised benchmark should always be made and published with the optimised
measurements. The PERFECT club idea of quantifying the effort involved in
the optimisation is viewed as desirable but impractical.



References



[1] A.J.G. Hey. The GENESIS distributed memory benchmarks. Parallel Com-
    puting, 17(10 & 11):1275-1283, 1991.



                                        9


[2] GMD mbH. The Argone/GMD Macros in FORTRAN for Portable Parallel
    Programming using the Message Passing Programming Model, April 1991.
    Version 5.0.


[3] PALLAS GmbH. PARMACS - Geting Started, October 1991.


[4] R.W. Hockney.  Performance parameters and benchmarking of supercom-
    puters. Parallel Computing, 17(10 & 11), 1991.


[5] C. Addison, J. Allwright, N. Binsted, N. Bishop, B. Carpenter, P. Dal-
    loz, D. Gee, V. Getov, A.J.G. Hey, R. Hockney, M. Lemke, J. Merlin,
    M. Pinches, C. Scott, and I. Wolton.  The GENESIS distributed memory
    benchmarks - I: Methodology and general relativity benchmark with re-
    sults for the SUPRENUM computer. Concurrency: Practice & Experience,
    4, 1992. (to appear).



                                       10
