With the continuing relevance of PC-clusters, SMP-clusters and the development of chip architectures into memory hierarchies and internal parallel processing, computer architectures have become more and more complex. Parallelism and the growing capacity of memory and storage has led to an increase of the growth of today's problems.

At present we all miss reliable system benchmarks (other than the Linpack performance used in TOP500), which support the user in rating and obtaining computer systems - especially parallel systems. Existing parallel benchmarks, like the NAS parallel benchmarks, are no longer meaningful these days. Applications, which are not numerical and yet using parallel systems, are still not covered. With the growing relevance of computing grids the situation will even become more aggravated in a few years.

In cooperation with colleagues at LBNL / NERSC, IPACS wants to define a new basis for benchmarks measuring system performance of distributed systems. The following topics are the main work packages in IPACS:

• Development and propagation of scalable, portable and realistic benchmarks,
• Development of methods of measurement for parameters which are specific to applications and hardware,
• Prediction of performance of commercial codes and
• Tools that support the user.

The goal here is to develop methods of measurement based on new applications, low level, and grid benchmarks, which also allow performance prediction.

A further significant element of the new benchmark environment is represented in the usability of the benchmarks. Mechanisms for the automatic installation and execution of benchmarks are being strived for.

With the help of grid-adaptive benchmarks it should be possible to check if an existing grid infrastructure is suited for a certain application. Another intent of grid benchmarks is to develop technologies for grid-adaptive applications and to put them at user's disposal. These different approaches should result in a simple, but salable benchmark, which might have the potential to supersede the Linpack benchmark. These developments should also lead the user in tools for optimizing performance.

The goal of the IPACS project is the development and propagation of new benchmarks with the following characteristics:

• Scalable: The size of the problem is adaptable to a wide class of computer systems and should last for the next ten years (petaflop systems)
• Portable: The benchmarks should be public, the installation and the execution of the benchmarks should be made very easy.
• Realistic: The benchmarks should reproduce real applications very well; they should take particularly the influence of system components like memory hierarchy, I/O system and the different medians of communication (grids) into consideration. Applications, which are not numerical should be taken into account as well.

In addition, IPACS wants to build the foundation, so that users, especially industrial users, are able to predict easily the system performance of commercial applications. The benchmarks can be classified into three groups:

• Open source benchmarks for industrial applications, particularly for the currently neglected area of non-numerical applications.
• Low-level benchmarks (like B_eff) to characterize the state of the computer or the grid.
• Scalable benchmarks for commercial codes

Based on a component architecture the open source benchmarks will be able to cooperate within a grid benchmark. The gain of experience will be used to a further development of tools for performance analysis and their adaption for grid applications. A further significant element of the new benchmark environment is represented in the usability of the benchmarks. Based on an object oriented model, mechanisms for the automatic installation and execution of benchmarks are being strived for. The benchmark client will enable the control of this process through interaction with the user, to load components of benchmarks from the repository and to publish the results automatically on a connected web server. This should lead to a better acceptance of these benchmarks. In conjunction with the grid benchmarks and the MDS service of Globus, the representation of performance states of a grid will be possible.