Based on existing codes, practical and scalable benchmarks for areas like fluid dynamics, image processing and others will be developed as open source codes. The scalability to run huge problems using the complete system environment is a major characteristic for these benchmarks. They will help to gain information about the usage of the memory hierarchy, the network, the CPU and the I/O system. The intent is to understand the sources of performance problems.

The purpose of low-level benchmarks is the evaluation of characteristics (signatures) of computer systems. Included are compute benchmarks (LINPACK), communication benchmarks (B_EFF), I/O benchmarks as well as benchmarks which characterize the underlying memory hierarchy. To use these benchmarks within the intended architecture of components (grid benchmark) the I/O interfaces have to be taken particularly into consideration. There is a separate working package, which deals with I/O benchmarks. The decisive factor is, that low-level benchmarks lead to a characterization of computer systems.

Up to now the evaluation of distributed computer systems has neglected the I/O perfomance. Therefore concepts for measuring and evaluating the I/O performance of distributed computer systems will be developed. These concepts will build the basis for the development of measuring methods (if necessary with the use of MIP I/O interfaces), which should characterize computer systems as low-level benchmarks. I/O concepts in the area of PC-clusters and computing grids are still not matured thus current developments have to be taken into account.

Scalable benchmarks will be developed for a variety of commercial applications (Fluent, Magmasoft, LS-Syna, etc.). The intent is to evaluate software-specific characteristics (e.g. data flow-rate), which are independent of manufacturers, and to put them at industrial user's disposal. At least two software packages of different areas should be treated. The validity of the benchmarks should be verified within a real production environment in cooperation with the users. Hence, the work is focussing at first on the conception of benchmark examples. The test of the software manufacturers will build the foundation of the benchmarks.

The results of the performance and the analysis of the previous working packages represent the context between the characteristics of the computer systems and of the applications. The goal of this working package is to collect enough information to picture this context in a mathematical manner with the use of system identification methods. At first, models of neuronal grids will be used. The purpose is to make predictions for the tested commercial software packages based on the low-level benchmarks only. The results of this working package should lead to a performance prediction tool for commercial software.

Within the framework of this working package a grid benchmark as well as the technology for grid-adaptive applications will be developed. The task of grid benchmark is to analyze and to parametrize the grid environment at first, followed by the customization and the execution of the benchmark. The benchmarks, which have been developed within the first three working packages are available from a benchmark repository and will be used from the benchmark infrastructure under development. These benchmarks are a basis for the further development and optimization of grid products.

The execution of benchmarks should be simplified. Therefore a benchmark client will provide the essential configuration tasks. The client connects to a web-based repository, which contains all benchmarks as source code and as binaries for a variety of architectures. The purpose of the client is to detect the local configuration of the computer in interaction with the user, to determine the type of test, to load the necessary sources and binaries and finally to publish the benchmark results on the web server.

This new approach of the execution of performance analysis should substitute the current tests, which last several weeks. It will also provide the user with detailed system analysis and performance prediction, not only MFLOP values.

Besides the development of application benchmark, the development of the benchmark environment is the most important working package within the IPACS project. The evaluation of the system configuration, the generation of the locally executable binaries and security matters makes this work so difficult.

Today's leading analysis tools for parallel applications, Vampir and Vampirtrace, are supplied by the IPACS project partner Pallas. Vampir provides a basic analysis of the communication of the inspected software. The purpose of this working package is to provide a complete analysis of the software, so that the trace files will contain all relevant information. The analysis of the benchmark measurements will be automized and standardized.

The execution of benchmarks with the use of the developed software is a necessary and essential part of the IPACS project. Existing computer systems of the project partners will be used as well as relevant external systems. Therefore the cooperation with the American groups will be very advantageous. The early inclusion of users should lead to a higher acceptance and quality assurance. Moreover, substantial contributions of the performance data should be provided by the users.

The results of the third working package (I/O benchmarks) will lead straight to suggestions of an approvement of the I/O performance of today's systems. Associative and adaptive storage concepts should be evaluated on the basis of the benchmark analysis of existing storage concepts. A prototype will be implemented on a PC-cluster.