©Copyright 2004, NIIT and SLAC.
projects home maggie tools meetings presentations team docs




Measurement and Analysis for the Global Grid and Internet End-to-end Performance



Les Cottrell (PI) and Warren Matthews

Stanford Linear Accelerator Center (SLAC)


Andrew Adams

Pittsburgh Supercomputing Center (PSC)


Vern Paxson

ICSI Center for Internet Research (ICIR)


Brian Tierney

Lawrence Berkeley National Laboratory (LBNL)




We propose to integrate numerous network and application performance monitoring tools into a scalable and secure infrastructure providing measurements, analysis and access to data. We propose to incorporate and extend the existing IEPM-PingER and IEPM-BW infrastructures, and extend the NIMI configuration and control software. We will also incorporate tools whose development is already being funded by the DoE/MICS SciDAC initiative.



1.     Introduction. 3

2.     Developing NIMI. 3

3.     Publishing to Grid Middleware. 4

4.     Characterizing the Traffic Mix. 5

5.     Analysis, Reporting and Trouble-shooting. 5

6.     Security Considerations. 6

7.     Man-power 7

8.     Support 7

9.     Bibliography. 7



1.     Introduction.

Scientific research has greater computing and networking power than ever before, but it is recognized in the scientific community that the increases in power also bring with them major challenges. The deployment of Grid technologies and a production quality Grid Service requires detailed information on network performance. It is widely believed that what is needed is to develop a ubiquitous monitoring infrastructure that would not only provide the measurements seen in multiple monitoring projects but also provide the novel addition of allowing us to co-ordinate and integrate tools in a co-operative framework. 


The monitoring project described here is distinguished from other monitoring work in the following ways:






The proposal also complements other projects. For example, the Internet2 End-to-End performance initiative (e2epi) and their Performance Evaluation Station (PIPES) [PIPES] project has been designed to aid end-to-end trouble-shooting for Internet2 connected Universities. By working with the PIPES group to implement the MAGGIE publishing scheme we expect to increase our ability to gain insight to the networks used by the DoE science community.

2.     Developing NIMI.

The following enhancements will be made to the NIMI system and infrastructure.










3.     Publishing to Grid Middleware

Network monitoring data is particularly important to Grid middleware [Globus] such as the Replica Manager. Selecting the best source to copy the data from requires a prediction of future end-to-end path characteristics between the destination and each possible source. Accurate predictions of the performance obtainable from each source requires measurement of available bandwidth (both end-to-end and hop-by-hop), latency, loss, and other characteristics important to file transfer performance.


The following tasks are required to make this network monitoring information useful to Grid applications and middleware.


Perform a classification of various types of network measurement characteristics, and determine which measurement tools provide each characteristic






Many of these tasks will be done working closely with the Global Grid Forum, and we will work to standardize this work to enable interoperability between all Grid projects.






4.     Characterizing the Traffic Mix

If demand arises, we will evaluate the use of passive monitoring techniques within the MAGGIE framework. The participants in this project are experienced with various tools, including





5.     Analysis, Reporting and Trouble-shooting

Making use of measurements to assist with engineering, grid-middleware, trouble-shooting, setting end-user expectation and a myriad of other tasks is the reason monitoring is performed. We will undertake the following to contribute in those areas.








6.     Security Considerations

Security is a primary concern of the NIMI architecture. The Access, Authorization and Authentication aspects of NIMI are already being funded and is not part of this proposal. However, the development of the packet daemon for policy enforcement and policing is part of this proposal and provides extra security. Security in terms of publishing the data is also a serious consideration. The developers will work with the Global Grid Forum to implement standards and enable control of the data.

7.     Man-power

We estimate the following full time effort (FTE) to complete the tasks described in this document:






Total is 3 FTE.

8.     Support


Deploying and testing network performance tools at high performance ESnet and ESnet collaborator sites will greatly assist in understanding and validating the scaling properties of the proposed tools/algorithms.


CAIDA supports this proposal and would like to collaborate with the MAGGIE team to assist in deploying, integrating and configuring the tools in their measurement probes. CAIDA will also assist in adapting their existing visualization tools.

9.     Bibliography

[PingER]          http://www-iepm.slac.stanford.edu


[IEPM-BW]     http://www-iepm.slac.stanford.edu/bw


[NIMI]             http://www.ncne.nlanr.net/nimi


[PIPES]           http://e2epi.internet2.edu


[SciDAC]         http://www.osti.gov/scidac


[Grid]               http://www.gridforum.org


[SOAP]           http://www.w3.org/TR/SOAP


[WSDL]           http://www.w3.org/TR/wsdl


[GMA]             http://www-didc.lbl.gov/GGF-PERF/GMA-WG/


[SNCM]          http://www-itg.lbl.gov/Net-Mon/Self-Config.html


[Bro]                http://www-nrg.ee.lbl.gov/bro-info.html


[CAIDA]         http://www.caida.org


[trends]            http://www.caida.org/projects/trends/