Project 72: Testing Numerical Software

Purdue 1999:
============

Gaffney presented a proposal for refereeing and publishing numerical
software. He advocated that real users do the testing and evaluation
of submitted packages and the refereeing would be monitored and
reported by newsgroups. The model would be to `publish' and then
`referee'. There would have to be a mechanism where a. newsgroup would
collect referees' reports (over time) and eventually accept/endorse
the package.

Testing Hamiltonian Solvers
---------------------------

Gladwell discussed some of the special difficulties that arise when
one attempts to evaluate and compare numerical methods designed for
long time integrations of special second order initial value problems.

Ottawa 2000:
============

Although there was no report on this project there was some discussion
of this project. Boisvert indicated that ACM TOMS would be willing to
listen and cooperate in assessing its Algorithms. Others expressed
some concerns with the `publish then referee' plan and noted that some
new metric for assessing `quality' may be required.

Amsterdam 2001:
===============

Although there was no report on this project there was a lively
discussion on the need and/or support for results in this area. Ford
pointed out that there is a renewed interest arising from recent court
cases and there is now likely to be more interest in certification of
numerical software. Pool pointed out the ned for more software testing
in large scale projects such as the ASCI project.  Gaffney promised to
prepare a report for our next meeting.

Portland 2002:
==============

Although there was no report on this project there was a lively
discussion of the related legal difficulties that are arising. Testing
and reporting comparisons can result in litigation. For example MS
licences forbid the publishing of `benchmarking' results. Nevertheless
it is important to continue testing and the investigation of where
existing software is likely to be unreliable.

Strobl 2003:
============
Ford pointed out the new incentives for measurement and analysis of
performance that has resulted from recent litigation which recognized
inadequate testing of software. This is an active area.

Washington 2004:
================

This is an active area and Ford noted the need for testing new
versions of software and programs.  He noted, in particular, the work
in the area of meteorology where one had to deal with uncertainty in the
measurements.

Prescott 2006:
==============

This is an active area with several members of the group engaged in
related research. The talks by Gaffney on "Virtual Manufacturing - The
Vision for Virtual Paint Operations" and by Smith on "A Test Harness
TH for Numerical Applications and Libraries" at the working conference
and the one by Enright on "An investigation of the
Reliability/Efficiency Trade-off in ODE Solvers" at the working group
meeting are examples of reports of activity in this area.

Uppsala 2007:
=============
  
This is an active area with several members of the group engaged in
related research. Several members requested that their names be added
to the list of participants in this project. Some discussion continued
on the suitability of this topic to be the theme of the next Working
Conference.

Toronto 2008:
=============

This is an active area with several members of the group engaged in
related research. Enright's workshop talk "Developing easy to use ODE
Software" is a report on one component of this activity. It was
pointed out that this will likely be an important component of WoCo 10.
 
Raleigh 2009:
=============

Wilfried Gansterer's presentation "Case Studies in Trading Accuracy
for Performance in Numerical Software" and Brian Smith's "3D Magnetic
Field Modeling Using The Boundary Element Method" at the "IFIP Working
Group 2.5 Symposium 2009" are reports on this project.

This is an active area with several members of the group engaged in
related research. Enright agreed to provide an updated paragraph.

Leuven 2010:
===========

The talk by Tim Hopkins "The collected algorithms of the ACM - past,
present and future" at ICCAM 2010 is a contribution on this topic.

Pat Gaffney and others will talk on this subject at WoCo10 in Boulder,
2011.

Boulder 2011:
=============

This is an active area with several members of the group engaged in
related research. The talks by Smith and Enright at the associated
WoCo can be considered reports of different aspects of this project.

Santander 2012:
===============

The papers by Dan Lozier "DLMF Tables: A New Source of Data for 
Mathematical Software Developers" and Annie Cuyt "Validated Evaluation
of Special Mathematical Functions" at the workshop can be considered
reports of activities related to this project.