Beetest

A script for functional testing of ab initio calculations

Introduction

The Beetest program was forked out of a test suite for VASP installations that I started develop while in Uppsala. It was originally designed as a tool designed to run VASP calculations and analyze the output. But the system is very flexible, so it has found use in other programs as well (RSPt). In general, beetest can be used to construct test suites for any kind of binary that reads input files and produces output files, i.e. typical batch jobs.

Prerequisites:

• Python >= 2.6
• A Unix-compatible system (the program has never been tested on Windows)

Get the latest version: 0.6 (September 2011), by cloning the git repository, i.e.:

git clone git://www.nsc.liu.se/~pla/beetest.git

Inside the folder, you find the actual program, which is called “beetest”. Copy this file to a directory in your $PATH, e.g. /usr/local/bin.

Usage:

The folder structure of a typical test suite using beetest looks like this:

TestSuite/
  |---Tests/
  |     |---Cu-fcc/
  |     |---Fe-bcc/
  |     |---Fe-bcc/
  |     |---Si-cd/
  |     |---TiO2-rutile/
  |---Scripts/
  |     |---Energy.sh
  |     |---Pressure.sh
  |     |---Volume.sh
  |     |---Forces.sh
  |
  |---beetest.xml

To construct a test suite, one would follow these steps:

1. Put some folders with input files into a folder called “Tests”. These are your “tests” / “jobs” / “calculations”. Make sure that they are set to start from scratch every time (e.g ISTART=0 for VASP).

2. Write some scripts to extract interesting results from your output files (energies, forces etc). Place these in a folder called “Scripts”.

3. Make an XML file defining your test suite. See “beetest.xml” for a detailed example. The idea is to first connect each script to a property to be examined. Then, you make test cases using these defined properties.

4. Run the tests by changing directory to TestSuite/ in this case, and issue beetest [path of executable] in the shell.

The beetest.xml file

Properties section:

<?xml version="1.0"?>
<beetest>
    <properties>    
        <property name="energy">
            <description>Total energy (eV)</description>
            <script>Scripts/Energy.sh</script>
            <type>Float</type>
            <relativeprecision>1.0E-5</relativeprecision>
        </property>
        <property name="magmom">
            <description>Magnetic moment (uB)</description>
            <script>Scripts/Magmom.sh</script>
            <type>Float</type>
            <absoluteprecision>1.0E-2</absoluteprecision>
        </property>

Tests section:

<testsuite name="quick">
    <!-- These are quick calculations of simple periodic systems
    with high symmetry. To ensure that these are sane calculations, the 
    results have also been cross-checked against previous published
    results in the literature
    -->
    <description>VASP 4/5 (small calculations)</description>
    <date>2011-02-03</date>
    <author>Peter Larsson</author>

    <testcase name="Fe bcc spin-polarized">
        <path>Tests/Fe-bcc</path>
        <energy>-8.431093</energy>
        <magmom>2.2105</magmom>
        <iterations>16</iterations>
        <firstncg>884</firstncg>
        <potcar>PAW_PBE/Fe_pv</potcar>
    </testcase>

There are two kind of objects: test suites and test cases. You group together several testcase, into a testsuite, which can then be run as one set of calculations/tests.

Extra options

Beetest has some flags to invoke more advanced features like regression versus earlier runs. Pass the “–help” flag to inspect available options.

[pla@matter1 vasptest]$ beetest --help
Usage: beetest [options]

Options:
  -h, --help            show this help message and exit
  -f TESTINPUT, --file=TESTINPUT
                        File with test suite definitions
  -p, --protocol        Write test protocol to disk
  -r REGRESSION, --regression=REGRESSION
                        Test for regression from earlier protocol
  -k, --keep-going      Don't stop if there are errors
  -v, --verbose         Print extra info
  -s SUITE, --suite=SUITE
                        Run named test suite(s)

The “-s” flag is especially useful if the beetest.xml file contains many test suites, and only want to run one of them.