Timeouts for unit tests

  Status: implemented
  • Created: 2006-06-15 22:01:52+0200
  • Categories: util.profiling.unittest
  • Author: friebe

Scope of Change

A timeout facility will be added to unit tests. When a test runs longer than the time span defined by the timeout facility, the test will be marked as timeouted.


Rationale

Integration tests that test certain system parts such as an LDAP server might want to test not only that accessing the server works but also that a query succeeds in a certain time.

Functionality

A test can be annotated with the @limit annotation as follows:

  class LdapIntegrationTest extends TestCase {
  
    #[@test, @limit(time= 1.0)]
    function lookup() {
      // ...
    }
  }

Test failure message: 

  util.profiling.unittest.AssertionFailedError (Timeout) { 
    expected: ["0.100"] but was: ["0.201"] 
  }

Security considerations

n/a

Speed impact

Minimal overhead for checking the existance of the @limit annotation.

Dependencies

n/a

Related documents

- RFC #0032 - This is where the annotation was originally suggested, but later on decided not to be implemented (see comments). It had a different meaning there, though.

- http://xp-framework.net/downloads/rfc0059.diff Patch implementing this RFC.

- http://xp-framework.net/downloads/rfc0059-unittests.diff Patch for XP framework unittest. The author is fully aware of the existance of the usleep() function but uses Windows at home

- http://unittests.xp-framework.net/?net.xp_framework.unittest.tests.LimitTest Verifies the limit functionality works

Comments

- friebe, Tue Jun 20 16:27:39 CEST 2006 Maybe for timeout we should not have TestFailure as result but TestTimeout to be able to distinguish?

- friebe, Mon Jun 26 16:23:00 CEST 2006 Based on discussion with the PHP-Dev team and Frank Kleine of Schlund+Partner, we decided to name the annotation @limit which is a more generic name for this and may be enhanced in the future to support @limit(memory= ...) etc. <EOF>


Table of contents