Vegatron 🌈✨

Kotlin's runTest

The flaky test

Recently we had a test that would fail. The test would set up some mocks and run a call, then some assertions.

@Test
fun `non-retryable status does not retry`() = runTest {
    val engine = MockEngine { _ -> respondError(HttpStatusCode.Unauthorized) }

    val result = ...

    assertTrue(result.isFailure)
    ...
    assertEquals(401, ex.statusCode)
    assertEquals(1, engine.requestHistory.size)
}

It seemed benign, almost boring.

But it failed anyway. Something was wrong, but it wasn’t immediately obvious.

runTest has caveats

We built the tests with runTest early in the project. Over time the client added features beyond a single request, like the retry logic under test above.

There’s a minor point in the docs with large implications:

On JVM and Native, this function behaves similarly to runBlocking, with the difference that the code that it runs will skip delays. 1

Time moves differently

runTest uses a synthetic clock that can advance. For simple tests without timing issues, this is great.

In other cases, where time plays a crucial role, this leads to confusing results.

Minimal example

package timing.test

import kotlin.test.Test
import kotlin.test.assertEquals
import kotlinx.coroutines.Dispatchers
import kotlinx.coroutines.delay
import kotlinx.coroutines.runBlocking
import kotlinx.coroutines.test.runTest
import kotlinx.coroutines.withContext
import kotlinx.coroutines.withTimeout

class VirtualTimeVsRealTimeTest {

    private suspend fun respondWithinTimeout(): String =
        withTimeout(timeMillis = 1000) {
            withContext(Dispatchers.Default) {
                delay(50) // real wall-clock delay: runs off the test scheduler
                "response"
            }
        }

    @Test
    fun `runTest fast-forwards`() = runTest {
        // Under virtual time the sequence is:
        //   - withTimeout schedules a delay(1000) on the TEST scheduler.
        //   - the body suspends on Dispatchers.Default (real time, off-scheduler).
        //   - the test scheduler now sees itself as idle, so it JUMPS the virtual
        //     clock straight to 1000ms and fires the timeout...
        //   - ...while the real 50ms delay has had ~no wall-clock time to run.
        
        // What we get:
        // assertFailsWith<TimeoutCancellationException> {
        //   respondWithinTimeout()
        //}
        
        // What we want:
        assertEquals("response", respondWithinTimeout())
        
    }

    @Test
    fun `runBlocking uses real time`() = runBlocking {
        // Real clock for everyone.
        assertEquals("response", respondWithinTimeout())
    }
}

The fix

The fix for the test was simple: runBlocking instead of runTest. Timing was crucial to the logic under test, so a synthetic clock was actively hostile to it. runBlocking runs everything against the real clock. There’s no test scheduler to idle-detect and jump ahead, so a withTimeout can never race a genuine delay the way it did above. The run-time cost was small, so trading a little speed for a test that means what it says was an easy call.

  1. https://kotlinlang.org/api/kotlinx.coroutines/kotlinx-coroutines-test/kotlinx.coroutines.test/run-test.html I’d love if runTest highlighted in a different manner in editors; it doesn’t stand out against the other syntax. ↩︎

#kotlin #test #time