Improving unit test performance in MEGA Android

MEGA on 2024-01-11

Improving unit test performance in MEGA Android

By Erick Sumargo, Senior Android Engineer, Mobile, MEGA

Overview

MEGA Android codebase has +3K unit tests running in our daily pipelines.

With build scan’s support, we observed that every time we run the unit test, the build system will execute following main tasks:

Out of these tasks, main source compilation takes the longest as it is executed in the same phase as assembling the APK. This means all its sub-tasks, e.g., injectCrashlyticsBuildIds, kapt*, etc. are running, which can be a lot of overhead just for running the unit test.

This post narrates our analysis effort in auditing and inventing a “lite” test configuration — a configuration for disabling unnecessary Gradle tasks, resulting in faster unit test compilation time, both on local and CI.

Disclaimer The experiment is conducted in local machine with following specs:

- Operating system: macOS 13.5 (aarch64) - CPU cores: 10 cores (MacBook Pro M1 Max) - Max Gradle workers: 10 workers - Java runtime: JetBrains s.r.o. OpenJDK Runtime Environment 17.0.6+0–17.0.6b829.9–10027231 - Java VM: JetBrains s.r.o. OpenJDK 64-Bit Server VM 17.0.6+0–17.0.6b829.9–10027231 (mixed mode) - Max JVM memory heap size: 10 GiB

injectCrashlyticsBuildIds

Let’s start with the low-hanging fruit. We use Crashlytics to track our app quality, however its Gradle task is registered in the build graph as default, which doesn’t bring any contribution at all for running the unit test. Disabling it can easily save us ~10 seconds with the following script:

tasks.matching { it.name.startsWith("injectCrashlytics") }
     .configureEach { enabled = false }

kaptKotlin

kapt is notoriously known for slowing down builds due to its processing nature as an interoperability bridge so Java’s APT can run. kapt often breaks incremental compilation because of its sensitivity to classpath changes and tendency to rerun often. This means the build-cache is likely invalidated every time we change main sources while iterating on tests, although no ABI has changed.

But unlike the previous task which we can easily disable, we can’t disable this kapt task completely because some of its subtasks, e.g., data binding, are still needed to have main sources successfully compile. Instead, the best we can do is to audit registered kapt libraries, and remove unnecessary ones, so kapt can run efficiently during unit test compilation.

Dagger-Hilt

Our unit tests align with Hilt testing philosophy where we directly instantiate the SUT by passing in fake or mock dependencies, so no Hilt injection required. That said, both kapt and kaptTest configurations for this library are not necessary to run because its generated classes are not needed at all.

The efficiency logic is, if kapt is disabled, then the java stubs gen task is skipped. Since no java stubs are generated, the annotation processor has no input to process, which usually takes rounds to emit other java sources. Because there is no java source output, javac task has much less sources to compile, which results in a huge time-saving gain.

To prove this thesis, we ran ./gradlew :app:testDebugUnitTest and all tests still passed.

Google’s AutoValue

This is simply an unused kapt in some modules which we probably forgot we could now remove after some refactoring work. Despite having no processor input, disabling unused kapt can still save couple of seconds because of the configuration warm-up time.

Hilt Plugin

Since kapts for Dagger-Hilt have been disabled, running tasks from the Hilt plugin brings no meaningful result. By removing the dagger.hilt.android.plugin we saved ~20 seconds.

kaptUnitTestKotlin

After removing all kaptTest configurations, we notice that this task was somehow still running. Although Gradle mark it NO-SOURCE which means no processor inputs, it can still take few seconds to run each time.

We reached out to the community about this weird behavior, and it appears to be expected since any kapt task extends the main kotlin-kapt plugin as discussed in KT-29481. They also redirected me to this article which addressed the exact same issue. The suggested workaround without breaking downstream kotlinc tasks is to simply remove the task with the following script:

tasks.matching { it.name.startsWith("kapt") && it.name.endsWith("TestKotlin") }
     .configureEach { enabled = false }

Result

With the above tasks filtered out, the unit test compilation time is reduced up to ~2 minutes (baseline ~8 minutes → +25%) for clean build, ~40 seconds (baseline ~160 seconds → +25%) for incremental build, and saves ~15 MB size of generated class from Dagger-Hilt. You can see the new scan result and the benchmark measurement for more details.

The improvement for incremental build actually depends on the number/type of modified module(s). For instance, we observe ~40 seconds improvement when modifying :app module only.

Conclusion

With such positive results, we continued to observe if the configuration introduced regression, but every single test is still working fine. This means that the “lite” configuration is an absolute win with zero trade-off!

As for the next improvement, we’re looking forward to removing kapt tasks completely in unit test compilation, once we have refactored data binding usage to view binding or compose.