This is part 2 in a series about different caches available to Android Gradle projects. In part 1 I wrote about the benefits provided by Gradle’s cache of incremental builds and the build cache directory. In the second half here, I write about Android’s build cache, the Gradle daemon, and dependency caching
Android’s build cache
Everything we talked about so far has been generic to all Gradle projects. However, the Android Gradle Plugin (AGP) adds some additional caching that warrants a separate discussion. As if the previous build-caching (which I’m now going to refer to as “Gradle build caching”) wasn’t complicated enough, AGP tries to do additional work for us. I’ll leave it up to you to decide if AGP is helping out or making things more complicated. This cache:
stores certain outputs that the Android plugin for Gradle generates when building your project (such as unpackaged AARs and pre-dexed remote dependencies).
…
The build cache also works on continuous integration servers and when running multiple build processes on a single local machine.
I’ve actually found the older documentation (you’ll have to stop the redirect to see it) more illuminating:
Note that the build cache works independently of Gradle’s cache management (e.g., reporting up-to-date statuses)
So this cache works across builds, across projects, and across machines, similar to the Gradle build cache.
The AGP build cache lives in <user-home>/.android/build-cache/ (or one of a few other possible locations). I’m using Android Gradle Plugin 3.6.0. So I see a sub directory for the cache
$ pwd
/Users/jasonatwood/.android/build-cache
$ ls -al
drwxr-xr-x 5 jasonatwood 160 Feb 7 15:32 .
drwxr-xr-x 35 jasonatwood 1120 Jan 9 17:41 ..
drwxr-xr-x 328 jasonatwood 10496 Feb 7 15:33 3.6.0-rc01
-rw-r--r-- 1 jasonatwood 0 Feb 7 14:46 3.6.0-rc01.lock
We can clear this AGP cache with:
$ ./gradlew cleanBuildCache
So let’s see what things look like if we clean, clear Gradle cache, and clear AGP cache:
$ ./gradlew clean
$ rm -rf $GRADLE_HOME/caches/build-cache-*
$ ./gradlew cleanBuildCache
$ ./gradlew --build-cache --scan app:assembleDebug
BUILD SUCCESSFUL in 1m 15s
279 actionable tasks: 189 executed, 85 from cache, 5 up-to-date
Remember from part 1 that even though we have cleared all the caches we still have some tasks that are marked as “from cache”.
Side note: I have since made changes to the test project so any direct time comparison with the Gradle executions in part 1 won’t be very meaningful.
We can then inspect the contents of the AGP cache directory.
$ pwd
/Users/jasonatwood/.android/build-cache/3.6.0-rc01
$ls -al
...
drwxr-xr-x 4 jasonatwood staff 128 Feb 7 15:33 fe99ffa14773dc747b88a457a32a97b88d639502d96e01a74549ce6c13c1e29b
-rw-r--r-- 1 jasonatwood staff 0 Feb 7 15:33 fe99ffa14773dc747b88a457a32a97b88d639502d96e01a74549ce6c13c1e29b.lock
drwxr-xr-x 4 jasonatwood staff 128 Feb 7 15:33 fecd098df0182fbf510891743186d52182a04b5712c989ad7ae8a14c63ad8cbf
-rw-r--r-- 1 jasonatwood staff 0 Feb 7 15:33 fecd098df0182fbf510891743186d52182a04b5712c989ad7ae8a14c63ad8cbf.lock
drwxr-xr-x 4 jasonatwood staff 128 Feb 7 15:33 ff42b3969608800b0003e60a96e6f50d6e95fd084b6fdbf25a01f7ef0f86f339
-rw-r--r-- 1 jasonatwood staff 0 Feb 7 15:33 ff42b3969608800b0003e60a96e6f50d6e95fd084b6fdbf25a01f7ef0f86f339.lock
These seem to have a similar UUID pattern as Gradle’s build cache.
Let’s see what we get by using AGP cache. If we clear the incremental build cache and Gradle build cache and rely on just the Android cache:
$ ./gradlew clean
$ rm -rf $GRADLE_HOME/caches/build-cache-*
$ ./gradlew --build-cache --scan app:assembleDebug
BUILD SUCCESSFUL in 55s
279 actionable tasks: 192 executed, 82 from cache, 5 up-to-date
This sounds counterintuitive: fewer tasks were pulled from cache, but we still got lower build times. I ran this test four times on my local machine and got similar results: a 5 – 20 second improvement in build time with nearly identical cached task count.
My guess is that when AGP hits the cache, it does not register tasks as FROM-CACHE the same way that Gradle’s cache does. So we are seeing the effect of the cache with a reduced build time. But we do not really have a way to visualize what tasks are being cached by AGP.
It should be noted that some of the documentation on this cache is is dated and now incorrect. Specifically the parts about disabling the cache. Adding android.enableBuildCache=false results in a gradle error
WARNING: The option setting ‘android.enableBuildCache=false’ is experimental and unsupported.
It is not entirely clear about the state of the AGP cache or where it is heading. During my research I stumbled upon this comment from a Googler
The Android Gradle build cache (different from the Gradle build cache) is now enabled by default and you shouldn’t have to tweak it manually (cleanBuildCache is also not needed as we auto-clean the cache automatically).
We are migrating the Android Gradle build cache to the Gradle build cache too, so this cache will be obsolete soon.
Since the AGP cache is enabled by default it was technically influencing with our results from part one, but since we were not clearing the cache either, I am confident that we can still understand the benefits of “incremental build” and Gradle’s build cache from prior tests. I’ll leave it up to you to repeat part one and clear AGP cache each time.
Memory cache with the Daemon
In each of our previous experiments, we had yet another cache working to our advantage that I have not talked about. This is caching provided by the Gradle Daemon. In terms of additional caching you can think of the Daemon as a process/memory cache. You can read more about how the daemon speeds up builds here
The daemon has the obvious benefit of only requiring Gradle to be loaded into memory once for multiple builds, as opposed to once for each build.
Code is progressively optimized during execution which means that subsequent builds can be faster purely due to this optimization process
The Daemon also allows more effective in memory caching across builds. For example, the classes needed by the build (e.g. plugins, build scripts) can be held in memory between builds. Similarly, Gradle can maintain in-memory caches of build data such as the hashes of task inputs and outputs, used for incremental building.
The daemon is enabled by default, so by not explicitly disabling it (with --no-daemon flag) or stopping it before each assemble (with $ ./gradlew --stop command), we included the benefit of the daemon in our previous build speed experiments. I did this to keep things simple but we can go back and re-run most of those experiments with daemon disabled to exclude any benefit of the daemon. This would have given us a better sense of how the incremental build, Gradle build cache, and Android build cache contributed to build speed. I won’t re-run those experiments here, save for one, our incremental build test:
incremental build + daemon benefits:
$ ./gradlew clean
$ ./gradlew --scan app:assembleDebug # run once
$ ./gradlew --scan app:assembleDebug # run again
BUILD SUCCESSFUL in 6s
317 actionable tasks: 2 executed, 315 up-to-date
incremental build + without daemon benefits:
$ ./gradlew clean
$ ./gradlew --scan app:assembleDebug # run once
$ ./gradlew --scan --no-daemon app:assembleDebug // again with no daemon
BUILD SUCCESSFUL in 27s
317 actionable tasks: 2 executed, 315 up-to-date
You can see that we were able re-use 315 up-to-date task in both experiments, but because we didn’t have the Gradle daemon to keep most of that in memory, it cost us 21 seconds to read all that back into memory in the second run.
Because the daemon is enabled by default you have likely been getting its benefit all along. The take away here is that you don’t want to disable it unless you have good reason, like a build failing for no apparent reason with cryptic error messages. More often than not, the best advice is to simply either stop all gradle instances (by using --stop), or simply killing all Gradle processes via your OS, and running the build again. You can always look at the Gradle daemons and their statuses by running $ ./gradlew --status.
Third-party dependency caching
Another cache that Gradle provides is the caching of third-party jars. Most Android apps depend on at least a few third party jars. Even if you are only using code from AndroidX libs then you still have jars that need to be downloaded from central repositories. Gradle caches these dependencies in the aptly named Dependency Cache.
Just like the build cache, this cache is available across projects on the same machine. In most cases, there’s no point sharing this cache across machines since downloading jars from a central repository is just as fast (if not faster) than downloading them from a build-cache-server type setup. All that being said, if your internet connection is not too fast and you need to pull a lot of dependencies, it can help to locally store them in your local area network and pull from there. If you use any form of CI containers it may make sense to use a dependency cache server so each instance can pull from the same local cache instead of downloading things every time.
Gradle stores this cache in $GRADLE_HOME/.gradle/caches/ specifically jars-3, modules-2 and transforms-2. I don’t have a good grasp of what goes in what. By exploring each directory I am guessing that modules-2 contains .jar files for third party dependencies:
jars-3 contains jars needed to actually start and run Gradle. transforms-2 contains transformed jars. On my machine, it contains lots of jetified jars created when non AndroidX dependencies get modified to reference AndroidX artifacts.
We can run some more experiments to see what this cache saves us in build time. Since this will result in re-downloading lots of jars, the time savings will really depend on your internet connection speed.
# clear incremental build cache
$ ./gradlew clean
# clear Gradle build cache
$ rm -rf $GRADLE_HOME/caches/build-cache-*
# clear AGP cache
$ ./gradlew cleanBuildCache
# clear 3rd party dependency cache
$ rm -rf $GRADLE_HOME/caches/modules-2
$ rm -rf $GRADLE_HOME/caches/transform-2
$ ./gradlew --scan app:assembleDebug
BUILD SUCCESSFUL in 8m 25s
279 actionable tasks: 193 executed, 81 from cache, 5 up-to-date
We can now clear everything but the 3rd party cache:
# clear incremental build cache
$ ./gradlew clean
# clear Gradle build cache
$ rm -rf $GRADLE_HOME/caches/build-cache-*
# clear AGP cache
$ ./gradlew cleanBuildCache
$ ./gradlew --scan app:assembleDebug
BUILD SUCCESSFUL in 1m 39s
279 actionable tasks: 186 executed, 88 from cache, 5 up-to-date
The HTML scan report has a great tab to show network activity
You can see that I had to download ~180 Mb at a cost of “19 minutes” of build time, versus zero network activity on subsequent builds. I’m assuming this is for all threads, sinde the whole build only took 8 minutes. But either way this is a substantial improvement.
Just for fun I repeated the previous experiment, but deleted only the transform-2 directory:
$ ./gradlew clean
$ rm -rf $GRADLE_HOME/caches/build-cache-*
$ ./gradlew cleanBuildCache
$ rm -rf $GRADLE_HOME/caches/transform-2
$ ./gradlew --scan app:assembleDebug
This required zero network activity which proves my idea that transform-2 includes only transformations to already-downloaded jars.
Cache cleanup
Periodically Gradle will cleanup the caches in $GRADLE_HOME for you, with unused-lifespan ranging from 7 days to 30 days depending on whether the cache can be rebuilt locally or not. These include the build cache and the third-party caches. If you need to disable this cleanup entirely you can add org.gradle.cache.cleanup=false to gradle.properties.
I do not believe that Gradle automatically cleans up the incremental-build cache. But given how often I call ./gradlew clean I don’t think it is a big deal.
Neither of these cleanups have any affect on the AGP Build cache. As we said before that can be cleaned up manually. It looks like each new version of Android Studio generates a new cache directory, so a manual cleanup won’t remove directories for other Android Studio versions. Those directories you’ll need to manually delete.
It should be noted that “Invalidate Caches / Restart …” in Android Studio does not affect any of these previously mentioned caches. It is specifically about deleting files cached as part of the IDE’s regular processes. In general, most of the IDE’s problems that require Invalidate Caches, are due to the index getting corrupted or out of date, meaning you get a lot of “red” references and errors, and even auto-complete is broken. In those instances, clearing the IDE’s cache, triggers a full index rebuild which fixes the problem. This is common if you change branches often and/or have a lot of submodules. The point here is that if you experience any trouble from a command line build, “Invalidate Caches / Restart” in Android Studio is not the answer.
Recap
We’ve talked in detail about five different caches available to Android Gradle projects:
- Incremental build: build directories within your project
- Gradle build cache: available across builds/projects/machines
- Android build cache: available across builds/projects, for AGP output
- Gradle daemon: in-memory/process caching
- Third party dependency caching: jars across builds/projects on the same machine
We briefly mentioned additional caches that might work for you:
- kapt output caching, potentially unsafe
- composite build caching, for builds of builds
My initial advice would be to turn on build cache and do not disable any of the rest of them. However most of these caches were not known to me before I started digging into them. So my real advice is to always use the latest versions of Gradle and AGP. Most of these caches are enabled by default or at least become the default state once they reach stability. Instead of always trying to stay up to date on the latest Gradle caching techniques, you are better off just staying on the latest versions of these tools and letting the Gradle and Android Tools developers silently make your builds faster with each release.
Just for the hell of it, let’s build with none of the caches and then rebuild with all of them:
# clear incremental build cache
$ ./gradlew clean
# clear Gradle build cache
$ rm -rf $GRADLE_HOME/caches/build-cache-*
# clear AGP cache
$ ./gradlew cleanBuildCache
# clear 3rd party dependency cache
$ rm -rf $GRADLE_HOME/caches/modules-2
$ rm -rf $GRADLE_HOME/caches/transform-2
# stop gradle daemon
$ ./gradlew --stop
# run a build
$ ./gradlew --scan app:assembleDebug
BUILD SUCCESSFUL in 11m 49s
279 actionable tasks: 192 executed, 82 from cache, 5 up-to-date
# re-run a build without changing anything
$ ./gradlew --scan app:assembleDebug
BUILD SUCCESSFUL in 9s
279 actionable tasks: 279 up-to-date
This is a massive improvement.
Thanks
I was initially introduced to the build cache concept by this post by Joshua de Guzman. That’s what got me researching all of this.
I then drew inspiration and details from these two additional posts by Vesselin Iliev and Fedor Korotkov.
A big thanks goes to Martin Marconcini for reviewing this post.
Lastly I want to thank anyone that’s ever contributed to the Gradle documentation. It is surprisingly easy to read and understand.