iOS Dependency Integration at LuizaLabs
Comparative study between Cocoapods, Swift Package Manager and Carthage on the LuizaLabs SuperApp project: build time, indexing, tests, app size and CI Bitrise impact.
By: Carlos Henrique, Gabriel Beltrame, Rafael Valer
Intro
We on the iOS mobile platform team at LuizaLabs decided to publish a study we ran internally, aiming to help the community and inform about dependency integration. Today there are different dependency managers in the iOS world and there is always the question of which one fits us best, is more performant, more scalable, etc. So we decided to get hands-on and gather the data.
We recently migrated from Buck to Tuist. With that, we are working to improve project performance during build and project generation, which lets us gain more time for actual coding.
One of the impactful points in this process is the way we integrate external dependencies into the project. For that reason we ran tests using two other well-known integration approaches - Carthage and Swift Package Manager (SPM) - to compare against the current setup (Cocoapods).
Test goals
- Analyse in detail the different dependency integration methods on the SuperApp and collect results around different performance aspects.
- Help the iOS mobile platform team decide which method to use when integrating external dependencies into the project.
Approach
To measure the impact and performance of the different existing integration modes, we ran tests focused on characteristics related to project compilation.
Since it was not possible to migrate the full set of external dependencies to both analysed managers, a direct comparison between Carthage vs. SPM could not be made. So only SPM vs. Cocoapods and Carthage vs. Cocoapods analyses were performed.
** For each observed metric we ran N data collection tests; the value of N is shown next to each result.*
Devices used
Cocoapods vs. SPM vs. Carthage - Application
- Cold Make Project - Project generation after a clean. (N=5)
- Warm Make Project - Project generation after it was already generated previously. (N=5)
- Cold Indexing - Project indexing after a clean. (N=3)
- Warm Indexing - Project indexing after it was already indexed. (N=7)
- Cold Build - Project build after a clean. (N=3)
- Warm Build - 1st build of the project after start, without dependencies having been wiped. (N=7)
- Incremental Build - Project rebuild after a file change. In this case we edited a method on one of the modules. (N=7)
- Tests were performed with dependencies already downloaded.
Results analysis
- SPM vs Cocoapods: average increase in Cold Make Project time of ~61.2% with SPM. This large difference during Cold Make Project is explained by the fact that SPM fetches dependencies during the make project step. Average increase in Warm Make Project time of ~5.4% with SPM.
- Carthage vs Cocoapods: average reduction in Cold Make Project time of ~10.2% with Carthage. Average reduction in Warm Make Project time of ~10.4%.
Project indexing
Although Cold Indexing time showed an average reduction of ~9% comparing SPM with Cocoapods, and ~8.9% with Carthage, the times measured on the Macbook Pro i7 machine bring unexpected results around this measure. Additional tests should be done in the future to draw more assertive conclusions.
Project compilation
- SPM vs. Cocoapods: average reduction in Cold Build time of ~47% with SPM. Warm Build inconclusive. Incremental Build with slight reduction of ~5% with SPM.
- Carthage vs Cocoapods: average reduction in Cold Build time of ~15% with Carthage. Average reduction in Warm Build of ~9%. Average reduction in Incremental Build of ~12%.
Project tests
Fastlane Test: SPM increased ~2.5%. Carthage reduced ~8.7%.
Warm Test Build: SPM reduced ~3%. Carthage reduced ~12%.
Incremental Test Build: SPM reduced ~14%. Carthage reduced ~22%.
CI time - Bitrise
We also noticed a reduction in total test duration on CI when using SPM/Carthage compared to Cocoapods.
- SPM: ~3.1% reduction
- Carthage: ~4.6% reduction
App size
- Testflight: SPM reduced ZIP size by ~3.8% compared to Cocoapods. SPM reduced installed-app size by ~2.4%. Carthage increased ZIP size by ~6.7%. Carthage increased installed-app size by ~1.6%.
Conclusions
With the tooling switch from Buck to Tuist for iOS project generation, it was possible to run this study. Its main goal is to test and compare different dependency managers and their distinct performance aspects.
Among the main results, we can highlight:
Build generation:
- SPM - increase in project generation time, both cold and warm, ~61.2% and ~5.4% respectively. This is easily explained since SPM fetches dependencies during this step. Additionally, this time can be reduced via the disablePackageVersionLocking flag (Config.swift | Tuist Documentation).
- Carthage - reduction in project generation time, both cold and warm, ~10.2% and ~10.4% respectively.
Project indexing
- Both managers (SPM and Carthage) showed partial results with around 9% reduction in cold indexing times.
Project compilation
- SPM - reduction in cold build and incremental build times, ~47% and ~5% respectively. Warm build inconclusive.
- Carthage - reduction in cold build, warm build and incremental build times, ~15%, ~9% and ~12% respectively.
Project tests
- SPM - slight reduction in test lane time of ~2.5%, in test warm build of ~3% and incremental build of ~14%.
- Carthage - reduction in test lane time of ~8.7%, in test warm build of ~12% and incremental build of ~22%.
CI time:
- Both managers (SPM and Carthage) showed a slight reduction in total CI test duration of ~3.1% and ~4.6% respectively.
App size:
- SPM - reduction in ZIP size (~3.8%) and installed-app size on device (~2.4%) for Testflight.
- Carthage - increase in ZIP size (~6.7%) and installed-app size on device (~1.6%) for Testflight.
This reduction in final binary size when using SPM has also been reported in other experiments (What Adding Dependencies Will Do To Your App in 2020).
Other analyses:
- Although in most measures Carthage outperformed SPM, since we migrated a larger number of dependencies to Carthage than to SPM, this difference may have been responsible for the first one’s better performance. Additionally, to use Carthage we must pre-compile external libraries and place them in a dedicated repository to download them as needed. This process generates somewhat heavy maintenance and falls short compared to the easy integration process via Pods/SPM.
- The comparison was done with a relatively small number of external dependencies, leaving open a margin of performance improvement as more external libs are migrated.
Our conclusion from this study is that migrating dependencies from Cocoapods to SPM will result in a series of benefits tied to build and indexing performance, in addition to also reducing the size of binaries generated in release versions.
Although Carthage also showed good results, we understand that its application would require many additional steps, raising platform complexity and maintenance. Also, the fact that Apple itself owns Swift Package Manager gives us a guarantee of maintenance and eventual improvements in how it integrates with native iOS projects. Carthage also creates a lot of headache to update Xcode versions and forces the entire team to always use the same version.
Finally, it is worth highlighting that the tool currently used for project generation - Tuist - also supports SPM. It even includes a new method for dependency integration (currently in Alpha Version - Adding external dependencies | Tuist Documentation).
Hope you enjoyed this study. These results are specific to our project setup and that does not mean the results will repeat for other projects with different configurations.
If you have any question, just comment.
Until next time, folks.
Originally published on Medium
This article was originally published on April 26, 2022. The version on buildcomcarlos.com is the integral editorial copy maintained on my site. You can read the original on Medium with the original layout, claps and responses.
- Original version on Medium
- Back to articles list