Restructuring Litho’s CircleCI config for Workflows

存储架构 2018-06-19 阅读原文

This is a guest post by Pavlos-Petros Tournaris. It originally appeared on his blog here . Pavlos-Petros works as an Android software engineer at Workable. We hope you enjoy!

Nearly a year and a half ago, Facebook released Litho as an open source project. Litho is an Android framework, which lets you define your UI in a declarative way. It immediately got my interest and I started getting my hands dirty with some examples and pet projects.

It was a nice experience getting in touch with Litho and its React-like nature, that was a first for me. The amount of interest in that new area, made me realise that I could dive a bit more into it, by also contributing to the project. So just like any other Open Source project, that I like I started checking the “Issues” tab to see if I could help resolving any bugs or contributing new features.

Automated testing

Litho, just like the majority of open source projects, includes a lot of tests (without tests it is usually hard to gain a developer’s trust). Testing infrastructure on Litho is based on unit tests that are run either through Buck or using Gradle.

But tests alone don’t mean anything if you are not able to have proper feedback on your PRs and also make sure each commit that goes into master branch is green.

Litho was using CircleCI to leverage the run of its unit test suite, gather test results, and publish a snapshot for each commit pushed on master .

Previous situation

Previously, Litho was using a custom Docker image made by Pascal Hartig , one of Litho’s Android engineers.

This image was downloading Buck, building it and saving it in the Docker environment. The same was done for Android NDK and Android SDK respectively. When CircleCI started a build for a commit, it would then configure some needed keys for archives uploading and also exporting Buck into Path. Finally, it started executing all BUCK & Gradle builds for sample projects, as well as executing Buck & Gradle tests, which was finalized by publishing a snapshot and storing test results.

While this worked great, there was the need to configure parallel jobs for Buck and Gradle tests and sample builds.

Problems with the previous situation

While starting to investigating what could be improved, there were certain things I noticed, that could be more performant and others that came up after discussing them with Pascal.

  • CircleCI configuration was using a custom Docker image that was effectively pulled each time the job was running, due to how CircleCI caches Docker images on its containers, costing around 5–6 mins of spinning an environment.
  • Builds & tests were not running in parallel, which meant that even if a Buck build required two minutes to execute tests, it would have to wait for Gradle to also finish executing tests before proceeding.

  • The regex used to collect test results was not collecting Buck test results.

Here comes Workflows

CircleCI provides a feature called Workflows, which allows us to define a list of jobs that will run in parallel, but you can also declare dependencies between jobs, which would effectively change their execution to be sequential.

- checkout_code      
- build:          
      - checkout_code      
- buck_sample_build:          
      - build      
- buck_sample_barebones_build:          
      - build      
- buck_sample_codelab_build:          
      - build      
- gradle_sample_kotlin_build:          
      - build      
- buck_litho_it_tests_run:          
      - build      
- buck_litho_it_powermock_tests_run:          
      - build      
- gradle_tests_run:          
      - build

Transitioning Litho to use Workflows

The first step was to create a job that would check out the repo code and start setting up required dependencies, including: Buck, dependencies that will help us build Buck, such as Ant, and Android NDK. This job was defined as a first step on Litho’s build_and_test workflow.

Depending on that job is the build job, which is responsible for building Litho and saving any Gradle-produced caches.

Afterwards, build is fanning out to seven other jobs responsible to build samples and execute tests. Each one of these jobs is also configured to store test results and upload any produced artifacts.

Fanning out

This workflow is finalized by a publish_snapshot job which is depending on every job that executes tests. After their successful execution, a Gradle task is responsible to upload an archive on Bintray containing the latest changes, as a snapshot.

- publish_snapshot:          
      - buck_litho_it_tests_run            
      - buck_litho_it_powermock_tests_run            
      - gradle_tests_run

Caches & Workspace

CircleCI’s config DSL, includes amongst others, cache & workspace . In order to better explain those terms, let me define them as follows:

  • cache is saved content that will be retained between workflow executions.

Caching in CircleCI

- &save-repo-cache    
    - ~/.gradle/caches      
    - ~/.gradle/wrapper
  • workspace is saved content that will be retained among jobs executions of the same workflow.

Saving content with workspaces

- persist_to_workspace:          
    root: workspace          
      - repo

For example, something that would be worth caching would be any Gradle-cached dependencies, provided that there is a proper cache-busting mechanism in place. Whereas our repo’s code is something that all jobs need so it’s worth persisting it to our workspace in order for every job to be able to “attach” and get access to that content.

attach_workspace: &attach_workspace  
    at: ~/litho-working-dir/workspace

Job configuration

Jobs that are part of a workflow can also define filters for branches. This is especially helpful, since we would not like to execute publish_snapshot job in case the commit we were running on is not on master branch, but rather a PR. This might seem like a small change but it can save 7-8 minutes of waiting for the workflow’s execution to finish.

    only: master

Post-restructure improvements

After a lot of commits, workflow executions and discussions with Pascal on improvable points, we finally managed to have a workflow running in around 15–20 minutes. Here are some points that we improved:

  • We removed the custom Docker image by using CircleCI’s Android image and pulling any needed dependencies on start-up. Gave us a 1sec container configuration for 95% (or even more) of job executions.
  • We used CircleCI’s Android Docker image Android SDK, instead of downloading from scratch.
  • We made test & build jobs run in parallel, allowing for early feedback in case something is broken only on a “sample” module for example. Combined with Github’s integration this is really informative, because you can instantly be notified if a Job responsible to execute tests for your changes, has failed or not, without having to wait for the whole workflow to finish.
  • We collected previously uncollected test results from Buck test executions.


It has been a fun experience and eye-opening experience. I have never dealt with CI tools in that extent before and I can totally say that DevOps or CI provisioning is a certainly a difficult job (respect to our colleagues out there who have to deal with that every day).

Also, thanks to Pascal Hartig for helping on the process and discussing points of improvement.

You can find the commit that restructures config.yml here:

And Litho’s final config.yml here:

The Circle Blog

责编内容by:The Circle Blog阅读原文】。感谢您的支持!


『中级篇』 Docker Bridge详解(26) 上节主要学习了network-namespace,并创建了network-namespace,并把2个network-namespace连接在一起,我们也演示了创建一个容器test1和test2,其实在创建容器的同时也创建了对应的一个net...
想让容器更快?这五种方法您必须知道!... 容器的卖点之一是容器化应用程序的部署速度通常比虚拟机快,且性能更佳。 虽然容器的默认速度比其他基础设施快,这并不意味着没有办法让它们更快。本文将演示如何通过优化Docker容器镜像构建时间、性能和资源消耗,来让容器的速度与性能更超默认值...
一文搞懂各种 Docker 网络 – 每天5分钟玩转 Docker 容器技术(72)... 前面各小节我们先后学习了 Docker Overaly,Macvaln,Flannel,Weave 和 Calico 跨主机网络方案。目前这个领域是百家争鸣,而且还有新的方案不断涌现。 本节将从不同维度比较各种网络方案,大家在选择的时候...
docker – 启动具有多个网络接口的容器 容器启动后,您可以使用“docker network connect”进行操作,但这意味着该进程已经在运行,可能会错过新的. 这个问题是关于码头和多个网络接口的搜索.虽然不是所需的版本在我离开这里的一些信息: 使用Do...
每日投融资速递 | 企业软件公司 Docker 又拿到钱,西婵、联易融等 44 家公司获得融资——2... 编者按:这里是36氪推出的【每日投融资速递】栏目,盘点当天国内外投融资动态。今日国内的焦点是医疗整形美容机构“西婵”,海外的投融资关注重点是企业软件公司 Docker 宣布获得7500万美元融资。 本日国内外共有 44 起融资...