5. POSS 2015
What do we do when a job fails?
5
Staging CI
According to Jez Humble and David Farley in the reference book
Continuous Delivery:
● “Don’t check in on a broken Build”
● To fix the issue:
○ “Never Go Home on a Broken Build”
○ “Always Be Prepared to Revert to the Previous Revision”
7. POSS 2015
Next?
7
Staging CI
So you need discipline to avoid a broken CI.
● Difficult at scale
● Difficult on Free Software projects as the level of implication is not
equal between participants
9. POSS 2015
Introduction to next gen CI
9
Gating CI
CI is now centered on the code review system:
Review Change Merge ChangeBuild Artifacts Run Tests
● No more broken builds
● Ease the reviewing process by filtering with the tests
● Promote parallel work
10. POSS 2015
Example of OpenStack
10
Gating CI
This CI process has been implemented by the OpenStack project
successfully. Every day:
● 1,000 proposed patchsets
● 7,500 posted comment and votes on the code review system
● 16,000 test environments spawn
● 250 changes merged
11. POSS 2015
Components used by the OpenStack CI
11
Gating CI
Local changes
submitted with
git-review
Gerrit code
review system
git.openstack.org
mirror
github
mirror
Zuul Gearman
Jenkins
servers
Nodepool
Jenkins
slaves
12. POSS 2015
Benefits of the OpenStack CI
12
Gating CI
● No broken master
● Parallel testing using speculative merging
● Cross project dependencies at the changeset level
● Optimized costs by provisioning the slaves on-demand
14. POSS 2015
Components
14
DIY
All the components are Free Software projects
● Download from source or pre-built packages
● Puppet modules for easy configuration
15. POSS 2015
Software Factory
15
DIY
Alternatively, you can use the pre-built integration from softwarefactory-project.io
● ready to use VM with the OpenStack CI components: Gerrit, Zuul, Jenkins and
Nodepool
● Extras:
○ prebuilt integration and delivery pipelines in Zuul
○ SSO
○ web menu and dashboard
○ configuration API and CLI
○ bug tracking system (Redmine)
○ collaboration tools: paste and etherpad